Repository: open-mmlab/mmdetection3d

Issue:
AssertionError: Input boxes shape should be [N, 5] when train point_rcnn
**Describe the bug**
This error appears when train point_rcnn using kitti.
The input shape of NMS in [point_rpn_head.py](https://github.com/open-mmlab/mmdetection3d/blob/c20b6cb637fa31b88b1c7b523116f4f606010094/mmdet3d/models/dense_heads/point_rpn_head.py#L340) is N\*7, while NMS function in [mmcv](https://github.com/open-mmlab/mmcv/blob/235c0253ab8806a2a2ee6954b4258a95358497ac/mmcv/ops/iou3d.py#L50) only accept input with shape N\*5 

**Reproduction**
mmdetection3d==1.0.0rc1
mmcv==1.4.8



Discussion:
I found that boxes3d is converted to bev before nms in origin code of [PointRCNN](https://github.com/sshaoshuai/PointRCNN/blob/1d0dee91262b970f460135252049112d80259ca0/lib/rpn/proposal_layer.py#L103). But this process does not appear in point_rpn_head.py

code block of origin PointRCNN
![image](https://user-images.githubusercontent.com/38032220/164153479-ab9f856e-a222-41c2-b384-5887d8fd6e5a.png)

code block of point_rpn_head.py
![image](https://user-images.githubusercontent.com/38032220/164153614-feee07a2-a4f3-4b25-bfea-28fe478ddd67.png)


Resolving patch:
diff --git a/mmdet3d/models/dense_heads/point_rpn_head.py b/mmdet3d/models/dense_heads/point_rpn_head.py
--- a/mmdet3d/models/dense_heads/point_rpn_head.py
+++ b/mmdet3d/models/dense_heads/point_rpn_head.py
@@ -3,6 +3,7 @@
 from mmcv.runner import BaseModule, force_fp32
 from torch import nn as nn
 
+from mmdet3d.core import xywhr2xyxyr
 from mmdet3d.core.bbox.structures import (DepthInstance3DBoxes,
                                           LiDARInstance3DBoxes)
 from mmdet3d.core.post_processing import nms_bev, nms_normal_bev
@@ -320,29 +321,33 @@ def class_agnostic_nms(self, obj_scores, sem_scores, bbox, points,
         else:
             raise NotImplementedError('Unsupported bbox type!')
 
-        bbox = bbox.tensor[nonempty_box_mask]
+        bbox = bbox[nonempty_box_mask]
 
         if self.test_cfg.score_thr is not None:
             score_thr = self.test_cfg.score_thr
             keep = (obj_scores >= score_thr)
             obj_scores = obj_scores[keep]
             sem_scores = sem_scores[keep]
-            bbox = bbox[keep]
+            bbox = bbox.tensor[keep]
 
         if obj_scores.shape[0] > 0:
             topk = min(nms_cfg.nms_pre, obj_scores.shape[0])
             obj_scores_nms, indices = torch.topk(obj_scores, k=topk)
-            bbox_for_nms = bbox[indices]
+            bbox_for_nms = xywhr2xyxyr(bbox[indices].bev)
             sem_scores_nms = sem_scores[indices]
 
-            keep = nms_func(bbox_for_nms[:, 0:7], obj_scores_nms,
-                            nms_cfg.iou_thr)
+            keep = nms_func(bbox_for_nms, obj_scores_nms, nms_cfg.iou_thr)
             keep = keep[:nms_cfg.nms_post]
 
-            bbox_selected = bbox_for_nms[keep]
+            bbox_selected = bbox.tensor[indices][keep]
             score_selected = obj_scores_nms[keep]
             cls_preds = sem_scores_nms[keep]
             labels = torch.argmax(cls_preds, -1)
+        else:
+            bbox_selected = bbox.tensor
+            score_selected = obj_scores.new_zeros([0])
+            labels = obj_scores.new_zeros([0])
+            cls_preds = obj_scores.new_zeros([0, sem_scores.shape[-1]])
 
         return bbox_selected, score_selected, labels, cls_preds
 


Test patch: