Repository: open-mmlab/mmdeploy

Issue:
[Bug] RTMDet-Ins inference with SDK , the result without mask 
### Checklist

- [X] I have searched related issues but cannot get the expected help.
- [X] 2. I have read the [FAQ documentation](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/faq.md) but cannot get the expected help.
- [X] 3. The bug has not been fixed in the latest version.

### Describe the bug

**Python:**
![image](https://user-images.githubusercontent.com/36351639/223660360-67e3b928-b5a3-4b9d-a195-f6d2ebfaf434.png)
![image](https://user-images.githubusercontent.com/36351639/223660878-617895cf-08d7-4e4f-be4b-7fae7247b1be.png)

**TIt doesn't exist in C++ either:**
![image](https://user-images.githubusercontent.com/36351639/223662237-677a1727-92ab-42fd-81ed-94a8c146461d.png)
![image](https://user-images.githubusercontent.com/36351639/223662895-b8ef9c95-ce40-455e-8617-9630017b13f5.png)

**This issue is also mentioned in this link:**
https://github.com/open-mmlab/mmdeploy/issues/1722#issue-1573976926 

How to fix it? Thanks!

### Reproduction

Official example:
https://github.com/open-mmlab/mmdetection/tree/3.x/configs/rtmdet 
https://mmdeploy.readthedocs.io/en/1.x/get_started.html#installation

### Environment

```Shell
It seems to have nothing to do with the environment.
```


### Error traceback

_No response_


Discussion:
1. Which deploy config are you using to convert the model?
2. Can you share the `pipeline.json` exported in the model work-dir?
> 1. Which deploy config are you using to convert the model?
> 2. Can you share the `pipeline.json` exported in the model work-dir?

**1**
```
python mmdeploy/tools/deploy.py \
  mmdeploy/configs/mmdet/instance-seg/instance-seg_rtmdet-ins_onnxruntime_static-640x640.py \
  mmdetection/configs/rtmdet/rtmdet-ins_tiny_8xb32-300e_coco.py \
  checkpoints/rtmdet-ins_tiny_8xb32-300e_coco_20221130_151727-ec670f7e.pth \
  mmdetection/demo/demo.jpg \
  --work-dir ./work_dirs/rtmdet-ins \
  --device cpu \
  --dump-info
```
**2**
`{
    "pipeline": {
        "input": [
            "img"
        ],
        "output": [
            "post_output"
        ],
        "tasks": [
            {
                "type": "Task",
                "module": "Transform",
                "name": "Preprocess",
                "input": [
                    "img"
                ],
                "output": [
                    "prep_output"
                ],
                "transforms": [
                    {
                        "type": "LoadImageFromFile",
                        "file_client_args": {
                            "backend": "disk"
                        }
                    },
                    {
                        "type": "Resize",
                        "keep_ratio": true,
                        "size": [
                            640,
                            640
                        ]
                    },
                    {
                        "type": "Pad",
                        "size": [
                            640,
                            640
                        ],
                        "pad_val": {
                            "img": [
                                114,
                                114,
                                114
                            ]
                        }
                    },
                    {
                        "type": "Normalize",
                        "to_rgb": false,
                        "mean": [
                            103.53,
                            116.28,
                            123.675
                        ],
                        "std": [
                            57.375,
                            57.12,
                            58.395
                        ]
                    },
                    {
                        "type": "Pad",
                        "size_divisor": 1
                    },
                    {
                        "type": "DefaultFormatBundle"
                    },
                    {
                        "type": "Collect",
                        "meta_keys": [
                            "valid_ratio",
                            "pad_shape",
                            "flip_direction",
                            "filename",
                            "flip",
                            "img_id",
                            "img_shape",
                            "img_norm_cfg",
                            "pad_param",
                            "img_path",
                            "ori_shape",
                            "scale_factor",
                            "ori_filename"
                        ],
                        "keys": [
                            "img"
                        ]
                    }
                ]
            },
            {
                "name": "rtmdet",
                "type": "Task",
                "module": "Net",
                "is_batched": false,
                "input": [
                    "prep_output"
                ],
                "output": [
                    "infer_output"
                ],
                "input_map": {
                    "img": "input"
                },
                "output_map": {}
            },
            {
                "type": "Task",
                "module": "mmdet",
                "name": "postprocess",
                "component": "ResizeBBox",
                "params": {
                    "nms_pre": 1000,
                    "min_bbox_size": 0,
                    "score_thr": 0.05,
                    "nms": {
                        "type": "nms",
                        "iou_threshold": 0.6
                    },
                    "max_per_img": 100,
                    "mask_thr_binary": 0.5
                },
                "output": [
                    "post_output"
                ],
                "input": [
                    "prep_output",
                    "infer_output"
                ]
            }
        ]
    }
}`
There is a bug when exporting the pipeline that cause the mask output to be ignored. You may try to modify "ResizeBBox" to "ResizeInstanceMask" in `pipeline.json` for now.

And the config `instance-seg_rtmdet-ins_onnxruntime_static-640x640.py` is exporting the mask post-processing in the model which conflicts with SDK mask post-processing. You may need to set `export_postprocess_mask=False` in the config for SDK to work.
> There is a bug when exporting the pipeline that cause the mask output to be ignored. You may try to modify "ResizeBBox" to "ResizeInstanceMask" in `pipeline.json` for now.
> 
> And the config `instance-seg_rtmdet-ins_onnxruntime_static-640x640.py` is exporting the mask post-processing in the model which conflicts with SDK mask post-processing. You may need to set `export_postprocess_mask=False` in the config for SDK to work.

**Hi! I tried to modify "ResizeBBox" to "ResizeInstanceMask" in `pipeline.json` and mask output does have information, but it just have the w,h instead of the points of mask. how can i get the mask? There are also some issues with the output image.**

![image](https://user-images.githubusercontent.com/36351639/223947909-aea97be3-f767-48d6-b354-4a060325bf65.png)
![detector_output](https://user-images.githubusercontent.com/36351639/223948034-9bdbbb5c-6a6c-4076-9f35-90dc062eb961.jpg)

**After I set `export_postprocess_mask=False` in the config and exported new json to infer with the SDK of C++ , I got an error:**

![image](https://user-images.githubusercontent.com/36351639/223951724-b8bbe52d-1d4b-4731-bbd1-b73d59cd3804.png)

I just verified that the `export_postprocess_mask` has no effect on the converted model. RTMDet-Ins is generating full frame masks of every instance anyway. 

The kind the of masks are not supported by the SDK yet. (It was verified on Mask R-CNN which generates 7x7 masks of the instances)
> I just verified that the `export_postprocess_mask` has no effect on the converted model. RTMDet-Ins is generating full frame masks of every instance anyway.
> 
> The kind the of masks are not supported by the SDK yet. (It was verified on Mask R-CNN which generates 7x7 masks of the instances)

Will RTMDet-Ins support SDK deployment in the future, and currently it will make an error when converting its own trained model, I mentioned here https://github.com/open-mmlab/mmdeploy/issues/1853#issue-1615054132
Waiting for your good news!

Resolving patch:
diff --git a/demo/python/object_detection.py b/demo/python/object_detection.py
--- a/demo/python/object_detection.py
+++ b/demo/python/object_detection.py
@@ -1,5 +1,6 @@
 # Copyright (c) OpenMMLab. All rights reserved.
 import argparse
+import math
 
 import cv2
 from mmdeploy_python import Detector
@@ -36,7 +37,10 @@ def main():
         if masks[index].size:
             mask = masks[index]
             blue, green, red = cv2.split(img)
-            mask_img = blue[top:top + mask.shape[0], left:left + mask.shape[1]]
+
+            x0 = int(max(math.floor(bbox[0]) - 1, 0))
+            y0 = int(max(math.floor(bbox[1]) - 1, 0))
+            mask_img = blue[y0:y0 + mask.shape[0], x0:x0 + mask.shape[1]]
             cv2.bitwise_or(mask, mask_img, mask_img)
             img = cv2.merge([blue, green, red])
 
diff --git a/mmdeploy/codebase/mmdet/deploy/object_detection.py b/mmdeploy/codebase/mmdet/deploy/object_detection.py
--- a/mmdeploy/codebase/mmdet/deploy/object_detection.py
+++ b/mmdeploy/codebase/mmdet/deploy/object_detection.py
@@ -311,7 +311,8 @@ def get_postprocess(self, *args, **kwargs) -> Dict:
             params['score_thr'] = params['rcnn']['score_thr']
             if 'mask_thr_binary' in params['rcnn']:
                 params['mask_thr_binary'] = params['rcnn']['mask_thr_binary']
-                type = 'ResizeInstanceMask'  # for instance-seg
+        if 'mask_thr_binary' in params:
+            type = 'ResizeInstanceMask'  # for instance-seg
         if get_backend(self.deploy_cfg) == Backend.RKNN:
             if 'YOLO' in self.model_cfg.model.type or \
                'RTMDet' in self.model_cfg.model.type:


Test patch: