Repository: open-mmlab/mmocr

Issue:
Batch inference during evaluation in text detection training
I'm training a `DBNet_r50dcn` model with `tools/train.py` script and, even though training epoch is executed in batch with `samples_per_gpu`, during evaluation one image at time is fed to the model. Clearly this makes the whole execution extremely slow if we have a large validation set.

Is there a solution to run in batch mode also the evaluation step?

To summarize: same problem of [this comment](https://github.com/open-mmlab/mmocr/issues/294#issuecomment-864533214), but in my case is for text detection.




Discussion:
Batch inference support for text detection is in our plan.
We will add new test pipeline that will resize and pad the image to the same size, so that batch inference can work.
Currently, the test pipeline is just keeping ratio resize (following academic setting), every instance in testset is in a different shape.
hi @cuhk-hbsun, that would be awesome. Is there an approximate release date? 
@antoniolanza1996 about 20/07/2021
@antoniolanza1996 Our code has already support batch inference for text detection based on `mmdet`. These is not necessary to develop this feature.
You can just delete [line92-93](https://github.com/open-mmlab/mmocr/blob/main/configs/textdet/psenet/psenet_r50_fpnf_600e_ctw1500.py#L92-L93) or set `val_dataloader=dict(samples_per_gpu=4), test_dataloader=dict(samples_per_gpu=4)` to enable batch inference.
The `dataloader` will pad all the images to the largest size in one batch to do batch inference.
@cuhk-hbsun I've tried to set `val_dataloader=dict(samples_per_gpu=4)` and `test_dataloader=dict(samples_per_gpu=4)` in a DBNet config, however I've got an error:

```python
File "tools/train.py", line 219, in <module>
    main()
  File "tools/train.py", line 215, in main
    meta=meta)
  File "/content/mmocr/mmocr/apis/train.py", line 161, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/epoch_based_runner.py", line 127, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/epoch_based_runner.py", line 54, in train
    self.call_hook('after_train_epoch')
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/base_runner.py", line 307, in call_hook
    getattr(hook, fn_name)(self)
  File "/usr/local/lib/python3.7/dist-packages/mmdet/core/evaluation/eval_hooks.py", line 146, in after_train_epoch
    results = single_gpu_test(runner.model, self.dataloader, show=False)
  File "/usr/local/lib/python3.7/dist-packages/mmdet/apis/test.py", line 27, in single_gpu_test
    result = model(return_loss=False, rescale=True, **data)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/parallel/data_parallel.py", line 42, in forward
    return super().forward(*inputs, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/parallel/data_parallel.py", line 153, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/fp16_utils.py", line 98, in new_func
    return old_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mmdet/models/detectors/base.py", line 183, in forward
    return self.forward_test(img, img_metas, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mmdet/models/detectors/base.py", line 160, in forward_test
    return self.simple_test(imgs[0], img_metas[0], **kwargs)
  File "/content/mmocr/mmocr/models/textdet/detectors/single_stage_text_detector.py", line 53, in simple_test
    for i in range(len(img_metas))
  File "/content/mmocr/mmocr/models/textdet/detectors/single_stage_text_detector.py", line 53, in <listcomp>
    for i in range(len(img_metas))
  File "/content/mmocr/mmocr/models/textdet/dense_heads/head_mixin.py", line 56, in get_boundary
    text_repr_type=self.text_repr_type)
  File "/content/mmocr/mmocr/models/textdet/postprocess/wrapper.py", line 30, in decode
    return db_decode(**kwargs)
  File "/content/mmocr/mmocr/models/textdet/postprocess/wrapper.py", line 208, in db_decode
    cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
cv2.error: OpenCV(4.1.2) /io/opencv/modules/core/src/copy.cpp:1170: error: (-215:Assertion failed) top >= 0 && bottom >= 0 && left >= 0 && right >= 0 && _src.dims() <= 2 in function 'copyMakeBorder'
```
---

To reproduce, I've updated a notebook [here](https://github.com/antoniolanza1996/miscellaneous/blob/c22880b5ea55180b0ce350f38346cb7b11b4c3ab/mmocr/issues/352/Batch_inference_during_evaluation_in_text_detection_training.ipynb) and the associated `dbnet_custom_config.py` [here](https://github.com/antoniolanza1996/miscellaneous/blob/c22880b5ea55180b0ce350f38346cb7b11b4c3ab/mmocr/issues/352/dbnet_custom_config.py).
@antoniolanza1996 Looks like dbnet sometimes generates invalid bounding boxes. My colleague is fixing that. We will let you know when we have progress.

Resolving patch:
diff --git a/mmocr/models/textdet/dense_heads/db_head.py b/mmocr/models/textdet/dense_heads/db_head.py
--- a/mmocr/models/textdet/dense_heads/db_head.py
+++ b/mmocr/models/textdet/dense_heads/db_head.py
@@ -66,7 +66,7 @@ def forward(self, inputs):
         thr_map = self.threshold(inputs)
         binary_map = self.diff_binarize(prob_map, thr_map, k=50)
         outputs = torch.cat((prob_map, thr_map, binary_map), dim=1)
-        return (outputs, )
+        return outputs
 
     def _init_thr(self, inner_channels, bias=False):
         in_channels = inner_channels
diff --git a/mmocr/models/textdet/losses/db_loss.py b/mmocr/models/textdet/losses/db_loss.py
--- a/mmocr/models/textdet/losses/db_loss.py
+++ b/mmocr/models/textdet/losses/db_loss.py
@@ -133,8 +133,6 @@ def forward(self, preds, downsample_ratio, gt_shrink, gt_shrink_mask,
         assert isinstance(gt_thr, list)
         assert isinstance(gt_thr_mask, list)
 
-        preds = preds[0]
-
         pred_prob = preds[:, 0, :, :]
         pred_thr = preds[:, 1, :, :]
         pred_db = preds[:, 2, :, :]


Test patch: