Repository: open-mmlab/mmocr

Issue:
When I want to train a fcenet, I met a problem
![QQ图片20210609200559](https://user-images.githubusercontent.com/53176521/121351451-5f6cb280-c95e-11eb-982b-b7c65345bb39.png)
It did run, but several epoch,there note that data not on the gpu.
how to solve it?


Discussion:
@ShiMinghao0208 Can you try again and see if the same problem arises?
> @ShiMinghao0208 Can you try again and see if the same problem arises?

Yes, I ran the code several times and this problem came up every time
@ShiMinghao0208 I create an environment following the guidance and train FCENet. It work well and doesn't raise the error you mentioned. Can you check if the same problem arises when train other models? If so, i think it is caused by environment.
Maybe it is caused by some bad data? Try to use binary search to locate
@ShiMinghao0208 Have you solved the problem yet? I met the same situation
> @ShiMinghao0208 Have you solved the problem yet? I met the same situation

The guidance have an error, the mmcv is cpu version.When I install the gpu-mmcv,the error disappeared.
> > @ShiMinghao0208 Have you solved the problem yet? I met the same situation
> 
> The guidance have an error, the mmcv is cpu version.When I install the gpu-mmcv,the error disappeared.

I will have a try, thank you very much.
@ShiMinghao0208 could you please share the location of improper guidance? We will fix it asap 
> @ShiMinghao0208 could you please share the location of improper guidance? We will fix it asap

I dont know the location.I asked others for help, and he said that was the reason
Thanks for the feedback!
I am having this exact same problem. I installed the MMCV that matches my Torch/CUDA.  What's even more peculiar is that I only get this error when training on certain datasets.  For others, it appears to begin training properly. 
@mattlee2 It would be great if the error could be reproduced.

it might be due to some imgs in the dataset triggered some corner cases in the code. so

> Try to use binary search to locate the bad data

I had the same problem. When I use fcenet + toy_dataset(or icdar2015_dataset), I have this problem. When I use other networks like drrg + toy_dataset, I don't have this problem.
Exactly the same problem. I have tried training fcenet+icdar2015 and fcenet+ctw1500, raising the same error. Other networks work fine for me.
@Zyq-scut Would you like to take a look?
> @Zyq-scut Would you like to take a look?

OK
@iywo Can you please provide your environment info? I can't reproduce this error in my environment.
I just followed the intallation guide https://mmocr.readthedocs.io/en/latest/install.html 
![00](https://user-images.githubusercontent.com/7261834/123569248-91a06000-d7f8-11eb-8dc0-70de4d7a2bfb.png)
![01](https://user-images.githubusercontent.com/7261834/123569256-982ed780-d7f8-11eb-819a-7ce07373f89a.png)
![02](https://user-images.githubusercontent.com/7261834/123569265-9b29c800-d7f8-11eb-97ab-ba4d323d55c0.png)

 GPU: NVIDIA-SMI 418.113      Driver Version: 418.113      CUDA Version: 10.1  
The first run of fcenet+icdar2015

![t1](https://user-images.githubusercontent.com/7261834/123572618-17270e80-d7ff-11eb-86ed-afd473374be9.png)
![t2](https://user-images.githubusercontent.com/7261834/123572623-18f0d200-d7ff-11eb-9f5e-bfab5f19413f.png)
![t3](https://user-images.githubusercontent.com/7261834/123572629-1bebc280-d7ff-11eb-8854-eb9d61c2f42b.png)
![t4](https://user-images.githubusercontent.com/7261834/123572636-1db58600-d7ff-11eb-933c-1fd998746679.png)
![t5](https://user-images.githubusercontent.com/7261834/123572641-2017e000-d7ff-11eb-850d-e7c32bd946c8.png)
![t6](https://user-images.githubusercontent.com/7261834/123572649-227a3a00-d7ff-11eb-8026-63cf53895026.png)



I also encountered the same error.
I tested it with `pytorch==1.5.0` and got the same error. However, `pytorch >= 1.6.0` works. Will update the documentation soon. 
Thanks!




------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "open-mmlab/mmocr"                                                                                    ***@***.***&gt;;
发送时间:&nbsp;2021年6月29日(星期二) 下午2:22
***@***.***&gt;;
***@***.******@***.***&gt;;
主题:&nbsp;Re: [open-mmlab/mmocr] When I want to train a fcenet, I met a problem (#271)





 
I tested it with pytorch==1.5.0 and got the same error. However, pytorch &gt;= 1.6.0 works. Will update the documentation soon.
 
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

Resolving patch:
diff --git a/mmocr/models/textdet/losses/fce_loss.py b/mmocr/models/textdet/losses/fce_loss.py
--- a/mmocr/models/textdet/losses/fce_loss.py
+++ b/mmocr/models/textdet/losses/fce_loss.py
@@ -120,6 +120,7 @@ def forward_single(self, pred, gt):
         return loss_tr, loss_tcl, loss_reg_x, loss_reg_y
 
     def ohem(self, predict, target, train_mask):
+        device = train_mask.device
         pos = (target * train_mask).bool()
         neg = ((1 - target) * train_mask).bool()
 
@@ -134,7 +135,7 @@ def ohem(self, predict, target, train_mask):
                 int(neg.float().sum().item()),
                 int(self.ohem_ratio * n_pos.float()))
         else:
-            loss_pos = torch.tensor(0.)
+            loss_pos = torch.tensor(0.).to(device)
             loss_neg = F.cross_entropy(
                 predict[neg], target[neg], reduction='none')
             n_neg = 100


Test patch: