Repository: open-mmlab/mmdetection

Issue:
assert len(indices) == self.total_size error during multiple GPU training
I am trying to train my dataset on 8 GPU's. However, after calling `./dist_train.sh` this error assertion appeares:

Traceback (most recent call last):                
  File "./tools/train.py", line 113, in <module>  
    main()               
  File "./tools/train.py", line 109, in main      
    logger=logger)       
  File "/mmdetection/mmdet/apis/train.py", line 58, in train_detector                                
    _dist_train(model, dataset, cfg, validate=validate)                                              
  File "/mmdetection/mmdet/apis/train.py", line 186, in _dist_train                                  
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)                                         
  File "/opt/conda/lib/python3.6/site-packages/mmcv/runner/runner.py", line 358, in run
    epoch_runner(data_loaders[i], **kwargs)       
  File "/opt/conda/lib/python3.6/site-packages/mmcv/runner/runner.py", line 260, in train
    for i, data_batch in enumerate(data_loader):  
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 193, in __iter__    return _DataLoaderIter(self)                  
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 493, in __init__
    self._put_indices()  
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 591, in _put_indices
    indices = next(self.sample_iter, None)        
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 172, in __iter__
    for idx in self.sampler:                      
  File "/mmdetection/mmdet/datasets/loader/sampler.py", line 138, in __iter__                        
    assert len(indices) == self.total_size        
...

in the config I tried various values for `imgs_per_gpu` and `workers_per_gpu`, currently it is:
`imgs_per_gpu=2,
workers_per_gpu=2,`
no settings was working though. Single-GPU training works well.

What is the meaning of this assert?
Thanks!


Discussion:
Please follow the `Error report` issue template.
> Please follow the `Error report` issue template.

Here it is, thanks for any help!

**Checklist**
1. I have searched related issues but cannot get the expected help.
yes
2. The bug has not been fixed in the latest version.
yup

**Describe the bug**
I am trying to train my custom dataset on 8 GPU's. However, after calling ./dist_train.sh the error showed below appeares. In the config I tried more values for `imgs_per_gpu` and `workers_per_gpu` (e.g. `imgs_per_gpu=2`, `workers_per_gpu=2`), no settings was working though.

Single-GPU training works well.

What is the meaning of the assert in the Traceback? What does not fit? Thanks!

**Reproduction**
1. What command or script did you run?
```
./tools/dist_train.sh MY_CONFIG 8 --validate
```
2. Did you make any modifications on the code or config? Did you understand what you have modified?
I modified number of classes, workers_per_gpu, imgs_per_gpu, dataset type and paths to the datasets. No changes in code.
3. What dataset did you use?
My own dataset of 8 classes converted to COCO format.

**Environment**
 - OS: Ubuntu 18.04
 - GCC 5.4.0
 - PyTorch version 1.1.0
-  I built the docker I use on the official pytorch+cuda docker
- GPU model: 8xV100
- CUDA version: 10.0, CUDNN version: 7.5

**Error traceback**
```
Traceback (most recent call last):
File "./tools/train.py", line 113, in 
main()
File "./tools/train.py", line 109, in main
logger=logger)
File "/mmdetection/mmdet/apis/train.py", line 58, in train_detector
_dist_train(model, dataset, cfg, validate=validate)
File "/mmdetection/mmdet/apis/train.py", line 186, in _dist_train
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/opt/conda/lib/python3.6/site-packages/mmcv/runner/runner.py", line 358, in run
epoch_runner(data_loaders[i], **kwargs)
File "/opt/conda/lib/python3.6/site-packages/mmcv/runner/runner.py", line 260, in train
for i, data_batch in enumerate(data_loader):
File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 193, in iter return _DataLoaderIter(self)
File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 493, in init
self._put_indices()
File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 591, in _put_indices
indices = next(self.sample_iter, None)
File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 172, in iter
for idx in self.sampler:
File "/mmdetection/mmdet/datasets/loader/sampler.py", line 138, in iter
assert len(indices) == self.total_size
```
One more detail, when I print `len(indices)` and `self.total_size` right before the critical assert, it's `9308` and `9312`. The size of my training dataset is `9306`..
@hellock Any ideas? Recently I found out that when I set the config file to train with 2 GPUs (2 img/gpu), the training (called with `./dist_train`) initiates well. However, training with any more GPUs results in the mentioned assert error. Seems like a bug to me.
You may count the images with aspect ratio >1 and <1. I suspect that there are only 2 images for one of the two groups.
> You may count the images with aspect ratio >1 and <1. I suspect that there are only 2 images for one of the two groups.

you are right, there are exactly 2 images with height > width
The problem lies in [this line](https://github.com/open-mmlab/mmdetection/blob/master/mmdet/datasets/loader/sampler.py#L135) where `len(indices) < extra`.
I meet the same issue, how to fix it?
@ZhexuanZhou 
Before bug fixed, simplest way is make your img_per_gpu as power of 2.
e.g. 2, 4, 8, 16 ...
Also, make your gpu number as power of 2.
This works for me.
> @ZhexuanZhou
> Before bug fixed, simplest way is make your img_per_gpu as power of 2.
> e.g. 2, 4, 8, 16 ...
> Also, make your gpu number as power of 2.
> This works for me.

This fix doesn't work for me, as is already mentioned in the bug description

> **Describe the bug**
> I am trying to train my custom dataset on 8 GPU's. However, after calling ./dist_train.sh the error showed below appeares. In the config I tried more values for `imgs_per_gpu` and `workers_per_gpu` (e.g. `imgs_per_gpu=2`, `workers_per_gpu=2`), no settings was working though.

The easiest work-around for me was to comment out the two asserts in the `sampler.py` :)
@FilipLangr 
Yeah, that might cause some samples lost but not that harmful. 
Hello!

I have situation, when 6 pics, where w>h.
33994 pics, when h>w.
And 2 pics, where h==w.

I have deleted h==w pics and have AssertionError: assert len(indices) == self.total_size anyway :(
Than I deleted w>h pics and get another error: TypeError: 'NoneType' object is not subscriptable

Resolving patch:
diff --git a/mmdet/datasets/loader/sampler.py b/mmdet/datasets/loader/sampler.py
--- a/mmdet/datasets/loader/sampler.py
+++ b/mmdet/datasets/loader/sampler.py
@@ -132,8 +132,12 @@ def __iter__(self):
                     math.ceil(
                         size * 1.0 / self.samples_per_gpu / self.num_replicas)
                 ) * self.samples_per_gpu * self.num_replicas - len(indice)
-                indice += indice[:extra]
-                indices += indice
+                # pad indice
+                tmp = indice.copy()
+                for _ in range(extra // size):
+                    indice.extend(tmp)
+                indice.extend(tmp[:extra % size])
+                indices.extend(indice)
 
         assert len(indices) == self.total_size
 


Test patch: