ZeroDivisionError: float division by zero (RetinaNet on custom dataset)

Egorundel: <h1><a name="intro-1" class="anchor" href="#intro-1"></a>Intro</h1>
<p>Hi, I want to train RetinaNet PyTorch on a custom dataset in coco format (<a href="https://github.com/pytorch/vision/tree/main/references/detection" rel="noopener nofollow ugc">https://github.com/pytorch/vision/tree/main/references/detection</a>).</p>
<h1><a name="structure-2" class="anchor" href="#structure-2"></a>Structure</h1>
<p>I have the following folder and file structure:</p>
<h2><a name="dataset-folder-3" class="anchor" href="#dataset-folder-3"></a>DATASET FOLDER:</h2>
<p><strong>annotations</strong> folder, <strong>train</strong> folder, <strong>val</strong> folder</p>
<h2><a name="annotations-folder-4" class="anchor" href="#annotations-folder-4"></a><strong>annotations</strong> folder:</h2>
<p>instances_train.json, instances_val.json</p>
<h2><a name="train-folder-5" class="anchor" href="#train-folder-5"></a><strong>train</strong> folder:</h2>
<p><strong>images</strong> folder, <strong>labels</strong> folder</p>
<h3><a name="images-folder-6" class="anchor" href="#images-folder-6"></a><strong>images</strong> folder:</h3>
<p>1.png<br>
3.png<br>
7d938bce-cda0-4814-b938-6d0467f51afb_50.png<br>
7d938bce-cda0-4814-b938-6d0467f51afb_59.png<br>
…</p>
<h3><a name="labels-folder-7" class="anchor" href="#labels-folder-7"></a><strong>labels</strong> folder:</h3>
<p>1.txt<br>
3.txt<br>
7d938bce-cda0-4814-b938-6d0467f51afb_50.txt<br>
7d938bce-cda0-4814-b938-6d0467f51afb_59.txt<br>
…</p>
<h2><a name="val-folder-8" class="anchor" href="#val-folder-8"></a><strong>val</strong> folder:</h2>
<p><strong>images</strong> folder, <strong>labels</strong> folder</p>
<h3><a name="images-folder-9" class="anchor" href="#images-folder-9"></a><strong>images</strong> folder:</h3>
<p>15.png<br>
24.png<br>
7d93fb50.png<br>
7d9359.png<br>
…</p>
<h3><a name="labels-folder-10" class="anchor" href="#labels-folder-10"></a><strong>labels</strong> folder:</h3>
<p>15.txt<br>
24.txt<br>
7d93fb50.txt<br>
7d9359.txt<br>
…</p>
<h1><a name="command-11" class="anchor" href="#command-11"></a>Command</h1>
<p>When I run the following command:<br>
<code>torchrun --nproc_per_node=1 train.py --dataset coco --model retinanet_resnet50_fpn --epochs 3 --lr-steps 16 22 --aspect-ratio-group-factor 3 --lr 0.01 --weights-backbone ResNet50_Weights.IMAGENET1K_V1</code></p>
<h1><a name="problem-12" class="anchor" href="#problem-12"></a>Problem</h1>
<p>I am facing this problem: <code>ZeroDivisionError: float division by zero</code></p>
<pre><code class="lang-auto">(retinaNet) egorundel@egorundel-B560M-H:~/projects/vision/references/detection$ torchrun --nproc_per_node=1 train.py --dataset coco --model retinanet_resnet50_fpn --epochs 3 --lr-steps 16 22 --aspect-ratio-group-factor 3 --lr 0.01 --weights-backbone ResNet50_Weights.IMAGENET1K_V1
| distributed init (rank 0): env://
Namespace(amp=False, aspect_ratio_group_factor=3, backend='pil', batch_size=2, data_augmentation='hflip', data_path='/home/egorundel/data/NAMI_data_coco_without_subfolders/', dataset='coco', device='cuda', dist_backend='nccl', dist_url='env://', distributed=True, epochs=3, gpu=0, lr=0.01, lr_gamma=0.1, lr_scheduler='multisteplr', lr_step_size=8, lr_steps=[16, 22], model='retinanet_resnet50_fpn', momentum=0.9, norm_weight_decay=None, opt='sgd', output_dir='.', print_freq=20, rank=0, resume='', rpn_score_thresh=None, start_epoch=0, sync_bn=False, test_only=False, trainable_backbone_layers=None, use_copypaste=False, use_deterministic_algorithms=False, use_v2=False, weight_decay=0.0001, weights=None, weights_backbone='ResNet50_Weights.IMAGENET1K_V1', workers=4, world_size=1)
Loading data
loading annotations into memory...
Done (t=0.97s)
creating index...
index created!
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
Creating data loaders
Using [0, 0.5, 0.6299605249474366, 0.7937005259840997, 1.0, 1.2599210498948732, 1.5874010519681994, 2.0, inf] as bins for aspect ratio quantization
Count of instances per bin: []
Creating model
Start training
Traceback (most recent call last):
  File "train.py", line 334, in &lt;module&gt;
    main(args)
  File "train.py", line 309, in main
    train_one_epoch(model, optimizer, data_loader, device, epoch, args.print_freq, scaler)
  File "/home/egorundel/projects/vision/references/detection/engine.py", line 27, in train_one_epoch
    for images, targets in metric_logger.log_every(data_loader, print_freq, header):
  File "/home/egorundel/projects/vision/references/detection/utils.py", line 200, in log_every
    print(f"{header} Total time: {total_time_str} ({total_time / len(iterable):.4f} s / it)")
ZeroDivisionError: float division by zero
[2023-11-07 16:21:15,565] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 30250) of binary: /home/egorundel/venvs/retinaNet/bin/python
Traceback (most recent call last):
  File "/home/egorundel/venvs/retinaNet/bin/torchrun", line 33, in &lt;module&gt;
    sys.exit(load_entry_point('torch', 'console_scripts', 'torchrun')())
  File "/home/egorundel/venvs/retinaNet/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 351, in wrapper
    return f(*args, **kwargs)
  File "/home/egorundel/venvs/retinaNet/lib/python3.8/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/home/egorundel/venvs/retinaNet/lib/python3.8/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/home/egorundel/venvs/retinaNet/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/egorundel/venvs/retinaNet/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train.py FAILED
------------------------------------------------------------
Failures:
  &lt;NO_OTHER_FAILURES&gt;
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-11-07_16:21:15
  host      : egorundel-B560M-H
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 30250)
  error_file: &lt;N/A&gt;
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
</code></pre>
<h1><a name="system-13" class="anchor" href="#system-13"></a>System</h1>
<p>OS: Ubuntu 20.04<br>
GPU: RTX 3060<br>
<code>torch</code>: 2.2.0.dev20231107+cu118<br>
<code>torchvision</code>: 0.17.0.dev20231107+cu118<br>
<code>torchaudio</code>: 2.2.0.dev20231107+cu118</p>
<h1><a name="question-14" class="anchor" href="#question-14"></a>Question</h1>
<p>What could be the problem?</p>

Egorundel: <p>In the source codes, I made the appropriate changes for my custom dataset:</p>
<p>I have 13 classes in my dataset, so I corrected 91 to 13 here.<br>
<strong>vision/references/detection/train.py</strong>:</p>
<pre><code class="lang-python">...
def get_dataset(is_train, args):
    image_set = "train" if is_train else "val"
    num_classes, mode = {"coco": (13, "instances"), "coco_kp": (2, "person_keypoints")}[args.dataset]
    with_masks = "mask" in args.model
    ds = get_coco(
        root=args.data_path,
        image_set=image_set,
        transforms=get_transform(is_train, args),
        mode=mode,
        use_v2=args.use_v2,
        with_masks=with_masks,
    )
    return ds, num_classes
...
</code></pre>
<p>I also changed <strong>train2017</strong> and <strong>val2017</strong> to <strong>train</strong> and <strong>val</strong>.</p>
<p><strong>vision/references/detection/coco_utils.py</strong>:</p>
<pre><code class="lang-python">...
def get_coco(root, image_set, transforms, mode="instances", use_v2=False, with_masks=False):
    anno_file_template = "{}_{}.json"
    PATHS = {
        "train": ("train", os.path.join("annotations", anno_file_template.format(mode, "train"))),
        "val": ("val", os.path.join("annotations", anno_file_template.format(mode, "val"))),
        # "train": ("val", os.path.join("annotations", anno_file_template.format(mode, "val")))
    }
...
</code></pre>
<p>But the error still remained.</p>