Training faster-rcnn on multiple gpus on single node

username: smadan
cooked: <p>I am getting started with torch, and trying to understand the object detection support in it using <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>
<p>I am having trouble paralleilizing the faster-rcnn model. I get the following error when running train_one_epoch in engine.py,</p>
<p>RuntimeError: chunk expects at least a 1-dimensional tensor (chunk at /pytorch/aten/src/ATen/native/TensorShape.cpp:188)</p>
<p>What am I doing wrong?</p>
<p>I parallelized using the following,</p>
<p>backbone = torchvision.models.resnet50(pretrained=True)   # commented out torch.flatten<br>
backbone.avgpool = torch.nn.Identity()<br>
backbone.fc = torch.nn.Identity()<br>
backbone.out_channels = 2048</p>
<p>model = FasterRCNN(backbone=backbone,<br>
rpn_anchor_generator=anchor_generator,<br>
box_roi_pool=roi_align,<br>
num_classes=81)</p>
<p>device = torch.device(“cuda”) if torch.cuda.is_available() else torch.device(“cpu”)<br>
model.to(device)<br>
parallel_model = torch.nn.DataParallel(model)  # need to first freeze batch norm</p>
<p>Debugging with pdb seems to suggest that it is trying to scatter “image_id” in the target dictionary. I thought I was applying torch.nn.DataParallel in the wrong place, so looked at <a href="https://github.com/pytorch/vision/blob/master/references/detection/train.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/references/detection/train.py</a>, but it seems fine.</p>
<p>I also unsuccessfully tried using torch.nn.parallel.DistributedDataParallel in the following way,<br>
torch.distributed.init_process_group(backend=‘nccl’, world_size=1, rank=0)<br>
parallel_model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[0])<br>
python3 -m torch.distributed.launch --nproc_per_node=1 --use_env train_faster_rcnn.py</p>

username: ptrblck
cooked: <p>Could you post the shapes of the data you are passing to the model after wrapping it in <code>nn.DataParallel</code>?</p>

username: smadan
cooked: <p>On line 27 in “train_one_epoch” in “engine.py”, just after,</p>
<p>“for images, targets in metric_logger.log_every(data_loader, print_freq, header):”</p>
<p>(Pdb) p type(images)<br>
&lt;class ‘tuple’&gt;<br>
(Pdb) p type(targets)<br>
&lt;class ‘tuple’&gt;<br>
(Pdb) p images[0].shape<br>
torch.Size([3, 640, 426])<br>
(Pdb) p images[1].shape<br>
torch.Size([3, 427, 640])<br>
(Pdb) p type(targets[0])<br>
&lt;class ‘dict’&gt;<br>
(Pdb) p targets[0].keys()<br>
dict_keys([‘boxes’, ‘labels’, ‘image_id’, ‘area’, ‘iscrowd’])<br>
(Pdb) p targets[0][‘boxes’]<br>
tensor([[  2.8600,   0.0000, 426.0000, 640.0000],<br>
[122.5200,   2.1600, 412.2500, 395.6800],<br>
[113.8600, 346.6700, 225.6200, 386.8300]])</p>

username: ptrblck
cooked: <p>I’m not sure, if <code>nn.DataParallel</code> works with tuples (of tensors or dicts).<br>
CC <a class="mention" href="/u/fmassa">@fmassa</a> who might know, how to properly use data parallel for segmentation models.</p>

username: smadan
cooked: <p>Thanks. Not sure how to CC.</p>
<p>In <a href="https://github.com/pytorch/vision/blob/master/references/detection/train.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/references/detection/train.py</a>, collate_fn for torch.utils.data.DataLoader seems to be the one in “utils.py”, which returns a tuple. The model in that file gets paralleized with torch.nn.parallel.DistributedDataParallel (I understand its different from torch.nn.DataParallel), and trained with “train_one_epoch”. I tried torch.nn.parallel.DistributedDataParallel in the way mentioned in the last part of the posted question, but was unsuccessful</p>

username: ptrblck
cooked: <p>Ah OK. What kind of error did you get using DDP?</p>

username: smadan
cooked: <p>Complete error with torch.nn.DataParallel. I replaced some directory names with “HOME” and “PATH_TO_SCRIPT”.</p>
<p>RuntimeError: chunk expects at least a 1-dimensional tensor (chunk at /pytorch/aten/src/ATen/native/TensorShape.cpp:188)<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x33 (0x7f3c89d6f273 in HOME/.local/lib/python3.6/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span> at::native::chunk(at::Tensor const&amp;, long, long) + 0x2ff (0x7f3c1a3ac0cf in HOME/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)<br>
frame <span class="hashtag-raw">#2:</span> at::TypeDefault::chunk(at::Tensor const&amp;, long, long) + 0x9 (0x7f3c1a6d4f99 in HOME/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)<br>
frame <span class="hashtag-raw">#3:</span> torch::autograd::VariableType::chunk(at::Tensor const&amp;, long, long) + 0x29a (0x7f3c1bfbfcca in HOME/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)<br>
frame <span class="hashtag-raw">#4:</span> torch::cuda::scatter(at::Tensor const&amp;, c10::ArrayRef, c10::optional&lt;std::vector&lt;long, std::allocator &gt; &gt; const&amp;, long, c10::optional&lt;std::vector&lt;c10::optional<a>c10::cuda::CUDAStream</a>, std::allocator&lt;c10::optional<a>c10::cuda::CUDAStream</a> &gt; &gt; &gt; const&amp;) + 0x3e1 (0x7f3c1ca938d1 in HOME/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)<br>
frame <span class="hashtag-raw">#5:</span>  + 0x5f3d8f (0x7f3c8afb6d8f in HOME/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#6:</span>  + 0x1d3ef4 (0x7f3c8ab96ef4 in HOME/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#7:</span> /usr/bin/python3.6() [0x50746c]<br>
frame <span class="hashtag-raw">#8:</span> _PyEval_EvalFrameDefault + 0x4dd (0x507edd in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#9:</span> /usr/bin/python3.6() [0x504e80]<br>
frame <span class="hashtag-raw">#10:</span> /usr/bin/python3.6() [0x506ac3]<br>
frame <span class="hashtag-raw">#11:</span> /usr/bin/python3.6() [0x507330]<br>
frame <span class="hashtag-raw">#12:</span> _PyEval_EvalFrameDefault + 0x4dd (0x507edd in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#13:</span> /usr/bin/python3.6() [0x504e80]<br>
frame <span class="hashtag-raw">#14:</span> /usr/bin/python3.6() [0x56cbbb]<br>
frame <span class="hashtag-raw">#15:</span> PyObject_Call + 0x3e (0x59fcee in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#16:</span> THPFunction_apply(_object*, _object*) + 0x9df (0x7f3c8adb791f in HOME/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#17:</span> /usr/bin/python3.6() [0x507217]<br>
frame <span class="hashtag-raw">#18:</span> _PyEval_EvalFrameDefault + 0x4dd (0x507edd in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#19:</span> /usr/bin/python3.6() [0x5057d7]<br>
frame <span class="hashtag-raw">#20:</span> _PyFunction_FastCallDict + 0xe8 (0x4531d8 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#21:</span> _PyObject_FastCallDict + 0x291 (0x59f051 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#22:</span> /usr/bin/python3.6() [0x5c377c]<br>
frame <span class="hashtag-raw">#23:</span> PySequence_Tuple + 0x1fc (0x5a10dc in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#24:</span> _PyEval_EvalFrameDefault + 0x5d41 (0x50d741 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#25:</span> /usr/bin/python3.6() [0x5057d7]<br>
frame <span class="hashtag-raw">#26:</span> _PyFunction_FastCallDict + 0xe8 (0x4531d8 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#27:</span> _PyObject_FastCallDict + 0x291 (0x59f051 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#28:</span> /usr/bin/python3.6() [0x5c377c]<br>
frame <span class="hashtag-raw">#29:</span> PySequence_Tuple + 0x222 (0x5a1102 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#30:</span> _PyEval_EvalFrameDefault + 0x5d41 (0x50d741 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#31:</span> /usr/bin/python3.6() [0x5057d7]<br>
frame <span class="hashtag-raw">#32:</span> _PyFunction_FastCallDict + 0xe8 (0x4531d8 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#33:</span> _PyObject_FastCallDict + 0x291 (0x59f051 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#34:</span> /usr/bin/python3.6() [0x5c377c]<br>
frame <span class="hashtag-raw">#35:</span> PySequence_Tuple + 0x19b (0x5a107b in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#36:</span> _PyEval_EvalFrameDefault + 0x5d41 (0x50d741 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#37:</span> /usr/bin/python3.6() [0x5057d7]<br>
frame <span class="hashtag-raw">#38:</span> _PyFunction_FastCallDict + 0xe8 (0x4531d8 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#39:</span> _PyObject_FastCallDict + 0x291 (0x59f051 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#40:</span> /usr/bin/python3.6() [0x5c377c]<br>
frame <span class="hashtag-raw">#41:</span> PySequence_Tuple + 0x1fc (0x5a10dc in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#42:</span> _PyEval_EvalFrameDefault + 0x5d41 (0x50d741 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#43:</span> /usr/bin/python3.6() [0x5057d7]<br>
frame <span class="hashtag-raw">#44:</span> /usr/bin/python3.6() [0x506ac3]<br>
frame <span class="hashtag-raw">#45:</span> /usr/bin/python3.6() [0x507330]<br>
frame <span class="hashtag-raw">#46:</span> _PyEval_EvalFrameDefault + 0x4dd (0x507edd in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#47:</span> /usr/bin/python3.6() [0x5057d7]<br>
frame <span class="hashtag-raw">#48:</span> /usr/bin/python3.6() [0x506ac3]<br>
frame <span class="hashtag-raw">#49:</span> /usr/bin/python3.6() [0x507330]<br>
frame <span class="hashtag-raw">#50:</span> _PyEval_EvalFrameDefault + 0x4dd (0x507edd in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#51:</span> /usr/bin/python3.6() [0x504e80]<br>
frame <span class="hashtag-raw">#52:</span> /usr/bin/python3.6() [0x506ac3]<br>
frame <span class="hashtag-raw">#53:</span> /usr/bin/python3.6() [0x507330]<br>
frame <span class="hashtag-raw">#54:</span> _PyEval_EvalFrameDefault + 0x1548 (0x508f48 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#55:</span> /usr/bin/python3.6() [0x5064e4]<br>
frame <span class="hashtag-raw">#56:</span> /usr/bin/python3.6() [0x507330]<br>
frame <span class="hashtag-raw">#57:</span> _PyEval_EvalFrameDefault + 0x4dd (0x507edd in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#58:</span> /usr/bin/python3.6() [0x504e80]<br>
frame <span class="hashtag-raw">#59:</span> /usr/bin/python3.6() [0x5b9928]<br>
frame <span class="hashtag-raw">#60:</span> PyObject_Call + 0x3e (0x59fcee in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#61:</span> _PyEval_EvalFrameDefault + 0x1ab3 (0x5094b3 in /usr/bin/python3.6)<br>
frame <span class="hashtag-raw">#62:</span> /usr/bin/python3.6() [0x504e80]<br>
frame <span class="hashtag-raw">#63:</span> /usr/bin/python3.6() [0x5b96f0]</p>
<blockquote>
<p>PATH_TO_SCRIPT/train_faster_rcnn.py(78)()<br>
 → train_one_epoch(parallel_model, optimizer, train_dataloader, device, curr_epoch, 100)</p>
</blockquote>

username: smadan
cooked: <p>The debugger ends in the module “scatter_gather.py” on line 13 in the nested function “def scatter_map(obj)”: return Scatter.apply(target_gpus, None, dim, obj). At this point, obj is “tensor(533958, device=‘cuda:0’)”. It I step up one level, the code is on line 15 in the same nested function “def scatter_map(obj)”: return list(zip(*map(scatter_map, obj))). Here “obj” is (‘image_id’, tensor(533958, device=‘cuda:0’)). “image_id” is a field in the target dictionary, which seems to indicate that the code is trying to scatter the fields in the target dictionary</p>

username: smadan
cooked: <p>My bad. In the target dict, target[“image_id”] needs to  be torch.tensor([idx]). I was setting it to torch.tensor(idx).</p>

username: ptrblck
cooked: <p>Good to hear, you’ve found the bug. Is it working now?</p>

username: smadan
cooked: <p>Still need to thoroughly test it, but it seems to be working with torch.nn.parallel.DistributedDataParallel. Thanks for your help!</p>