Multi-GPU training on COCO

Meisam_Hasani: <p>I am trying to train the coco dataset on a 3-gpu system. I put model in <code>torch.nn.DataParallel</code> but the dataloader can not spread the data between the gpus and it uses only the first gpu!</p>
<p>When I load all data into a python list and then in the epochs I read them from that python array, all 3 gpus work fine! It seems to be a problem of dataloader (or cocodetection class) but I don’t have a clue.</p>
<p>What am I missing here for using all the gpus?</p>

ptrblck: <p><code>nn.DataParallel</code> will clone the model onto al specified devices, split the input batch in <code>dim0</code>, and send each chunk to the corresponding device. It won’t do anything with your data loading. A good description is given in <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">this blogpost</a>. If you want to use load a subset of the data for each device, use <code>DistributedDataParallel</code> with the <code>DistributedSampler</code>, which would also yield a larger speedup.</p>

Meisam_Hasani: <p>Thank you ptrblck. Spliting the batch and sending them to multiple devices is all that I need, but it doesn’t do that. In each epoch I have a loop as below:</p>
<pre><code class="lang-auto">for images, targets in train_loader:	
        images = list(torchvision.transforms.ToTensor()(image).to(device) for image in images)
        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
        loss_dict = model(images, targets)
        ...
</code></pre>
<p>After loading each batch I send them to cuda devices, but it utilizes only the first gpu. If instead of train_loader I use an array (already filled with the data), this code works with all gpus.</p>

ptrblck: <p>Could you post a minimal, executable code snippet using random input tensors, which would show this behavior, please?</p>

Meisam_Hasani: <p>Sorry I don’t know how to reproduce it with random tensors, but this is exactly what I did (it’s just the training part):<br>
(It won’t give any error and works smoothly but with just one gpu)</p>
<pre><code class="lang-auto">device = torch.device("cuda")
device_ids = [i for i in range(torch.cuda.device_count())]

model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask,
													256,
													num_classes)	
params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)


model=  torch.nn.DataParallel(model, device_ids=[0, 1, 2])
model.to(device)
model =	model.cuda()

train_data = get_coco(path_to_data, "train", None)  

train_loader = torch.utils.data.DataLoader( train_data 
												,batch_size=batch_size
												,num_workers=num_workers
												,collate_fn=collate_fn
											)
											
											
for epoch in range(num_epochs):
		
		model.train()	
		bj = 0	

		for images, targets in train_loader:	

			images = list(torchvision.transforms.ToTensor()(image).to(device) for image in images)
			
			targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
			
			loss_dict = model(images, targets)
			
			losses = sum(loss for loss in loss_dict.values())
		
			# reduce losses over all GPUs for logging purposes
			loss_dict_reduced = utils.reduce_dict(loss_dict)
			losses_reduced = sum(loss for loss in loss_dict_reduced.values())
		
			loss_value = losses_reduced.item()
		
			if not math.isfinite(loss_value):
				print("Loss is {}, stopping training".format(loss_value))
				print(loss_dict_reduced)
				sys.exit(1)
		
			optimizer.zero_grad()
			losses.backward()
			optimizer.step()
</code></pre>
<p>Thanks.</p>

ptrblck: <p>Thanks for the code. I don’t see any obvious issues and cannot reproduce the issue in my setup as I see 3 GPUs used:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A100-SXM...  On   | 00000000:07:00.0 Off |                    0 |
| N/A   37C    P0    95W / 400W |   6416MiB / 40536MiB |     30%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA A100-SXM...  On   | 00000000:0F:00.0 Off |                    0 |
| N/A   36C    P0    83W / 400W |   6712MiB / 40536MiB |     31%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA A100-SXM...  On   | 00000000:47:00.0 Off |                    0 |
| N/A   34C    P0    85W / 400W |   6568MiB / 40536MiB |     34%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
</code></pre>

Meisam_Hasani: <p>One thing is strange for me that, the code gets all gpus engaged, because their memories are a little filled up, but not that much that could work!<br>
When I train with one gpu I set batch_size=2 and for 3gpus I should set batch_size=6, but if I set that as 6, the code gives me gpu OOM error which shows that only one gpu is engaged with the data.</p>
<p>[I don’t know if that is related or not but I am using docker container to run the code! with` --gpus all` of course]</p>
<p>Here is the output of my smi:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33fdbe35a517fc26867a6402412b1594a803417f.png" data-download-href="https://discuss.pytorch.org/uploads/default/33fdbe35a517fc26867a6402412b1594a803417f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33fdbe35a517fc26867a6402412b1594a803417f.png" alt="image" data-base62-sha1="7pVZyCbgAwaNN1pZzpg6bU8QHpB" width="579" height="499" data-dominant-color="484848"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">642×554 52.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>