Debugging DataParallel, no speedup and uneven memory allocation

mattmacy: With one GPU and a batch size of 14 an epoch on my data set takes about 24 minutes. With 2 GPUs and a batch size of 28 it’s still taking 24 minutes per epoch. Any suggestions on what might be going wrong? Does the batch normalization layer try to normalize across both GPUs and thus add large amounts of extra memory traffic? Please say it doesn’t. Thanks. Top shows 2 CPUs saturated: <SCODE>Tasks: 255 total,   1 running, 254 sleeping,   0 stopped,   0 zombie
%Cpu(s): 16.3 us,  2.5 sy,  0.1 ni, 81.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 65885928 total, 40001592 free, 11878640 used, 14005696 buff/cache
KiB Swap: 67017724 total, 67017724 free,        0 used. 52840116 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                           
 4622 mmacy     20   0 49.225g 5.809g 977604 S 200.0  9.2 111:32.30 work/vnet.base.
<ECODE> The memory allocation on the two GPUs is also uneven. If they’re both doing the same operations with the same batch size, why is GPU1 using 1/3rd more memory than GPU0? <SCODE>+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.57 Driver Version: 367.57 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 TITAN X (Pascal) Off | 0000:01:00.0 On | N/A |
| 51% 82C P2 74W / 250W | 7906MiB / 12186MiB | 99% Default |
+-------------------------------+----------------------+----------------------+
| 1 TITAN X (Pascal) Off | 0000:02:00.0 Off | N/A |
| 47% 78C P2 107W / 250W | 10326MiB / 12189MiB | 95% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1086 G /usr/lib/xorg/Xorg 105MiB |
| 0 8469 C work/vnet.base.20170316_0434 7797MiB |
| 1 8469 C work/vnet.base.20170316_0434 10323MiB |
+-----------------------------------------------------------------------------+

<ECODE> I also see that parallel_apply in data_parallel relies on python threading, which isn’t very worthwhile given how much of the code has to run under the GIL. The only way to get any sort of reasonable parallelism while using regular python GIL protected code is to run separate python processes. Are other people actually seeing a speedup from DataParallel? I think I’m probably only seeing one thread make progress at a time.

apaszke: No, it doesn’t. I can’t understand what’s the problem. Are you expecting a larger speedup? You’ve doubled the amount of computing power along with the input size, so the time staying constant is a great result. Maybe you have some conditional branching in your model? our multi-GPU code is nearly as fast as it can be. You still need the GIL to execute most of the autograd operations so it doesn’t matter if we don’t use Python threads. In our CNN benchmarks we have perf very similar to that of Caffe2, and faster than Lua Torch, which has very good multi-GPU implementation.

mattmacy: Usually “data parallel” means data operations run in parallel, but here data parallel only means that the forward passes, the fast part, have any parallel component. That’s just not very useful - as larger batches can reduce the convergence rate. So I’m using twice the power, generating twice the heat and am getting no real benefit. The use of python makes parallelism unduly hard but one could run the autograd pass in separate processes just like hogwild, but without sharing the model weights by having each process broadcast the updates from the cost function’s backward pass. Each process would average in the updates from other GPUs.

apaszke: Yes, and the operations run in parallel indeed. Both in forward and in backward. I’m not sure how is that not useful and what’s the problem. Data parallelism is a well known term used to describe weak scaling of these models, and this is exactly what happens in here. It’s very useful, because in some cases you need batch sizes so large, that they don’t fit on a single GPU. The behaviour is exactly what’s wanted in a lot of situations. It doesn’t make it hard, I really don’t understand what’s bothering you. You can do hogwild training, even on GPUs if you wan. No one is stopping you. We even have a hogwild example in the examples repo.

smth: as adam pointed out, DataParallel is very very well defined in Deep Learning. We split the mini-batches over multiple GPUs, and accumulate the gradients at the end from all the GPUs before doing the optimization step. Getting a linear speedup when doubling the batch size is the best case scenario, and you are hitting that. If there was a confusion in terminology, i hope it’s clarified now.

ngimel: 

smth: @ngimel in terms of weak-scaling, this seems appropriate right? We cannot expect any better speedup than this theorerically.

ngimel: 

smth: 

mattmacy: Thank you for your time.

ngimel: DataParallel also distributes backward pass, it is hidden in autograd. DataParallel has to broadcast and reduce all the parameters, so parallelization efficiency decreases when you computation time is small and you have a lot of parameters.

smth: in the backward pass of DataParallel, we reduce the weights from GPU2 onto GPU1. Our DataParallel algorithm is roughly like this: in forward: scatter mini-batch to GPU1, GPU2 replicate model on GPU2 (it is already on GPU1) model_gpu1(input_gpu1), model_gpu2(input_gpu2) (this step is parallel_apply) gather output mini-batch from GPU1, GPU2 onto GPU1 in backward: scatter grad_output and input parallel_apply model’s backward pass reduce GPU2 replica’s gradients onto GPU1 model 
Now there is only a single model again with accumulated gradients from GPU1 and GPU2
 gather the grad_input Maybe your situation is similar?

mattmacy: Is it necessary to replicate all of the gradients? Couldn’t you just replicate the output of the backward pass of just the loss function and then average the results? There are no fully connected layers. I guess 3d convolutions with >= 128 channels have an exorbitant number of parameters. I may have made an error in going up to 512 when I only meant to go up to 256, so at least you’ve prompted me to take a closer look at the model. Thanks again.

mattmacy: Thanks.

mattmacy: 

smth: sweet, that’s great news.

FuriouslyCurious: Soumith and Adam, I am having a great time exploring PyTorch!  Thanks for the awesome library. I am trying to saturate a 64-core/256-thread CPU in addition to the GPUs.  Any pointers on how I can extend Data_parallel.py to create 3 scatters on GPU0, GPU1, and CPU(0-255)?

apaszke: 

RicherMans: So for DNN’s there is no speedup expected, if the model parameters are large?

bottanski: <SCODE>+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 980 Ti  Off  | 0000:06:00.0     Off |                  N/A |
|  0%   51C    P2   167W / 300W |   6070MiB /  6078MiB |     73%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 980 Ti  Off  | 0000:07:00.0      On |                  N/A |
|  0%   45C    P2    93W / 300W |   1021MiB /  6075MiB |     35%      Default |
+-------------------------------+----------------------+----------------------+
<ECODE> I have tried putting the linear layers outside of DataParallel to no avail - machine ran out of memory. I understand that the computations on GPU 1 require more memory than those on GPU 2, but I was expecting more memory to be used on GPU 2. Am I at the maximum capacity / performance? Is there anything I can do with this seq-to-seq model to train more batches at a time or shorten training time.