Multiple GPU use significant first GPU memory consumption

username: che85
cooked: <p>Hi there,</p>
<p>I am training on two GPUs and I noticed that the first GPU uses significantly more memory. How can that be? I moved the model to both GPUs with DataParallel.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c98720e85c89aef7ebc756223f19e4267c771a4d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c98720e85c89aef7ebc756223f19e4267c771a4d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c98720e85c89aef7ebc756223f19e4267c771a4d_2_690x199.jpeg" alt="image" data-base62-sha1="sKNtXakNRvDcn3z69DuuemxTno9" width="690" height="199" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c98720e85c89aef7ebc756223f19e4267c771a4d_2_690x199.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c98720e85c89aef7ebc756223f19e4267c771a4d_2_1035x298.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c98720e85c89aef7ebc756223f19e4267c771a4d_2_1380x398.jpeg 2x" data-dominant-color="141414"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1456×420 188 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks for your help.</p>

username: JuanFMontesinos
cooked: <p>It’s about the optimizer you are using (I guess) and probably because you may be preallocating the whole batch in gpu 0.</p>
<p>DataParallel has a kind of “main gpu”. Even if the whole model is copied in each gpu, thing such us optimizer parameters remain in the main one. Let me know if using plain SGD you still observing such a big difference.</p>

username: che85
cooked: <p>SGD consumes the same amount.</p>

username: JuanFMontesinos
cooked: <p>Hi,<br>
are you loading a pretrained model? Loading those weights may occupy some memory if you load them directly on gpu.<br>
Be aware output for both gpus are gathered back in the main one.</p>

username: che85
cooked: <p>Hi Juan,</p>
<p>that might be the case yes (it’s a “small” pretrained autoencoder model I am loading next to the one I want to train). Are there any best practices/examples how to evenly use GPU memory when using more than 1 GPU?</p>
<p>I also had situations where output was on one GPU but target on the other one.</p>
<p>When building the model, is there any good example to avoid those kind of situations?</p>

username: che85
cooked: <p>I might have the behavior explained in <a href="https://github.com/pytorch/pytorch/issues/8637" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/8637</a></p>
<p>Is there any resource that explains bad and good model design?</p>

username: JuanFMontesinos
cooked: <p>There is an issue in which if you load a state_dict, it maps to gpu. Once loaded it’s never fred.</p>
<p>This means that when you do:<br>
<code>model.load_state_dict(torch.load(path_to_weights))</code> those weights are using gpu memory never fred.<br>
Try this way.<br>
<code>state_dict = torch.load(directory, map_location=lambda storage, loc: storage) model.load_state_dict(state_dict)</code><br>
it will force state dict to be loaded on RAM instead of gpu directly.</p>
<p>Regarding the issue you mentioned, it’s not very typical.<br>
In general, dataparallel duplicates everything in each gpu, thus, there shouldn’t be higher memory comsuption.</p>
<p>Extra compsumtion usually comes from one of these 3 issues:<br>
input/output tensor are gathered on main gpu (which should denote an slightly higher mem comp)<br>
optimizer memory requirments<br>
and state_dict issues.</p>