Checkpoint in Multi GPU

username: gnadaf
cooked: <p>Is there any difference between, saving checkpoint when training with a single GPU and saving checkpoint with 2 GPUs?</p>
<p>Example: If I use DataParallel to train on 2 GPUs, if I save checkpoint after each epoch, which parameters will be saved? GPU1 info saved or GPU-2 info saved in checkpoint ?</p>
<p>How to check while tranining</p>

username: Janine
cooked: <p>Having same doubts here, any guidance would be greatly appreciated!</p>

username: ptrblck
cooked: <p><code>nn.DataParallel</code> will reduce all parameters to the model on the default device, so you could directly store the <code>model.module.state_dict()</code>.<br>
If you are using <code>DistributedDataParallel</code>, you would have to make sure that only one rank is storing the checkpoint as otherwise multiple process might be writing to the same file and thus corrupt it.<br>
<a href="https://github.com/pytorch/examples/blob/a5de3ce4da81bacc21864ace40dbc8a2e810df5a/imagenet/main.py#L254-L262">Here</a> is an example how to do so.</p>
<p>CC <a class="mention" href="/u/janine">@Janine</a></p>

username: gnadaf
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="97852">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>f you are using <code>DistributedDataParallel</code> , you would have to make su</p>
</blockquote>
</aside>
<p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, in nn.DataParallel if we use 2 GPUs, checkpoint saves the data from default device (i.e GPU-0), right? GPU-2 data is not important?</p>

username: ptrblck
cooked: <p>Yes, that’s the case since <code>nn.DataParallel</code> will create new model replicas in each iteration and thus when you store the <code>state_dict</code> outside of the forward or backward pass, there should only be a single model on the initially specified device.</p>

username: gnadaf
cooked: <p>yes, I verified with the experimen.</p>
<p>If we use two GPUs to train model in DataParallel(), Which portion of the saved data is identical between GPU 0 and GPU 1? weights, gradients, LR, etc the same across 2 GPUs?</p>
<p>I think, saved data from only GPU-0 (not from GPU-2), is it correct?</p>

username: gnadaf
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>In case of DataParallel().<br>
If we use two GPUs, initially model is replicated on both GPUs, weights and bias are same on both GPUs and after calculation, weights are updated GPU-0 (default), but GPU-1 holds the same updated value?</p>

username: ptrblck
cooked: <p>The replicas will be recreated in each forward pass using <code>nn.DataParallel</code> (which is also why it’s slower than <code>DistributedDataParallel</code>).<br>
Your code will most likely just use the single model, as seen here:</p>
<pre><code class="lang-python">model = MyModel()
model = nn.DataParallel(model)
model.to('cuda:0') # push to default device

output = model(data) # DataParallel will automatically create the replicas
loss = ...
loss.backward() # DataParallel will automatically call the backward pass on all models and reduce the gradients
optimizer.step()

torch.save(model.module.state_dict(), path) # use the default model
</code></pre>
<p>You can find more information about the underlying workflow in <code>DataParallel</code> in <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">this blog post</a>.</p>

username: gnadaf
cooked: <p>DataParallel().<br>
In case of 2 GPUs, both GPUs hold the same weights. is it correct?</p>

username: seungjun
cooked: <p>Model parameters on multiple GPUs by DataParallel and DistributedDataParallel are the same. (unless GPU communication glitch happens)</p>
<p>You can just save parameters on GPU 0 and load them later.</p>
<ul>
<li>saving</li>
</ul>
<pre><code class="lang-python">if isinstance(model, (DataParallel, DistributedDataParallel)):
    torch.save(model.module.state_dict(), model_save_name)
else:
    torch.save(model.state_dict(), model_save_name)
</code></pre>
<ul>
<li>loading</li>
</ul>
<pre><code class="lang-python">state_dict = torch.load(model_name, map_location=current_gpu_device)
if isinstance(model, (DataParallel, DistributedDataParallel)):
    model.module.load_state_dict(state_dict)
else:
    model.load_state_dict(state_dict)
</code></pre>

username: gnadaf
cooked: <p>DataParallel().<br>
If we use two GPUs, we know model will replicate on both GPUs, but I would like to know while saving model in checkpoint ---- Does checkpoint save the both replicated model?</p>
<p>I believe, the model which is present on default device (GPU-0) will be saved in the checkpoint, Right?</p>

username: seungjun
cooked: <p>Yes, for <code>DataParallel</code>, if you save by <code>torch.save(model.state_dict())</code>, it will save parameters on GPU 0.</p>
<p>But the parameters will be saved under <code>model.module</code> which cannot be loaded to non-DataParallel formats. That’s why I suggest the above code that makes saving/loading compatible with <code>nn.Module</code> format and <code>nn.DataParallel</code> format.</p>
<p>If you use <code>DistributedDataParallel</code>, you should only save parameters from local rank 0. Otherwise, each DDP process will try to save model on each GPU and overwrite each other.</p>

username: gnadaf
cooked: <p>Thanks for detailed info.</p>
<p>Having deep insight, we know models are replicated in DataParallel(), and the weights of both models are the same.</p>
<p>Where or in which code part, model weights are updated? ?(in parallel_apply() models are replicated)</p>
<p>Which model’s weight is saved in checkpoint? (GPU-0 model weights and GPU-1 model weights)</p>
<p>model present on GPU-0 weights saved in checkpoint OR model present on GPU-1 weights are saved in checkpoint?</p>

username: gnadaf
cooked: <p>Thanks for detailed info.</p>
<p>Having deep insight, we know models are replicated in DataParallel(), and the weights of both models are the same.</p>
<p>Where or in which code part, model weights are updated? ?(in parallel_apply() models are replicated)</p>
<p>Which model’s weight is saved in checkpoint? (GPU-0 model weights and GPU-1 model weights)</p>
<p>model present on GPU-0 weights saved in checkpoint OR model present on GPU-1 weights are saved in checkpoint?</p>

username: seungjun
cooked: <p>It would be a good idea to review the blog post suggested by ptrblck above.</p>
<p>In DataParallel, <code>parallel_apply</code> does not perform parameter update or synchronization.<br>
<code>replicate</code> function synchronizes model parameters across GPUs.<br>
<a href="http://https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/data_parallel.py#L160" rel="noopener nofollow ugc">dataparallel forward</a></p>
<p>As I noted, weights on GPU 0 is saved.<br>
You can almost consider weights on other GPUs to be not really there constantly. Parameters on other GPUS are only created just before their forward computation by the above <code>replicate</code> function. And the <code>replicate</code> function is called at every forward call.</p>

username: gnadaf
cooked: <aside class="quote no-group" data-username="seungjun" data-post="11" data-topic="97852">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/90ced4/48.png" class="avatar"> seungjun:</div>
<blockquote>
<p><code>istribut</code></p>
</blockquote>
</aside>
<p>Can’t we load <code>nn.DataParallel</code>  format checkpoint to <code>nn.Module</code>  format model?<br>
Is it an issue present in Pytorch checkpoint?</p>

username: seungjun
cooked: <p>No. It is not an issue.<br>
<code>nn.DataParallel</code> saves the parameters under <code>self.module</code>.<br>
For example, let’s assume your original single-gpu model had <code>self.conv</code> layer.<br>
In your <code>DataParallel</code> model, it will move to <code>self.module.conv</code>.<br>
That’s why I recommend saving <code>self.module.state_dict()</code> as in the above example code.</p>

username: gnadaf
cooked: <p>Thanks <a class="mention" href="/u/seungjun">@seungjun</a> for the detailed Info</p>