Distributed launch utility unstable

kirk86: <p>The distributed launch utility seems like unstable in usage.<br>
Executing the same program once with the following command</p>
<p><code>python -m torch.distributed.launch --nproc_per_node=3 --nnodes=1 --node_rank=0 --master_addr="127.0.0.1" --master_port=62123 main.py</code></p>
<p>Works fine:</p>
<pre><code class="lang-auto">1.0, 0.05, 2.1814, 0.1697, 2.0053, 0.2154
1.0, 0.05, 2.1804, 0.1674, 1.9767, 0.2406
1.0, 0.05, 2.1823, 0.1703, 1.9799, 0.2352
2.0, 0.05, 2.1526, 0.1779, 2.1166, 0.1908
2.0, 0.05, 2.1562, 0.1812, 2.0868, 0.2076
2.0, 0.05, 2.1593, 0.1741, 2.0935, 0.192
3.0, 0.05, 1.9386, 0.2413, 1.8037, 0.3017
3.0, 0.05, 1.9319, 0.2473, 1.8041, 0.2903
3.0, 0.05, 1.9286, 0.2443, 1.815, 0.2939
4.0, 0.05, 1.7522, 0.3153, 1.828, 0.3131
4.0, 0.05, 1.7504, 0.3207, 1.7613, 0.3245
</code></pre>
<p>After the program is finished executing again the same command i.e., calling launch with the same arguments results in an error</p>
<pre><code class="lang-auto">File "/home/kirk/miniconda3/envs/torch/lib/python3.6/site-packages/torch/serialization.py", line 386, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "/home/kirk/miniconda3/envs/torch/lib/python3.6/site-packages/torch/serialization.py", line 580, in _load
    deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
RuntimeError: storage has wrong size: expected 4333514340733757174 got 256
</code></pre>

ptrblck: <p>The error point to a broken checkpoint, not the distributed launch.<br>
It seems you’ve saved some checkpoints in the previous runs without making sure only a single process (e.g. rank0) writes to the files.<br>
This might yield to multiple processes writing to the same checkpoint file and thus breaking it.<br>
Could this be the case?</p>

kirk86: <p>Thanks for the reply. That could be the case but in all the examples I’ve seen using distributed launch didn’t show how to properly save the checkpoint. When I save the checkpoint I was just using <code>torch.save</code>. Should I be using something else?</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="55973">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>It seems you’ve saved some checkpoints in the previous runs without making sure only a single process (e.g. rank0) writes to the files.</p>
</blockquote>
</aside>
<p>How do I do that? Any pointers or examples where to look would be much appreciated!</p>

kirk86: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> So I did  try your suggestion in the following way:</p>
<pre><code class="lang-auto">if torch.distributed.is_available() and torch.distributed.is_initialized():
   if os.environ['RANK'] == 0:
      torch.save(checkpoint)
else:
      torch.save(checkpoint)
</code></pre>
<p>But still I’m getting the same error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f7311958668856883d949b75e5995aded56cb56c.png" data-download-href="https://discuss.pytorch.org/uploads/default/f7311958668856883d949b75e5995aded56cb56c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f7311958668856883d949b75e5995aded56cb56c_2_690x62.png" alt="image" data-base62-sha1="zgL819YACn8ZV3v2QEiWOs8q86E" width="690" height="62" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f7311958668856883d949b75e5995aded56cb56c_2_690x62.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f7311958668856883d949b75e5995aded56cb56c_2_1035x93.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/f7311958668856883d949b75e5995aded56cb56c.png 2x" data-dominant-color="292E29"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1111×100 36.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I’m not sure if the <code>RANK</code> env variable is useful at this point. In the <a href="https://github.com/pytorch/examples/blob/ee964a2eeb41e1712fe719b83645c79bcbd0ba1a/imagenet/main.py#L252" rel="nofollow noopener">ImageNet example</a> the <code>args.rank</code> variable is used. Could you try that?</p>

kirk86: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the pointer. My scenario is single node multi-gpu. Considering that case <code>rank=0</code> and <code>world_size=2</code>. According to the imagenet example it says save the checkpoint if torch distributed is running or if torch distributed is running and the rank is equal to the num_gpus.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="55973">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I’m not sure if the <code>RANK</code> env variable is useful at this point.</p>
</blockquote>
</aside>
<p>Why is not useful, my understanding is that it is set dynamically from the <code>launch</code>  utility and it will contain whichever rank is currently running?</p>

ptrblck: <p>The environment variable is the legacy approach and is just set, if <code>use_env</code> was specified as seen <a href="https://github.com/pytorch/pytorch/blob/2b52c1d9826e8e8c83f1ddafd4888bdb3bd00af9/torch/distributed/launch.py#L179" rel="nofollow noopener">here</a>.<br>
From the docs:</p>
<pre><code class="lang-auto">5. Another way to pass ``local_rank`` to the subprocesses via environment variable
``LOCAL_RANK``. This behavior is enabled when you launch the script with
``--use_env=True``. You must adjust the subprocess example above to replace
``args.local_rank`` with ``os.environ['LOCAL_RANK']``; the launcher
will not pass ``--local_rank`` when you specify this flag.
.. warning::
    ``local_rank`` is NOT globally unique: it is only unique per process
    on a machine.  Thus, don't use it to decide if you should, e.g.,
    write to a networked filesystem.  See
    https://github.com/pytorch/pytorch/issues/12042 for an example of
    how things can go wrong if you don't do this correctly.
</code></pre>

kirk86: <p>Thanks for the clarifications, reading through the github issues it seems that:</p>
<ol>
<li>
<p><code>local_rank</code>  is actually the ID  <em>within</em>  a worker; multiple workers have a  <code>local_rank</code>  of  <code>0</code> , so they’re probably trampling each other’s checkpoints.</p>
</li>
<li>
<p>added a  <code>--global_rank</code>  command line argument as well. (solution)</p>
</li>
<li>
<p>someone else comments, <code>torch.distributed.launch</code>  <a href="https://github.com/pytorch/pytorch/blob/929bffe020be508cd269b50b372a56c397073160/torch/distributed/launch.py" rel="nofollow noopener">sets up a  <code>RANK</code>  environment variable</a> which can be used to detect if you are on the master process (with  <code>os.environ['RANK'] == '0'</code>  from python</p>
</li>
<li>
<p>you can use  <code>torch.distributed.get_rank()</code>  to get the global rank. (I suppose this might be the most appropriate way to do it?)</p>
</li>
</ol>