How to change DDP parameter 'find_unused_parameters'=True to False during training

jhp: <p>Hi I have a question about DDP.</p>
<p>When I use DDP with dynamically freezing or unfreezing my model’s parameters during the training, one option to make a non-error function is to use <code>find_unused_parameters</code>. But when there are no parameters that are unused during backpropagation, a warning occurs as below.</p>
<p><code>[W reducer.cpp:1050] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters, consider turning this flag off. Note that this warning may be a false positive your model has flow control causing later iterations to have unused parameters. (function operator())</code></p>
<p>Is there any way to disable <code>find_unused_parameters</code> during the process?</p>
<p>P.S) Using pytorch_lightning is not the option for me now <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"></p>

ptrblck: <p>I’m not sure, if you could completely disable it, but based on the usage of <code>TORCH_WARN_ONCE</code> as seen <a href="https://github.com/pytorch/pytorch/blob/479fc4e41250a1710e34cc1dcaa6272c89832fb4/torch/csrc/distributed/c10d/reducer.cpp#L1294-L1301">here</a> I would expect to see this warning only once or are you seeing it in each iteration?</p>

jhp: <p>Only once when I unfreeze the rest of the parameters.</p>
<p>By the way, does that also makes performance (i.e accuracy) worse?<br>
I thought only the speed issue will occur, but the metric value drops significantly and then comes up again repeatedly. (or maybe my model is not stable <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"> )</p>

ptrblck: <p>No, the accuracy should not change only the speed as additional checks are needed.</p>

jhp: <p>Thanks for the reply. Right now I use <code>add_param_group</code> in optimizer for unfreezing parameters.<br>
Hope that<code> find_unused_parameters</code> can dynamically updated later in Pytorch</p>

dlsf: <p>This is very annoying, it should be possible to disable this.<br>
What is the workaround to use DDP and only update a subset of parameters?</p>

pritamdamania87: <p><a class="mention" href="/u/dlsf">@dlsf</a> Thanks for the feedback! Is your use case that you would like to train few iterations/epochs with <code>find_unsued_parameters=True</code>, then switch to <code>find_unused_parameters=False</code> for later iterations? If so, one option is to just create a new DDP instance with a different <code>find_unused_parameters</code> value for the remaining iterations.</p>

stevelu: <p>My use case is to train a subset of the whole model. The model contains multiple modules. It update module parameters based on different input data. So how to use DDP for this case? <code>find_unused_parameters=False</code> will result into error.</p>

JiamingLiu-Jeremy: <p>Hi ptrblck,</p>
<p>Thank you so much for your answer. I found setting “find_unused_parameters=True” will significantly lower the the training speed. Is this true in general ?</p>
<p>Thanks!</p>

ptrblck: <p>A slowdown is expected and you might want to check if <code>static_graph</code> would work instead as it could potentially reduce the slowdown. From the docs:</p>
<blockquote>
<ol start="5">
<li>Potentially improve performance when there are unused parameters, as DDP will not search graph in each iteraton to detect unused parameters when static_graph is set to be <code>True</code>. To check whether you can set static_graph to be <code>True</code>, one way is to check ddp logging data at the end of your previous model training, if <code>ddp_logging_data.get("can_set_static_graph") == True</code>, mostly you can set <code>static_graph = True</code> as well.</li>
</ol>
</blockquote>