ModuleList of unused parameters on distributed training

username: stevelu
cooked: <p>Hi all!<br>
I have a autograd and distributed training problem using PyTorch. It is a NLP problem. I want to design a neural network like the following figure.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/548166a7a815180d228aad256b45dde8689c7720.png" data-download-href="https://discuss.pytorch.org/uploads/default/548166a7a815180d228aad256b45dde8689c7720" title="module arch"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/4/548166a7a815180d228aad256b45dde8689c7720.png" alt="module arch" data-base62-sha1="c3zlzdKrZINWYTnNvWuAUATZjfG" width="375" height="500" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">module arch</span><span class="informations">451×601 5.86 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My dataset contains 50 kinds of text data. Each kind of text data has its own distribution. I want to use<br>
modules of same architecture but different weight parameters. So each module can learn distribution of its related data input. Just like the left part of  the figure.</p>
<p>What’s more, I want to use a module to capture some common features shared by different kinds of input data. (right part of the figure)</p>
<p>Now I do it like this. For modules of same architecture but different weights (left part of the figure), I use a <code>nn.ModuleList</code>. For example, if there are 50 kinds of data. I initialize 50 modules. <code>category</code> is a category list. The actual network is series of pytorch modules and is more complex than the figure and the demo code.</p>
<pre data-code-wrap="python"><code class="lang-python">self.blocks = nn.ModuleList([
    Block(...) 
    for i in range(len(category))
])
</code></pre>
<p>and for <code>forward()</code> method, I do it like this. I just iterate the batch and use different module defined in the above <code>nn.ModuleList</code>.</p>
<pre data-code-wrap="python"><code class="lang-python">block_result_list = []
for i, category in enumerate(batch['category'].split(1)):
    text_ids = batch['input_text']
    text_embed = self.embed(text_ids[i])
      
    x = self.blocks[category](text_embed)
    block_result_list.append(x)

block_result = torch.cat(block_result_list)
</code></pre>
<p><code>batch['category']</code> is category info of one batch and <code>batch['input_text']</code> is training text input.</p>
<p>I use HuggingFace <a href="https://github.com/huggingface/accelerate" rel="noopener nofollow ugc">accelerate</a> for distributed training. However, when I use distributed training. I got the following error.</p>
<pre><code class="lang-auto">RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`, and by 
making sure all `forward` function outputs participate in calculating loss. 
</code></pre>
<p>There may exists some unused parameters in forward pass because in every batch not all ModuleList may be involved. It depends on this batch of input. If this batch of input contains all categories, then all parameters are used.</p>
<p>If I set <code>find_unused_parameters=True</code>, the the output accuracy is much lower than expected.</p>
<p>If I do not use distributed training, just use one gpu, the output accuracy is correct.</p>
<p>So anybody has any ideas on how to design distributed training network of different modules conditioned on different distributions?</p>

username: Shen
cooked: <p>Hi, I’m facing a similar problem. Did you find any solutions to this?</p>

username: pbelevich
cooked: <p>CC <a class="mention" href="/u/rvarm1">@rvarm1</a> could please take a look?</p>