How does Data parallel  broadcast out the parameters of a model? 

username: Bi_Glare
cooked: <p>I have some questions on Data parallel broadcasting. Dose it broadcast layer by layer or gather all parameters of the whole model?  I can find some C++ codes of it in “/torch/csrc/cuda/comm.cpp”, such as : <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a46161ff382ff9cb29eeb000257728f00c65258.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1a46161ff382ff9cb29eeb000257728f00c65258" title="1609934675304"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a46161ff382ff9cb29eeb000257728f00c65258_2_690x427.jpeg" alt="1609934675304" data-base62-sha1="3KqzzjUIWxECYJ30SEiK2RGUhCw" width="690" height="427" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a46161ff382ff9cb29eeb000257728f00c65258_2_690x427.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a46161ff382ff9cb29eeb000257728f00c65258.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a46161ff382ff9cb29eeb000257728f00c65258.jpeg 2x" data-dominant-color="2D383D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1609934675304</span><span class="informations">981×608 87.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/0/809aca87953e8d1e034bb64a853d6ebc23717372.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/809aca87953e8d1e034bb64a853d6ebc23717372" title="1609934720816"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/809aca87953e8d1e034bb64a853d6ebc23717372_2_479x500.jpeg" alt="1609934720816" data-base62-sha1="ilGKN10Agz6YUTPGoyNEuGPaCQy" width="479" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/809aca87953e8d1e034bb64a853d6ebc23717372_2_479x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/8/0/809aca87953e8d1e034bb64a853d6ebc23717372.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/0/809aca87953e8d1e034bb64a853d6ebc23717372.jpeg 2x" data-dominant-color="2F3940"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1609934720816</span><span class="informations">660×688 82 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have two GPUs, one called GPU0 which is the main GPU and another is GPU1.</p>
<p>I have added some logs in the python files(torch/nn/parallel/comm.py). The results show that it broadcasts layer by layer.</p>
<p>If it broadcasts layer by layer, another question is  that does the GPU1 receive the next layer’s parameters from GPU0 and make computation at the same time?<br>
<strong>Thanks for your attention and answers !</strong></p>

username: Yanli_Zhao
cooked: <p>looking at ‘torch/nn/parallel/replicate.py’, computation starts after all layers are coalesced and replicated on all devices</p>

username: Bi_Glare
cooked: <p>Oh!  Thanks a lot! I’m clear now. 十分感谢！！<br>
The “params” in function “replicate” are the layers of my model, and the “buffers” is the buffer, but I still not know what “modules” means in function “replicate” of ‘torch/nn/parallel/replicate.py’. Can you help me ?</p>

username: Bi_Glare
cooked: <p>The number of “modules” is only one more than the “params”, and is the <em>“for key, param in module._parameters.items():”</em>  loop really the place to replicate for every layer?</p>

username: Yanli_Zhao
cooked: <p>yes, replicating modules mostly means replicating parameters and buffers</p>

username: Bi_Glare
cooked: <p>Thanks again!<br>
I think I can continue with my work.</p>