Layer identifier in powerSGD_hook

username: hamidreza_ramezani
cooked: <p>Hello everyone,</p>
<p>I’m looking for an identifier that specifies a neural net’s layer in <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/algorithms/ddp_comm_hooks/powerSGD_hook.py" rel="noopener nofollow ugc">this</a>.</p>
<p>Also, I would like to know if <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/algorithms/ddp_comm_hooks/powerSGD_hook.py#L364" rel="noopener nofollow ugc">this</a> loop iterates over all layers?</p>
<p>Cheers,</p>

username: gcramer23
cooked: <p>Hello hamidreza_ramezani</p>
<blockquote>
<p>I’m looking for an identifier that specifies a neural net’s layer in <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/algorithms/ddp_comm_hooks/powerSGD_hook.py" rel="noopener nofollow ugc">this</a>.</p>
</blockquote>
<p>That is ddp communication hook. It is used during the backward pass.</p>
<blockquote>
<p>Also, I would like to know if <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/algorithms/ddp_comm_hooks/powerSGD_hook.py#L364" rel="noopener nofollow ugc">this</a> loop iterates over all layers?</p>
</blockquote>
<p>That loop iterates over all the parameters stored in the bucket. The parameters in the bucket are determined at construction time.</p>
<p>ddp doc <a href="https://pytorch.org/docs/stable/notes/ddp.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch 2.1 documentation</a></p>
<p>ddp communication hook doc <a href="https://pytorch.org/docs/stable/ddp_comm_hooks.html" class="inline-onebox" rel="noopener nofollow ugc">DDP Communication Hooks — PyTorch 2.1 documentation</a></p>

username: hamidreza_ramezani
cooked: <p>Hey Garrett,</p>
<p>Thanks for the reply.</p>
<blockquote>
<p>Blockquote That is ddp communication hook. It is used during the backward pass.</p>
</blockquote>
<p>I see. I guess there should be a variable that represents a layer like <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/algorithms/ddp_comm_hooks/powerSGD_hook.py#L335" rel="noopener nofollow ugc">this</a>. I’m not sure if <code>bucket_index</code> represents a layer though. Are bucket and layer the same thing in this context? I printed the value of <code>bucket_index</code> and noticed that it only takes two values 0 and 1 (the application was training RN20 on CIFAR10).</p>

username: rvarm1
cooked: <p><code>bucket.get_index()</code> does not necessarily correspond to layer indices.</p>
<p>We use GradBuckets to store gradients in DDP’s reducer which is passed to the gradient communication hook. Buckets can contain gradients from 1 or more parameters that correspond to 1 or more layers, and the index is just the index of this bucket in the list of all buckets.</p>
<p>In addition, there is an API <code>bucket.get_per_parameter_tensors()</code> (<a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/algorithms/ddp_comm_hooks/powerSGD_hook.py#L335" class="inline-onebox" rel="noopener nofollow ugc">pytorch/powerSGD_hook.py at master · pytorch/pytorch · GitHub</a>) that will allow you to get the tensors for a given parameter.</p>

username: hamidreza_ramezani
cooked: <p>I see. Thank you very much.</p>