Get per-sample gradient without clipping and noise adding

username: Torcione
cooked: <p>Hi,<br>
Given a training process I need, at each step, to get all gradients tensors associated to each individual sample of the batch; then I need to perform some operation on each of this gradients, finally collect them together and perform the <code>.optimizer.step()</code>.<br>
I found that <a href="https://github.com/pytorch/opacus" rel="noopener nofollow ugc">Opacus </a>  could fit my problem, but:</p>
<ul>
<li>
<p>I don’t need nor the gradient clipping neither the noise adding</p>
</li>
<li>
<p>on the other hand I need to access the set of gradients associated to each batch’s element and perform some operation on them; after these operations I’ll get a single gradient that will be used for the weight update (<code>.optimizer.step()</code>).<br>
<strong>Is it possible to do this with Opacus library?</strong></p>
</li>
</ul>

username: ffuuugor
cooked: <p>Hi <a class="mention" href="/u/torcione">@Torcione</a><br>
Yes, you can use opacus for that; Take a look at <code>GradSampleModule</code> (opacus/grad_sample/grad_sample_module.py)</p>
<p>It’s a wrapper around <code>nn.Module</code> that encapsulates per sample gradient computation. When you wrap your model with GradSampleModule, each trainable parameter will get <code>.grad_sample</code> attribute containing per-sample gradients. No noise or clipping is performed.</p>
<p>Hope this helps</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/ffuuugor">@ffuuugor</a>,</p>
<p>Is it possible to use this <code>GradSampleModule</code> on loss function which themselves contain derivatives? I’ve managed to calculate per-sample gradients using a combination of <code>forward_pre_hook</code> and <code>full_backward_hook</code>, however, I’ve noticed that if your loss function contains terms that are derivatives using hooks naively fails. Could opacus be a solution for this?</p>
<p>For clarity, this was briefly discussed in this thread: <a href="https://discuss.pytorch.org/t/per-sample-gradient-should-we-design-each-layer-differently/57293/22" class="inline-onebox">Per-sample gradient, should we design each layer differently? - #22 by AlphaBetaGamma96</a> and I wrote a small example snippet which highlights where it fails here: <a href="https://github.com/AlphaBetaGamma96/per-sample-gradient-limitation/blob/main/example.py" class="inline-onebox" rel="noopener nofollow ugc">per-sample-gradient-limitation/example.py at main · AlphaBetaGamma96/per-sample-gradient-limitation · GitHub</a></p>
<p>Do you think opacus could solve this issue?</p>
<p>Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ffuuugor
cooked: <p>I’ll take a closer look at the code a bit later, but my first hunch would be that opacus won’t make much of a difference. We also use hooks (although it’s regular backward hooks, not <code>full_backward_hook</code>, that are being deprecated in the newer PyTorch version), so I don’t see a reason why we won’t face the same problem in opacus</p>

username: chensquan
cooked: <p>Hi, I’m interesting this topic.How to get per-sample gradient.Can you show some examples?</p>

username: karthikprasad
cooked: <p>Hi <a class="mention" href="/u/chensquan">@chensquan</a>, <a href="https://medium.com/pytorch/differential-privacy-series-part-2-efficient-per-sample-gradient-computation-in-opacus-5bf4031d9e22" class="inline-onebox" rel="noopener nofollow ugc">Differential Privacy Series Part 2 | Efficient Per-Sample Gradient Computation in Opacus | by PyTorch | PyTorch | Medium</a> explains this with an example. You can also check out <a href="https://opacus.ai/tutorials/guide_to_grad_sampler" rel="noopener nofollow ugc">this guide</a>.</p>