Influence of `max_grad_norm` on DP parameter `epsilon`

username: QianZhang20
cooked: <p>I ran the <a href="https://github.com/pytorch/opacus/blob/main/examples/mnist.py" rel="noopener nofollow ugc">MNIST example</a> provided on github with different choices of argument <code>c</code> (<code>max-per-sample-grad_norm</code>) which determines the <code>max_grad_norm</code> parameter in <code>privacy_engine</code>. The other arguments are the set to their default values. However, the output DP parameter <code>epsilon</code> does not change with different values of <code>max_grad_norm</code>. The screenshots of the output are provided below.</p>
<p>I am very confused because I would expect that <code>epsilon</code> would be smaller for smaller <code>max_grad_norm</code> according to the theory of DP. Is there anything wrong with the code? Thanks.</p>
<p>Commands:</p>
<blockquote>
<p>python mnist.py -c 1<br>
python mnist.py -c 10</p>
</blockquote>
<p>Results:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e909867a05ef60e80daf7f07949642ef0c83e187.png" data-download-href="https://discuss.pytorch.org/uploads/default/e909867a05ef60e80daf7f07949642ef0c83e187" title="Merged_document"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e909867a05ef60e80daf7f07949642ef0c83e187_2_567x500.png" alt="Merged_document" data-base62-sha1="xfxEPaUTxtL1OpHl7f1MnkAggSP" width="567" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e909867a05ef60e80daf7f07949642ef0c83e187_2_567x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e909867a05ef60e80daf7f07949642ef0c83e187_2_850x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e909867a05ef60e80daf7f07949642ef0c83e187_2_1134x1000.png 2x" data-dominant-color="393939"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Merged_document</span><span class="informations">2525×2226 428 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: zoher
cooked: <p>Hi <a class="mention" href="/u/qianzhang20">@QianZhang20</a></p>
<p>As far I understand that the privacy budget (epsilon) expended so far each epochs Is depending on two arguments</p>
<ul>
<li><strong>delta</strong>: target delta</li>
<li><strong>alphas</strong>: List of RDP orders (alphas) used to search for the optimal conversion<br>
between RDP and (epd, delta)-DP<br>
you can see here <a href="https://github.com/pytorch/opacus/blob/32a465bfa980777e60a763616c83429785aa2ac6/opacus/accountants/rdp.py#L70" rel="noopener nofollow ugc">link</a><br>
and changing the max_grad_norm should not influence the privacy budget</li>
</ul>

username: HuanyuZhang
cooked: <p><a class="mention" href="/u/zoher">@zoher</a> has provided a correct answer. Although the standard deviation of the noise added in each iteration is proportional to clipping_norm * noise_multiplier (thus dependent on clipping_norm), it is noise_multiplier that decides the final epsilon. In other words, clipping_norm only affects the mode accuracy.</p>