What is the Privacy Accounting usage?

Leonmac: <p>Hello Opacus team,<br>
Can someone explain the usage of the " <a href="https://opacus.ai/api/accounting/accounting.html" rel="noopener nofollow ugc">Privacy Accounting</a>" and, is there any application documentation, or code example regarding it?</p>

ashkan_software: <p>Hello,</p>
<p>Privacy accounting is used to track the privacy budget. It is useful to know how much privacy is lost at different stages of the training.</p>
<p>And this is how to use it: <a href="https://github.com/pytorch/opacus/blob/main/Migration_Guide.md#privacy-accounting" class="inline-onebox" rel="noopener nofollow ugc">opacus/Migration_Guide.md at main · pytorch/opacus · GitHub</a></p>
<p>And some examples (search for “get_epsilon”):</p>
<ul>
<li><a href="https://github.com/pytorch/opacus/blob/main/tutorials/building_text_classifier.ipynb" class="inline-onebox" rel="noopener nofollow ugc">opacus/building_text_classifier.ipynb at main · pytorch/opacus · GitHub</a></li>
<li><a href="https://github.com/pytorch/opacus/blob/main/tutorials/building_image_classifier.ipynb" class="inline-onebox" rel="noopener nofollow ugc">opacus/building_image_classifier.ipynb at main · pytorch/opacus · GitHub</a></li>
<li><a href="https://github.com/pytorch/opacus/blob/main/examples/mnist_lightning.py" class="inline-onebox" rel="noopener nofollow ugc">opacus/mnist_lightning.py at main · pytorch/opacus · GitHub</a></li>
</ul>

Leonmac: <p><a class="mention" href="/u/ashkan_software">@ashkan_software</a> Thanks for the reply.<br>
From tutorials and some blogs I get the concept of privacy budget as the “Quantitative for privacy protection", in Opacus it is the (epsilon, delta) pair.<br>
Below are some of my understanding, I am not sure if there are correct or not.</p>
<p>It looks like there are two ways:</p>
<ol>
<li>User specifies the “total epsilon to be used-- target_epsilon” and then uses the below API–in such case, epochs/target_epsilon/delta/max_grad_norm are the argument, and it looks likely the if accumulate the actual epsilon spent for each epoch train, that shall not exceed “target_epsilon”.</li>
</ol>
<pre><code class="lang-auto">model, optimizer, data_loader = privacy_engine.make_private_with_epsilon(
    module=model,
    optimizer=optimizer,
    data_loader=data_loader,
    epochs=EPOCHS,
    target_epsilon=EPSILON,
    target_delta=DELTA,
    max_grad_norm=MAX_GRAD_NORM,
)
</code></pre>
<ol start="2">
<li>another way is like this, in such a case, user specifies the noise_multiplier and the max_grad_norm as arguments, and then the epsilon/delta/(and a sigma) can be get from “get_epsilon”</li>
</ol>
<pre><code class="lang-auto">privacy_engine = PrivacyEngine()
model, optimizer, data_loader = privacy_engine.make_private(
    module=model,
    optimizer=optimizer,
    data_loader=data_loader,
    noise_multiplier=1.1,
    max_grad_norm=1.0,
)
</code></pre>
<p>My questions:</p>
<ol>
<li>Is my above description correct?</li>
<li>What is the major difference between these two methods particularly for the using scenarios (they are “mutually exclusive”, looks like for me the mutually exclusive between epsilon and noise_multiplier ).</li>
<li>I did not find many documents explaining the noise_multiplier and max_grad_norm (maximum gradient l2 norm, i.e. the Clipping value–is that Sigma in project), can you suggest whatever material for that?</li>
<li>Finally, I did not find the obvious link between the privacy budget (epsilon/delta etc…) vs <a href="https://opacus.ai/api/accounting/accounting.html" rel="noopener nofollow ugc">Privacy Accounting</a>, which includes api like  <a href="https://opacus.ai/api/accounting/iaccountant.html" rel="noopener nofollow ugc">IAccountant</a>, <a href="https://opacus.ai/api/accounting/rdp.html" rel="noopener nofollow ugc">RDPAccountant</a> and <a href="https://opacus.ai/api/accounting/gdp.html" rel="noopener nofollow ugc">GaussianAccountant</a> what is the logic behind that?</li>
</ol>

ashkan_software: <p>Hello,</p>
<p>Let me answer your questions:</p>
<ol>
<li>Your understandings are correct. Just keep in mind that when calling <code>privacy_engine.make_private_with_epsilon</code>, we want to make sure that epsilon is below the targe_epsilon, <em>across all</em> epochs, not just one (similar to <a href="https://github.com/pytorch/opacus/blob/main/tutorials/building_lstm_name_classifier.ipynb" rel="noopener nofollow ugc">here</a> where the epsilon does not exceed 12). Your understanding of the second way of calling <code>make_private_*</code> you described is correct.</li>
<li>The major difference is simply in scenarios where you know your privacy budget and want to keep the privacy below that threshold vs. the scenarios where you do not know the privacy budget in advance and trying to see what privacy you get for different noise and clipping parameters.</li>
<li>We have some great tutorials and videos listed here in <a href="https://opacus.ai/tutorials/" rel="noopener nofollow ugc">our tutorials</a>  that answer your question about <code>noise_multiplier</code> (or <code>sigma</code>) and <code>max_grad_norm</code> (or <code>c</code>) (for example, <a href="https://bit.ly/dp-sgd-algorithm-explained" rel="noopener nofollow ugc">intro to DP-SGD</a> or the videos).</li>
<li>Privacy is calculated in the privacy accounting classes. Think of it this way “privacy accounting” = privacy calculation". Now there are different ways of calculating that privacy. We implemented 2 of them: RDP and Gaussian</li>
</ol>

Leonmac: <p>Thank you so much and I still have some further detailed questions…</p>
<blockquote>
<ol>
<li>Your understandings are correct. Just keep in mind that when calling <code>privacy_engine.make_private_with_epsilon</code>, we want to make sure that epsilon is below the targe_epsilon, <em>across all</em> epochs, not just one (similar to <a href="https://github.com/pytorch/opacus/blob/main/tutorials/building_lstm_name_classifier.ipynb" rel="noopener nofollow ugc">here</a> where the epsilon does not exceed 12). Your understanding of the second way of calling <code>make_private_*</code> you described is correct.</li>
</ol>
</blockquote>
<p>For make_private_with_epsilon API, let’s say, the target_epsilon = 50, and epoch = 10 , max_grad_norm = 1.2, targe_delta = 1/(2*len(dataset))<br>
1  the epsilon spent for each training epoch, will be smaller than 50, or will be smaller than 50/10=5?, or some other value? in my test, I see the reported epsilon for each epoch will be increased epoch by epoch from about 48 to more than 100…<br>
2. how to choose a proper epsilon value anyway?<br>
3 How does max_grad_norm is defined then, this is the max grad clipping value used for “clip each p.grad in model parameters” from my study.</p>
<p>For noise_multiplier API, there is another parameter “max_per_sample_grad_norm”:<br>
4. what is the difference between “max_grad_norm” and “max_per_sample_grad_norm” (this likely only matter when a clipping method is different(CLIP_PER_LAYER = false, clipping = “per_layer”)<br>
5. furthermore in the example code, if a flag named CLIP_PER_LAYER = true, clipping = “flat”, there is a calculation as <code> max_grad_norm = [max_per_sample_grad_norm / np.sqrt(n_layers) ] * n_layers</code><br>
Could you explain a little more about this-- I don’t understand why an sqrt is used here.<br>
6. what does the “noise_multiplier” really mean and how to choose that properly: in the api document it is defined as “The ratio of the standard deviation of the Gaussian noise to the L2-sensitivity of the function to which the noise is added  (How much noise to add)” but that is still not clear to me, any detail reference document can be refered?</p>

ffuuugor: <p>Hi <a class="mention" href="/u/leonmac">@Leonmac</a></p>
<blockquote>
<p>the epsilon spent for each training epoch, will be smaller than 50, or will be smaller than 50/10=5?</p>
</blockquote>
<p>The total privacy budget doesn’t increase linearly with epoch, and the epsilon you “spend” on each epoch will decrease.</p>
<p>That said, <code>target_epsilon</code> parameter passed to <code>make_private_with_epsilon</code> is the expected epsilon at the end of training (assuming you trained for a given number of epochs). If that’s not the case, I suggest looking into the implementation of <a href="https://github.com/pytorch/opacus/blob/d48716e58ff707464fcdfd233425b54b7d20df91/opacus/accountants/utils.py#L23" rel="noopener nofollow ugc">get_noise_multiplier</a> - some of the default values there might have been picked under the assumption the target epsilon is smaller (&lt;20)</p>
<blockquote>
<p>how to choose a proper epsilon value anyway?</p>
</blockquote>
<p>Good question, but unfortunately there’s no good answer. Understanding how to interpret epsilon is tricky, and so is picking the most appropriate value for your use case. What I can recommend thought is to look at what values are used in existing literature and start from there. You can also check which values were used in real-world applications (e.g. <a href="https://desfontain.es/privacy/real-world-differential-privacy.html" class="inline-onebox" rel="noopener nofollow ugc">A list of real-world uses of differential privacy - Ted is writing things</a>)</p>
<p>Off the top of my head, people usually work with eps&lt;10 and something around 3 is considered strong privacy</p>
<blockquote>
<p>How does max_grad_norm is defined then, this is the max grad clipping value used for “clip each p.grad in model parameters” from my study.</p>
</blockquote>
<p>I’m not sure I understand the question. <code>max_grad_norm</code> isn’t used during privacy accounting. On each training step we compute per sample gradient norm (i.e. imagine each sample is processed in its own batch and all of the parameter gradients from all the layers are put into one vector) and shrink it (or “clip”) so that the L2 norm is no greater than <code>max_grad_norm</code></p>
<blockquote>
<p>what is the difference between “max_grad_norm” and “max_per_sample_grad_norm”</p>
</blockquote>
<p>They are the same thing, <code>max_grad_norm</code> is just shorter and easier to write</p>
<blockquote>
<p>Could you explain a little more about this-- I don’t understand why an sqrt is used here.</p>
</blockquote>
<p>Ok, we can think about it this way. Imagine we have two layers with 100 parameters each in our network and we’re using <code>max_grad_norm=1</code><br>
If we’re using regular flat clipping, it just means that L2 norm of a 200-element vector should be &lt;=1.<br>
With per-layer clipping we want to have separate clipping thresholds for each layer. However, the top-line requirement still stays - overall gradient norm (L2 norm of a 200-element vector) should be &lt;= 1.</p>
<p>This means, that each layer’s total norm should be <code>&lt;= 1.0 / sqrt(2)</code>. Square root is popping up because L2 norm is a square root of sum of squares - therefore if we want the sum of two elements to have L2 norm &lt;= 1, each element should be &lt;=1/sqrt(2)</p>
<blockquote>
<p>what does the “noise_multiplier” really mean and how to choose that properly:</p>
</blockquote>
<p>One thing that could help is reading Algorithm 1 from the <a href="https://arxiv.org/pdf/1607.00133.pdf" rel="noopener nofollow ugc">original DP-SGD paper</a>, sigma there is the same as <code>noise_multiplier</code> in our code</p>