DP-SGD using Opacus

NBu: <p>Hi all,</p>
<p>I have followed tutorials regards DP Image Classification using Resnet18.<br>
I have some questions:</p>
<ul>
<li>When a model has many layers, it wasn’t able to convergence under DP. Are there any recommended approaches to overcome this problem for large models with many Fully connected layers?</li>
<li>When I decreased batch_size using the same model (due to memory size 8GB), the loss goes up to 80-200. That’s mean it is challenging to make a large batch with DP. Large batch size sometimes helps to improve model accuracy.</li>
<li>With trying different DL models, training non-private model takes a reasonable time (e.g., 6 minutes) compared to the private model (19 minutes) with lower accuracy.</li>
</ul>
<p>Is DP-SGD very slow due to per-sample gradients? Is there any way to speed up processing time?</p>
<p>Can we achieve a comparable accuracy with the baseline model under a modest privacy budget?</p>
<p>Is the DP-Deep Learning model more sensitive to hyperparameters like batch size and noise level or the structure of NN?</p>
<p>Thanks,</p>

karthikprasad: <p>Hello <a class="mention" href="/u/nbu">@NBu</a>,<br>
I really don’t know how this post went unattended. Sorry for the delay.</p>
<blockquote>
<ul>
<li>When a model has many layers, it wasn’t able to convergence under DP. Are there any recommended approaches to overcome this problem for large models with many Fully connected layers?</li>
</ul>
</blockquote>
<p>Do you mind sharing an example notebook of this? Hyper-parameter tuning should typically help with these cases. Some tips: <a href="https://opacus.ai/docs/faq#my-model-doesnt-converge-with-default-privacy-settings-what-do-i-do" class="inline-onebox" rel="noopener nofollow ugc">FAQ · Opacus</a></p>
<blockquote>
<ul>
<li>When I decreased batch_size using the same model (due to memory size 8GB), the loss goes up to 80-200. That’s mean it is challenging to make a large batch with DP. Large batch size sometimes helps to improve model accuracy.</li>
</ul>
</blockquote>
<p>The memory requirement is an unfortunate consequence of maintaining per-sample gradients. Opacus provides a concept of <code>virtual_batch</code> to overcome this issue. Please try it out if you haven’t already.  <a href="https://opacus.ai/docs/faq#what-is-virtual-batch-size" class="inline-onebox" rel="noopener nofollow ugc">FAQ · Opacus</a></p>
<blockquote>
<p>Is DP-SGD very slow due to per-sample gradients? Is there any way to speed up processing time?</p>
</blockquote>
<p>There is some overhead to computing per-sample gradients. The speed degradation depends on the layers as well (eg Linear vs LSTM). We can take a look at your notebook to make some suggestions.</p>
<blockquote>
<p>Can we achieve a comparable accuracy with the baseline model under a modest privacy budget?</p>
</blockquote>
<p>Depends a lot on the task. As you can see on <a href="https://opacus.ai" rel="noopener nofollow ugc">https://opacus.ai</a>, the answer is a resounding yes for MNIST with a small network. However, there is a ~20pp gap in accuracy for CIFAR-10 with ResNet18.</p>
<p>Bridging this gap is an active area of research, and we whole heartedly welcome your ideas and contributions. One of the main goals of open-sourcing opacus is to help push this research.</p>