Text Classification tutorial without frozen layers

long21wt: <p>Hi,</p>
<p>I tried the tutorial notebook on Text Classification. It works well. However, I don’t understand if I don’t freeze any layer, there will be a problem in training step. More specifically:</p>
<pre><code class="lang-python">/usr/local/lib/python3.7/dist-packages/opacus/optimizers/optimizer.py in clip_and_accumulate(self)
    397             g.view(len(g), -1).norm(2, dim=-1) for g in self.grad_samples
    398         ]
--&gt; 399         per_sample_norms = torch.stack(per_param_norms, dim=1).norm(2, dim=1)
    400         per_sample_clip_factor = (self.max_grad_norm / (per_sample_norms + 1e-6)).clamp(
    401             max=1.0

RuntimeError: stack expects each tensor to be equal size, but got [8] at entry 0 and [1] at entry 
</code></pre>
<p>Any idea ? Thanks</p>

ashkan_software: <p>Hello <a class="mention" href="/u/long21wt">@long21wt</a></p>
<p>Thank you for reporting this. This is likely a bug in our tutorial. Do you mind sending us your full stack error, along with our <a href="https://colab.research.google.com/drive/1R-dnKipK9LOVV4_oKbuHoq4VvGKGbDnd?usp=sharing" rel="noopener nofollow ugc">template Colab</a> and post here the link?<br>
Please paste your colab link here. Remember: SET IT TO PUBLIC <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

long21wt: <p>Thank you. Here is the link:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1qdVVe040AOZQ9OJhRql5WV7YMw7GNkoT">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20220815-060053-RC00_467636172" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1qdVVe040AOZQ9OJhRql5WV7YMw7GNkoT" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1qdVVe040AOZQ9OJhRql5WV7YMw7GNkoT" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
As far as I know, it seems like you would need to modify forwarding method of BERT ( <a href="https://github.com/lxuechen/private-transformers" rel="noopener nofollow ugc">lxuechen/private-transformers: make differentially private training of transformers easy (github.com)</a>)<br>
And Roberta works out of the box with opacus in other experiments.<br>
Best</p>

ashkan_software: <p>Thanks for creating this. We are looking into this!</p>

long21wt: <p>Hi,</p>
<p>After a while, I’m back to this issue. By printing the model’s parameters:</p>
<pre><code class="lang-python">for n, p in model.named_parameters():
    print("{:50s} {}".format(n, list(p.grad_sample.shape) if hasattr(p, "grad_sample") else None))
</code></pre>
<p>I found the <code>position_embeddings</code> cause the problem to the optimizer, do you have any idea to fix this ?</p>
<pre><code class="lang-python">_module.bert.embeddings.word_embeddings.weight     [7, 28996, 768]
_module.bert.embeddings.position_embeddings.weight [1, 512, 768]
_module.bert.embeddings.token_type_embeddings.weight [7, 2, 768]
_module.bert.embeddings.LayerNorm.weight           [7, 768]
_module.bert.embeddings.LayerNorm.bias             [7, 768]
</code></pre>

alexandresablayrolle: <aside class="quote no-group" data-username="long21wt" data-post="5" data-topic="159167">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/long21wt/48/52368_2.png" class="avatar"> long21wt:</div>
<blockquote>
<p>I found the <code>position_embeddings</code> cause the problem to the optimizer, do you have any idea to fix this ?</p>
<pre><code class="lang-auto">
</code></pre>
</blockquote>
</aside>
<p>I will try to take a look. In the meantime, I believe that functorch can alleviate the issue because it computes per-sample gradients in a different way (using the “no_op” version of the grad sample module, see e.g. <a href="https://github.com/pytorch/opacus/blob/main/examples/cifar10.py" rel="noopener nofollow ugc">https://github.com/pytorch/opacus/blob/main/examples/cifar10.py</a>)</p>