How to calculate the Jacobian matrix of a neural network?

username: zwj
cooked: <p>How to calculate the Jacobian matrix of a neural network?</p>

username: Andrei_Cristea
cooked: <p>Have you tried <a href="https://pytorch.org/docs/stable/generated/torch.autograd.functional.jacobian.html" rel="noopener nofollow ugc"> torch.autograd.functional.jacobian</a>?</p>

username: zwj
cooked: <p>Yes,I tried, but it seems run as I get the gradient all zero</p>

username: zwj
cooked: <p>And my network Is just a single fullly connected network, no non-linear activation function, so I think the gradient should not change when the input change. It seems the gradient is changing as the input change.</p>

username: Andrei_Cristea
cooked: <p>Could you share a small snippet that shows this? It seems to not give all zeros for me:</p>
<pre><code class="lang-auto">net = nn.Sequential(
    nn.Conv2d(3, 16, kernel_size=3, padding=1),
    nn.AvgPool2d(2),
    nn.Flatten(1),
    nn.Linear(16*4*4//4, 10)
)
input = torch.randn(1, 3, 4, 4)
torch.autograd.functional.jacobian(net, input)

tensor([[[[[[-6.3562e-03, -1.0175e-02, -2.5312e-02, -1.1178e-02],
            [-1.4468e-02,  1.4982e-04, -7.6495e-03, -8.9791e-04],
            [-1.8474e-02, -2.5984e-02, -2.6214e-02,  8.7420e-03],
            [-6.6065e-03, -1.8409e-02, -6.9709e-03,  1.7825e-02]],
            ...
</code></pre>

username: zwj
cooked: <p><a href="http://211.81.52.14:8888/notebooks/jupyter/Untitled2.ipynb" class="onebox" target="_blank" rel="noopener nofollow ugc">http://211.81.52.14:8888/notebooks/jupyter/Untitled2.ipynb</a><br>
hello,sir, can you open this link</p>

username: Andrei_Cristea
cooked: <p>It won’t load. Try a Google Colab.</p>

username: zwj
cooked: <aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1SKZWGGFzitGkvMTE_B4AAEo5191nt2p_?usp=sharing">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20220602-060045-RC00_452505088" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1SKZWGGFzitGkvMTE_B4AAEo5191nt2p_?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1SKZWGGFzitGkvMTE_B4AAEo5191nt2p_?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
hello,sir, This one</p>

username: Andrei_Cristea
cooked: <p>I don’t think you’re passing the right inputs to it.</p>
<p>Instead of this:</p>
<pre><code class="lang-auto">func = lambda *params: model(data)
jacs = F.jacobian(func, params)
</code></pre>
<p>It should be this:</p>
<pre><code class="lang-auto">jacs = F.jacobian(model, data)
</code></pre>

username: zwj
cooked: <p>I am confused why the different input get different gradient, as you know, The network is a single fully-connected neural network. y=W*x+b, so when calculate dy/dx,  we are supposed to get W. W should not change when x is changing.</p>

username: Andrei_Cristea
cooked: <p>The gradient of the network means dy/dw, not dy/dx.</p>

username: zwj
cooked: <p>so how to calculate the dy/dx</p>

username: Andrei_Cristea
cooked: <p>I found these threads that might help you:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/derivative-with-respect-to-the-input/49254">thread 1</a></li>
<li><a href="https://discuss.pytorch.org/t/derivative-of-model-outputs-w-r-t-input-features/95814/2">thread 2</a></li>
<li><a href="https://discuss.pytorch.org/t/derivative-of-models-output-with-respect-to-its-input/90767">thread 3</a></li>
</ul>

username: zwj
cooked: <p>well, Thank you very much</p>