Autograd with respect to input?

hyperactve: <p>This is my network:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, device, channels=[2,10,10,10,10,10,1]):
                
        super(Net, self).__init__()        
        weights = []
        biases = []
        gammas = []
        
        N = len(channels)
        
        self.layers = N-1
        
        for i in range(self.layers):
            weights.append(nn.Parameter(torch.randn((channels[i],channels[i+1]))).to(device))
            biases.append(nn.Parameter(torch.randn((channels[i+1]))).to(device))
            gammas.append(nn.Parameter(torch.randn((channels[i+1]))).to(device))
            
        self.weights = weights
        self.biases = biases
        self.gammas = gammas
        
        self.nl = torch.tanh
    
    def forward(self, inp):
                
        for i in range(self.layers-1):
            x_1 = torch.mm(inp,self.weights[i]) 
            x_2 = x_1 + self.biases[i]
            x_2 = x_2 * self.gammas[i]
            inp = self.nl(x_2)
            
        x_1 = torch.mm(inp,self.weights[self.layers-1]) + self.biases[self.layers-1]
        out = x_1 * self.gammas[self.layers-1]
       
        return out
</code></pre>
<p>Now when I try to use autograd on the input:</p>
<pre><code class="lang-auto">out=net(inp)
out_inp = torch.autograd.grad(out,inp, create_graph=True)
</code></pre>
<p>I get two errors.</p>
<ol>
<li>If I use more than one input in batch size say a 4x2 matrix as <code>inp</code>, where 4 samples used as inputs, I get the following error:</li>
</ol>
<pre><code class="lang-auto">RuntimeError: grad can be implicitly created only for scalar outputs
</code></pre>
<p>However, for such a network these are 4 different samples so it should be able to compute 4 different gradients. Is there a way to get this effect?</p>
<ol start="2">
<li>If I use the a batch size of 1, I get the following error:</li>
</ol>
<pre><code class="lang-auto">RuntimeError: One of the differentiated Tensors does not require grad
</code></pre>
<p>So, I guess I have set requires_grad True for <code>inp</code>. However, I cannot do that because it is a tensor object that I use as input. Is there any way I can do this?</p>
<p>Probably I should elaborate. When I try to make requires_grad true I get the following error:</p>
<pre><code class="lang-auto">TypeError: as_tensor() got an unexpected keyword argument 'requires_grad'
</code></pre>
<p>Thanks in advance!</p>

tom: <aside class="quote no-group" data-username="hyperactve" data-post="1" data-topic="71088">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/edb3f5/48.png" class="avatar"> hyperactve:</div>
<blockquote>
<p>However, for such a network these are 4 different samples so it should be able to compute 4 different gradients. Is there a way to get this effect?</p>
</blockquote>
</aside>
<ol>
<li>If you only need to gradients w.r.t. the inputs (and don’t have bad things like batch norm), summing the outputs and then calling <code>.backward</code> will get you gradients. For weights it is more tricky to avoid gradients being accumulated.</li>
<li><code>inp.requires_grad_()</code> before passing it into the network will tell PyTorch that you will ask to differentiate by <code>inp</code>.</li>
</ol>
<p>Best regards</p>
<p>Thomas</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/tom">@tom</a>, sorry for opening up such an old thread but I was wondering if there were any examples on how to use autograd to get the gradient w.r.t the weights (for all samples) without gradients being accumulated?</p>

tom: <p>Some time ago, <a href="https://github.com/cybertronai/autograd-lib" class="inline-onebox" rel="noopener nofollow ugc">GitHub - cybertronai/autograd-lib</a> had something for this.<br>
The new way is supposed to be with vmap, but it seems it is not quite there yet <a href="https://github.com/pytorch/pytorch/issues/7786" class="inline-onebox" rel="noopener nofollow ugc">[feature request] Simple and Efficient way to get gradients of each element of a sum · Issue #7786 · pytorch/pytorch · GitHub</a> .<br>
I did it manually when I needed it.<br>
I also want to write a source-to-source differentiation thing for TorchScript if I ever find the time.</p>
<p>Best regards</p>
<p>Thomas</p>