Using torch.no_grad inside forward function

username: sanchit2843
cooked: <pre><code class="lang-auto">    def forward(self, x):
        x = self.model(x)
        with torch.no_grad():
            x = x.view(-1,150,1024,size,size)
        x = self.c1(x)
        with torch.no_grad():
            x = x[0][0][:,-1,:,:,:]
        x = x.view(-1,512*size*size)
        x = self.dp(self.relu(self.fc1(x)))
        x = self.dp(self.relu(self.fc2(x)))
        x = self.dp(self.fc3(x))
        return x
</code></pre>
<p>Is this the correct way to reshape the array inside forward function or is there any better way.<br>
Thanks in advance.</p>

username: tom
cooked: <p>I must admit I have a hard time imagining when it would be a good idea to put just these things into torch.no_grad?</p>
<p>Best regards</p>
<p>Thomas</p>

username: sanchit2843
cooked: <p>Sorry I am new to programming and was not sure about this. So can I use it like this?<br>
Regards<br>
Sanchit</p>

username: tom
cooked: <p>Yeah, well. I would consider it likely that it will not achieve what you intend, but without knowing anything about what you actually want to do here, it is hard to say.</p>
<p>Best regards</p>
<p>Thomas</p>

username: sanchit2843
cooked: <p>I want to know the best way to reshape tensors inside forward function as they take up a lot of memory.</p>
<pre><code class="lang-auto">with torch.no_grad():
            x = x.view(-1,150,1024,size,size)
</code></pre>
<p>or<br>
x = x.view(-1,150,1024,size,size).detach()<br>
I am not sure which one is better or they both are the same, but the time it took to process one batch was 1.7 sec in the first case and 2.2 sec in the second case. Also, how will backpropagation take place in both cases. Sorry for not making it clear.</p>

username: tom
cooked: <p>If you want backpropagation, you need to stay clear of both <code>no_grad</code> and <code>detach</code>.</p>
<p>Benchmarking tends to be a bit delicate due to the asynchronous nature of CUDA, so you would need to take care of synchronization before measurement.</p>
<p>Generally, views share the memory with the array they are viewing, so you would not have to worry about it.</p>
<p>Best regards</p>
<p>Thomas</p>

username: dejanbatanjac
cooked: <p>Hi <a class="mention" href="/u/sanchit2843">@sanchit2843</a>,</p>
<p>I believe what Thomas meant in one of the reply, is:</p>
<p><strong>You should separate forward and backward phases.</strong></p>
<p>Inside forward there is no gradient calculation. Forward is one side of the PyTorch medal and backward is another. Backward phase is where the gradients are calculated.</p>
<p>And usually this looks like a backward on loss. Say <code>loss.backward()</code>.<br>
You can always check what is going on, if you ask for a gradient on a tensor.</p>
<p>If you have a tensor <code>t</code>, you can ask for a <code>t.grad</code>. Tensor and it’s gradient should have the same dimension, in case gradient is not <code>None</code>.</p>
<p>Here is an  example for you:<br>
You create a tensor <code>x</code> and you ask for three more things in general:</p>
<pre><code class="lang-auto">x = torch.ones(2, 2, requires_grad=True)
print(x)
print(x.grad)
print(x.grad_fn)
print(x.is_leaf)
</code></pre>
<p>And the output of this will be:</p>
<pre><code class="lang-auto">
tensor([[1., 1.],
        [1., 1.]], requires_grad=True)
None
None
True
</code></pre>
<p>Now, you create <code>y</code>:</p>
<pre><code class="lang-auto">y = x + 2

print(y)
print(y.grad)
print(y.grad_fn)
print(y.is_leaf)
</code></pre>
<p>Out:</p>
<pre><code class="lang-auto">tensor([[3., 3.],
        [3., 3.]], grad_fn=&lt;AddBackward0&gt;)
None
&lt;AddBackward0 object at 0x0000028C403BC198&gt;
False
</code></pre>
<p>So what you get is there is no gradient in <code>y</code>. Can you tell me why?</p>
<p>But let’s go further.</p>
<pre><code class="lang-auto">
loss = torch.mean(y)
loss.backward()

print(y)
print(y.grad)
print(y.grad_fn)
print(y.is_leaf)
</code></pre>
<p>Out:</p>
<pre><code class="lang-auto">tensor([[3., 3.],
        [3., 3.]], grad_fn=&lt;AddBackward0&gt;)
None
&lt;AddBackward0 object at 0x0000028C411582E8&gt;
False
</code></pre>
<p>Still, no gradient in <code>y</code>.<br>
But look there is no gradient in <code>loss</code> also:</p>
<pre><code class="lang-auto">
print(loss)
print(loss.grad)
print(loss.grad_fn)
print(loss.is_leaf)
</code></pre>
<pre><code class="lang-auto">tensor(3., grad_fn=&lt;MeanBackward0&gt;)
None
&lt;MeanBackward0 object at 0x0000028C403BC6D8&gt;
False
</code></pre>
<p>So where is the gradient?</p>
<pre><code class="lang-auto">
print(x)
print(x.grad)
print(x.grad_fn)
</code></pre>
<pre><code class="lang-auto">
tensor([[1., 1.],
        [1., 1.]], requires_grad=True)
tensor([[0.2500, 0.2500],
        [0.2500, 0.2500]])
None
</code></pre>
<p>It is calculated for <code>x</code>.</p>
<p>Now you will probable go to the definition of <a href="https://pytorch.org/docs/stable/autograd.html" rel="noopener nofollow ugc"><code>backward()</code></a> and find out it calculates the gradients for the inputs.</p>
<blockquote>
<p>Computes the sum of gradients of given tensors w.r.t. graph leaves.</p>
</blockquote>
<p>So, by definition, <code>backward</code> computes the gradients for the leaves.</p>
<p>One example, where you can set manual gradients:</p>
<pre><code class="lang-auto">x = torch.ones(2, 2, requires_grad=True)
y = x * 2

y.backward(gradient=torch.tensor([[0.5,1.],[1.5,2.]]))

print(x)
print(x.grad)
print(x.grad_fn)
print(x.is_leaf)
</code></pre>
<p>Out:</p>
<pre><code class="lang-auto">tensor([[1., 1.],
        [1., 1.]], requires_grad=True)
tensor([[1., 2.],
        [3., 4.]])
None
True
</code></pre>
<p>This speaks clearly that backward computation is little more complicated that you may expect.<br>
You can set the manual gradients and this cannot be done unless a graph behind.</p>
<p>Here is what I mean <em>unless a graph behind</em>. If you task yourself this:</p>
<pre><code class="lang-auto">
a=1
b=1
c=1
d=1
def f(a,b,c,d):
  return 2*(a + b + c + d)

e =0.01 # small number

'''gradients da'''

da = (f(a+e,b,c,d)-f(a,b,c,d))/e
print(da)
</code></pre>
<p>This will output:</p>
<p><code>1.9999999999999574</code></p>
<p>But to calculate the gradient for <code>a</code> we haven’t created anything like a graph. We just used the calculus.</p>
<p>So, I probable there is no sense to use <code>torch.no_grad</code> inside <code>forward</code>.</p>

username: sanchit2843
cooked: <p>Thanks a lot. It helped</p>