Experiment: Read the gradients of random initialization

blackbirdbarber: <p>Wanted to show that this is better</p>
<pre><code class="lang-auto">w = torch.randn(2,5)
w.requires_grad_()
</code></pre>
<p>instead of</p>
<pre><code class="lang-auto">w = torch.randn(2, 5, requires_grad=True)
</code></pre>
<p>not to include gradients of initialization.</p>
<pre><code class="lang-auto">w = torch.randn(2, 5, requires_grad=True)
w.backward(retain_graph=True)
print(w.grad)
</code></pre>
<p>But my example to show grads failed with the error:</p>
<blockquote>
<p>RuntimeError: grad can be implicitly created only for scalar outputs</p>
</blockquote>
<p>How can I read the gradients of <code>w</code> initialization in the second case?</p>

JuanFMontesinos: <p>I don’t really understand wat are you asking for.<br>
In addition you only can backward scalars. w is a tensor.</p>
<p>Gradients are not “initialize”, they are equal to None until you backpropagate something.<br>
You can access to gradients by doing w.grad<br>
However nothing will be returned as there are no gradients… (unless you backprop an scalar which involves w)</p>

blackbirdbarber: <p>Both</p>
<pre><code class="lang-auto">w = torch.randn(2,5)
w.requires_grad_()
</code></pre>
<p>and</p>
<pre><code class="lang-auto">w = torch.randn(2, 5, requires_grad=True)
</code></pre>
<p>will work in PyTorch without any compilation errors. I tried hard to make this example to get the gradient of a jump from uninitialized to random value, but I am missing some important tip to check this.</p>

blackbirdbarber: <aside class="quote no-group" data-username="JuanFMontesinos" data-post="2" data-topic="48625">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>unless you backprop an scalar which involves w</p>
</blockquote>
</aside>
<p>How can I do that. This looks like the way.</p>

JuanFMontesinos: <p>You can brackpropagate w.mean(), w.sum() or whatever which returns an scalar. Anyway I there is no such a thing as initialization…</p>

blackbirdbarber: <p>Thanks <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>, ah, I had such a hard time making this test, you helped.</p>
<p>Here is what I have when using the function <code>w.mean</code>.</p>
<pre><code class="lang-auto">w = torch.randn(2, 5, requires_grad=True)
print(w)
r = w.mean()
r.backward(retain_graph=True)
print(w.grad)
# tensor([[ 0.6635, -0.3868,  2.0399,  0.3369, -0.5601],
#         [ 0.0737,  0.7711, -1.0142,  0.8241,  0.7964]], requires_grad=True)
# tensor([[0.1000, 0.1000, 0.1000, 0.1000, 0.1000],
#         [0.1000, 0.1000, 0.1000, 0.1000, 0.1000]])
</code></pre>
<p>I am afraid I cannot understand this result. Do we have in here the gradients accumulation?</p>

JuanFMontesinos: <p>You have the derivative of a sum wrt to each element: d((x+sum)N)/dx for each element which equals 1/10</p>

ptrblck: <p>What exactly would you like to show?<br>
You can pass a gradient of the same shape as <code>w</code>, which would only set <code>w.grad</code> to this particular value:</p>
<pre><code class="lang-python">w = torch.randn(2, 5, requires_grad=True)
w.backward(torch.ones_like(w))
print(w.grad)
&gt; tensor([[1., 1., 1., 1., 1.],
          [1., 1., 1., 1., 1.]])
</code></pre>

blackbirdbarber: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, this <a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html#sphx-glr-beginner-nn-tutorial-py" rel="nofollow noopener">url</a> has the intriguing text:  " For the weights, we set <code>requires_grad</code> <strong>after</strong> the initialization, since we don’t want that step included in the gradient." which  I wanted to prove.</p>
<p>Maybe this is not more the case.</p>

ptrblck: <p>I would recommend to wrap evenrything in a <code>with torch.no_grad()</code> block, which should not be tracked by Autograd.<br>
While this approach of manually setting <code>requires_grad(True)</code> will most likely work for this manual approach, it’s a bit clearer for me to just use <code>torch.no_grad</code>.</p>

blackbirdbarber: <p>OK, this is something I will certainly do when I don’t plan to track autograd, but in here, I just wanted to check if we can track gradients of random initialization:</p>
<pre><code class="lang-auto">w = torch.randn(2, 5, requires_grad=True)
</code></pre>

ramonav98: <p>Here is what you are trying to prove. Let T1 and T2 be two tensors created by the <code>torch.randn()</code> function, provided the same random seed, with the only difference them between being the moment at which <code>requieres_grad</code> is set to <code>True</code>. That is,</p>
<pre><code class="lang-auto">seed = 42
# use seed to create the first random tensor
torch.random.manual_seed(seed)
T1 = torch.randn(2,5, requires_grad = True)
# use the same seed to create the second random tensor
torch.random.manual_seed(seed)
T2 = torch.randn(2,5)
T2.requieres_grad_(True) # notice the inplace operation
</code></pre>
<p>Now, let us perform the exact same operations in both tensors <code>T1</code> and <code>T2</code>. In this way, once we call the <code>backward()</code> method with some different tensor of the same shape as input (in this case I chose a tensor of all ones), both tensors <code>T1</code> and <code>T2</code> should have the same value at their <code>grad</code> atribute.</p>
<pre><code class="lang-auto"># for T1
x1 = 3 * T1 
y1 = x1 + 1
z1 = y1 * y1

# for T2
x2 = 3 * T2 
y2 = x2 + 1
z2 = y2 * y2

# calling the backward method
z1.backward(torch.ones_like(z1))
z2.backward(torch.ones_like(z2))

# printing the .grad for T1 and T2
print(T1.grad)
print(T2.grad)
#tensor([[ 12.0604,   8.3186,  10.2203,  10.1460, -14.2114],
#        [  2.6461,  45.7476,  -5.4839,  14.3098,  10.8123]])
#tensor([[ 12.0604,   8.3186,  10.2203,  10.1460, -14.2114],
#        [  2.6461,  45.7476,  -5.4839,  14.3098,  10.8123]])
</code></pre>
<p>You get the same value for both.</p>
<p><strong>HOWEVER</strong>, the intriguing text inside the link you are referring to (<a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html#sphx-glr-beginner-nn-tutorial-py" rel="nofollow noopener">url</a>), aims for something different. Let me paste the code that originated the confussion,</p>
<pre><code class="lang-auto">weights = torch.randn(784, 10) / math.sqrt(784)
weights.requires_grad_()
</code></pre>
<p>For this case, <em><strong>it does</strong></em> matter where <code>requieres_grad</code> is set to <code>True</code>. If you try to set it at the first line, like this</p>
<pre><code class="lang-auto">weights = torch.randn(784, 10, requires_grad=True) / math.sqrt(784)
</code></pre>
<p>after arbitatry operations are performed on <code>weights</code> and  the <code>backward()</code>method is called, you will see a warning from PyTorch saying that yout are trying to access the <code>grad</code> attribute of a non leaf tensor, so <code>weights.grad</code> is set to <code>None</code>. Why? Becasue in such case, <code>weights</code> does not follow the definition of a leaf tensor: A leaf Variable is a variable that no operation tracked by the autograd engine created it (see this <a href="https://discuss.pytorch.org/t/valueerror-cant-optimize-a-non-leaf-tensor/21751">post</a> for further examples). So, what is keeping <code>weights</code> from being a leaf variable? The division by <code>sqrt(784)</code>.</p>
<p>Try it yourself and let me know!</p>