Initialization in-place and `tensor.data`

username: murph213
cooked: <p>Like others, I have found that trying to manually initialize weights in-place is successful when through <code>weight.data</code> but not through <code>weight</code> alone.  But I am trying to understand this more deeply.  The following code shows three possible (not necessarily equivalent) ways and whether they work.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.init as init

def do_init(layer):
    # works
    init.xavier_uniform_(layer.weight)
    
    # does not work
    #layer.weight.random_()
    
    # works
    #layer.weight.data.random_()
    
lin = nn.Linear(4, 5)
print(lin.weight)
do_init(lin)
print(lin.weight)
</code></pre>
<ol>
<li>Accessing <code>weight</code> directly causes errors of an in-place edit of a leaf variable.  But how is accessing <code>weight.data</code> any different?  How is it treated differently under the hood?</li>
<li>Is an in-place edit using <code>data</code> actually safe or does it confuse the computation graph?  It seems we are tricking PyTorch into letting us manually initialize the gradients, but I am not comfortable trying to trick PyTorch without understanding the mechanism.</li>
<li>What do <code>nn.init</code> in-place methods do to avoid the “leaf variable” error?  Do they also edit the <code>data</code>?</li>
</ol>
<p>I tried looking at the code but could not determine for sure.</p>
<p>Thank you!</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>.data</code> should never be used. It is doing very dangerous things under the hood and has many unwanted side effects.</p>
<p>To make operations that are not recorded by the autograd, you should use <code>with torch.no_grad():</code>.<br>
This is exactly what the nn.init module does as you can seee <a href="https://github.com/pytorch/pytorch/blob/9441c7a9443dbc810c1190e4ca1e06c7c8308e47/torch/nn/init.py#L12-L19">here</a> for example.</p>

username: murph213
cooked: <p>So it would seem that a safe way to implement a function that does my own custom initialization would be to use <code>torch.no_grad</code> as you suggested:</p>
<pre><code class="lang-auto">torch.manual_seed(1)
def do_init2(layer):
    with torch.no_grad():
        layer.weight.normal_(0, 10)  #  10 so we can check expected behavior
        layer.bias.fill_(0)

lin = nn.Linear(4, 5)
print("before")
print(lin.weight)
print(lin.bias)

do_init2(lin)

print("after")
print(lin.weight)
print(lin.bias)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">before
Parameter containing:
tensor([[ 0.2576, -0.2207, -0.0969,  0.2347],
        [-0.4707,  0.2999, -0.1029,  0.2544],
        [ 0.0695, -0.0612,  0.1387,  0.0247],
        [ 0.1826, -0.1949, -0.0365, -0.0450],
        [ 0.0725, -0.0020,  0.4371,  0.1556]], requires_grad=True)
Parameter containing:
tensor([-0.1862, -0.3020, -0.0838, -0.2157, -0.1602], requires_grad=True)
after
Parameter containing:
tensor([[ 11.7120,  17.6743,  -0.9536,   1.3937],
        [-12.1501,   7.3117,  11.7180,  -9.2739],
        [  5.4514,   0.6628,  -4.3704,   7.6260],
        [ 11.6327,  -0.0907,  -8.4246,   1.3741],
        [  9.3864,  -1.8600,  -6.4464,  15.3925]], requires_grad=True)
Parameter containing:
tensor([0., 0., 0., 0., 0.], requires_grad=True)
</code></pre>
<p>Thank you for helping me understand these mechanics a bit better!</p>