Manually re-connect a broken forward path in y=wx operation

username: Shen
cooked: <p>Hello, my work to customize linear multiplication <strong>y = wx</strong> in a neural network requires to manipulate the shape of <strong>w</strong> and <strong>x</strong> matrices. For example, previously <strong>w</strong> and <strong>x</strong> have the shape <em><code>mxk</code></em> and <em><code>kxn</code></em> separately; now I need to temporarily expand them to <em><code>mx2k</code></em> and <em><code>2kxn</code></em> to do some processing. Making changes on <strong>w</strong> and <strong>x</strong> directly doesn’t seem to be a good idea, since it will change the parameters to update.</p>
<p>Therefore I created temporary tensors <strong>w1</strong> and <strong>x1</strong> to have the expanded shape and values, and compute <strong>y</strong> using these two. i.e.</p>
<pre><code class="lang-auto">y = w1x1, where w1 has the shape mx2k and x1 has the shape 2kxn
</code></pre>
<p>However, the neural network parameters do not get updated properly by doing this. My guess is this customization broke the computational graph (i.e. <strong>y</strong> is not properly linked to <strong>w</strong> and <strong>x</strong>). Is there a way to set any properties of <strong>y</strong> to fix this broken path?</p>

username: tom
cooked: <p>I think to get a useful answer, you’d have to say a bit more what you want the relationship between w1 and x1 to be (e.g. do you want w1 to have two blocks with the entries of w or something completely different or so).</p>
<p>Best regards</p>
<p>Thomas</p>

username: Shen
cooked: <p>Hi, thanks for your reply. One example of the relationship between w1,x1 and w,x could be to round each element in w and x to integer first, then expand them to bits expression. For instance, if <code>x = [[1,0], [1,2]]</code>, then <code>x1 wuold be [[0,0], [1,0], [0,1], [1,0]]</code>. Then the result y would be <code>w1x1</code> plus some post processing.</p>
<p>In this case, <code>y</code> is not directly computed from <code>wx</code>, and hence the backpropogation would not flow properly. I wonder if there is a way to tell the neural network to treat <code>y</code> as computed from <code>wx</code>?</p>

username: tom
cooked: <p>One trick that often helps for “pretend it has been calculated with x even when I used x1” is to use <code>y = wx + (w1x1 - wx).detach_()</code>: In the forward wx is cancelled out, so y = w1x1, but in the backward, the detach_ causes gradients to only flow to wx.<br>
Would that work for you?</p>
<p>Best regards</p>
<p>Thomas</p>

username: Shen
cooked: <p>Hi, I’ve been investigating this issue these days. The approach you provided seems working properly. Thank you so much!!!</p>
<p>However, I also tried something like this:</p>
<pre><code class="lang-auto">y = wx
y1 = w1x1
y.data = y1.data
</code></pre>
<p>From my understanding, this should also work. But it actually led to different result from your approach. Do you know why is that?</p>

username: InnovArul
cooked: <p>Autograd does not trace this operation (manually changing the <code>data</code> attribute) and hence different result.</p>

username: Shen
cooked: <p>Hi, thanks for your reply. I still don’t get what do you mean by “not trace this operation”. I tried to create a simple network with only 2 parameters, w1 and w2. The middle layer is given by y1=w1x, and the output layer is given by y2 = w2y1. Now if I simply do</p>
<pre><code class="lang-auto">y1 = w1x
temp = torch.tensor([1.0, 2.0])
y1.data = temp.data
</code></pre>
<p>The backward gradient seems getting calculated properly. How and in which cases do <code>.data</code> cause problems?</p>

username: ptrblck
cooked: <p>As <a class="mention" href="/u/innovarul">@InnovArul</a> said, the manipulation of <code>data</code> won’t be traced and thus might lead to a wrong result.<br>
Here is a simple example demonstrating this issue.<br>
In the first part of the code, we just calculate the loss for our operations and apply the gradient on <code>w</code>.<br>
We expect values of <code>[[17., 17.]]</code> after the update.<br>
In the second example, I’ve manipulated the underlying data of <code>w</code> after the gradient calculation.<br>
Autograd did not trace this manipulation and the gradients for the original <code>w</code> are now applied on the manipulated <code>w</code>.</p>
<pre><code class="lang-python">
# Standard approach
x = torch.ones(1, 2)
w = torch.ones(2, 1, requires_grad=True)
target = torch.full((1,), 10.)
optimizer = torch.optim.SGD([w], lr=1.)

output = x.mm(w)
loss = (output - target)**2
loss.backward()
print(w.grad)
&gt; tensor([[-16.],
          [-16.]])
optimizer.step()
print(w)
&gt; tensor([[ 17.],
          [ 17.]])

# Now manipulate the underlying data
w = torch.ones(2, 1, requires_grad=True)
optimizer = torch.optim.SGD([w], lr=1.)

output = x.mm(w)
loss = (output - target)**2
loss.backward()
print(w.grad)
&gt; tensor([[-16.],
          [-16.]])
w.data = torch.full((2, 1), -100.)
optimizer.step()
print(w)
&gt; tensor([[-84.],
          [-84.]])
</code></pre>

username: Shen
cooked: <p>Oh, I see. So in my example above, the backward computation will use the manipulated value of y1 (i.e. temp) instead of the original one (i.e. w1x), right?</p>
<p>But for the this code blow:</p>
<pre><code class="lang-auto"># method 1
y = wx
temp = w1x1
y.data = temp.data

# method 2
y = wx
temp = w1x1
y += (temp - y).detach_()
</code></pre>
<p>I still think they are equivalent, since during the backward computation, the value of <code>y</code> will be changed to <code>temp</code> in both cases. Please correct me if I’m wrong.</p>

username: InnovArul
cooked: <p>Ofcourse, there are many ways to make the operation equivalent in calculating gradients. It’s just not advised to do access <code>.data</code>, if you want to completely depend on autograd for the accuracy of your gradients.</p>
<p>From your question on the top,  I feel that your use case would be achievable by using <code>.repeat()</code> function of tensor.  Can you have a look?</p>
<p>for example,  you can say</p>
<pre><code class="lang-auto">y = w.repeat(1,2) * x.repeat(1,2)
</code></pre>

username: Shen
cooked: <p>Hi, thank you for help!! This could work, but instead of this one line code, can I split them to different lines?</p>
<pre><code class="lang-auto">w1 = w.repeat(1,2)
x1 = x.repeat(1,2)
y = w1*x1
</code></pre>
<p>Since <code>w</code> has <code>.requires_grad</code> set to <code>True</code>, Pytorch document indicates <code>w1</code> will also has <code>.requires_grad</code> set to <code>True</code>. Will this change the network topology or increase the number parameters?</p>

username: InnovArul
cooked: <p>You can absolutely do this and it will not increase the params. i.e., <code>w</code> is the only param in your code (assuming <code>x</code> is data).</p>

username: tom
cooked: <p>I would generally recommend against using data at all.<br>
There are two parts:</p>
<ul>
<li>If you want to break the graph at a point, use <a href="https://pytorch.org/2018/04/22/0_4_0-migration-guide.html#what-about-data" rel="nofollow noopener"><code>.detach()</code></a> instead.</li>
<li>If you want calculation that without autograd use <code>with torch.no_grad():</code>.</li>
</ul>
<p>Using <code>.data</code> is a bit like using those two, but except in very special situations (e.g. the optimizer updates internally) there isn’t a good reason to use it except if you like headaches (then it is much cheaper than too much beer at the Octoberfest, though).</p>
<p>Best regards</p>
<p>Thomas</p>

username: Shen
cooked: <p>Thank you all for the help!!</p>

username: Shen
cooked: <p>Hi, these days I went back to this problem for some reason, it seems autograd did trace <code>data</code> attribute. Here is an example based on yours:</p>
<pre><code class="lang-auto">x = torch.ones(1, 2)
w = torch.ones(2, 1, requires_grad=True)
target = torch.full((1,), 10.)

# manipulate data attribute
x.data = torch.cat((x,x), dim=1)
w.data = torch.cat((w,w), dim=0)

output = x.mm(w)
optimizer = torch.optim.SGD([w], lr=1.)

loss = (output - target)**2
loss.backward()
print(w.grad)
&gt; tensor([[-12.],
          [-12.],
          [-12.],
          [-12.]])
</code></pre>
<p>If autograd did not trace data attribute, should <code>w.grad</code> have a size of <strong>2x1</strong> instead of <strong>4x1</strong>? Please correct me if I’m wrong. Thx!</p>

username: ptrblck
cooked: <p>Thanks for the example!<br>
The manipulation of the <code>.data</code> attribute will work in your example, as you are manipulating it before performing any computation.<br>
I would still discourage the usage of it, as it might still break, if you manipulate the data after some of the computation graph was already created.</p>