Multiplying pytorch's returned weights don't give the correct output 'y'

username: Saif_Ali_Khan
cooked: <p>Hey, i’m a newcomer to Pytorch. I created a dummy dataset after generating random weight vector and bias, and obtaining ‘y’ using WXt + b. After that I implemented a simple linear regression using Pytorch’s single linear layer, and trained the data with same X and y generated above. The rmse is extremely small and the output is also virtually same as original y, but weight vector is completely different than the original. Bias is same though.<br>
I tried to see if Pytorch somewhat normalizes the weights, so i divided each index of Original W with corresponding index of trained W, and got a constant.<br>
For example:</p>
<pre><code class="lang-auto">Original_W = [1.85623121, 0.02777083, 0.32021133]
Trained_W = [11.1365,  0.1659,  1.9205]
Trained_W / Original_W = [5.99952201, 5.97389396, 5.99760163]
</code></pre>
<p>(The deviation is probably due to training error)<br>
Can someone please explain?</p>
<p>Thanks!<br>
Saif</p>

username: KFrank
cooked: <p>Hi Saif!</p>
<aside class="quote no-group" data-username="Saif_Ali_Khan" data-post="1" data-topic="118753" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saif_ali_khan/48/37319_2.png" class="avatar"> Saif_Ali_Khan:</div>
<blockquote>
<p>weight vector is completely different than the original.</p>
</blockquote>
</aside>
<p>I’ve implemented a version of what I believe you are doing and<br>
<code>Linear.weight</code> converges to the weight vector used to generate<br>
the training data.</p>
<p>Here is the script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (2021)

a = torch.randn (3)
b = torch.randn (1)

model = torch.nn.Linear (3, 1)

nOpt = 100
nPrint = 10
opt = torch.optim.SGD (model.parameters(), lr = 0.1)
loss_fn = torch.nn.MSELoss()

for  i in range (nOpt):
    x = torch.randn (3)
    t = a @ x + b
    p = model (x)
    loss = loss_fn (p, t)
    if  i % (nOpt / nPrint) == 0  or  (i + 1) == nOpt:
        print ('loss =', loss)
    opt.zero_grad()
    loss.backward()
    opt.step()

print ('a =', a)
print ('model.weight =', model.weight.data)
print ('b =', b)
print ('model.bias =', model.bias.data)
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.7.1
loss = tensor(81.5180, grad_fn=&lt;MseLossBackward&gt;)
loss = tensor(0.8514, grad_fn=&lt;MseLossBackward&gt;)
loss = tensor(0.0013, grad_fn=&lt;MseLossBackward&gt;)
loss = tensor(5.5018e-06, grad_fn=&lt;MseLossBackward&gt;)
loss = tensor(1.4914e-05, grad_fn=&lt;MseLossBackward&gt;)
loss = tensor(7.4943e-07, grad_fn=&lt;MseLossBackward&gt;)
loss = tensor(2.5289e-07, grad_fn=&lt;MseLossBackward&gt;)
loss = tensor(1.3833e-07, grad_fn=&lt;MseLossBackward&gt;)
loss = tensor(8.2538e-10, grad_fn=&lt;MseLossBackward&gt;)
loss = tensor(2.2737e-13, grad_fn=&lt;MseLossBackward&gt;)
loss = tensor(1.4211e-12, grad_fn=&lt;MseLossBackward&gt;)
a = tensor([ 2.2871,  0.6413, -0.8615])
model.weight = tensor([[ 2.2871,  0.6413, -0.8615]])
b = tensor([-0.3649])
model.bias = tensor([-0.3649])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: Saif_Ali_Khan
cooked: <p>Thanks a lot for your reply and implementation. I copied your code and you’re right, it does work. But when i merge your code with mine (The only difference is that i’m training whole x which is of shape [125 x 3]). In the end, i print out rmse and last 15 output i.e y[110:] (you can notice they’re the same, but still weights are different. This time even bias is also different)</p>
<pre><code class="lang-auto">x_path = Path('../datasets/') / 'synthetic_datasets/1/X.npy'
x = np.load(x_path)
x = torch.from_numpy(x)
x = x.type(torch.FloatTensor)

_ = torch.manual_seed (2021)
a = torch.randn (3)
b = torch.randn (1)
model = torch.nn.Linear (3, 1)
t = a @ x.T + b
t = torch.reshape(t, (len(t),1))
print(t.shape)
nOpt = 10000
optimizer = torch.optim.SGD (model.parameters(), lr = 0.01)
loss_fn = torch.nn.MSELoss()

for  i in range (nOpt):
    y_pred = model(x)
    optimizer.zero_grad()
    y_pred = model(x)
    
    loss = loss_fn(y_pred, t)
    loss.backward()
    optimizer.step()
prediction(model, x, t, 'regression')
print ('a =', a)
print ('b =', b)
for param in model_new.parameters():
    print(param)
</code></pre>
<p>Output:</p>
<blockquote>
<p>torch.Size([125, 3])<br>
1.7.1<br>
torch.Size([125, 1])<br>
tensor([[10.0661],<br>
[ 9.2046],<br>
[ 8.3431],<br>
[ 7.4816],<br>
[ 6.6201],<br>
[10.7075],<br>
[ 9.8460],<br>
[ 8.9845],<br>
[ 8.1230],<br>
[ 7.2614],<br>
[11.3488],<br>
[10.4873],<br>
[ 9.6258],<br>
[ 8.7643],<br>
[ 7.9028]]) [[10.0661125]<br>
[ 9.204607 ]<br>
[ 8.3431015]<br>
[ 7.481596 ]<br>
[ 6.6200905]<br>
[10.7074585]<br>
[ 9.845953 ]<br>
[ 8.9844475]<br>
[ 8.122942 ]<br>
[ 7.2614365]<br>
[11.348804 ]<br>
[10.487299 ]<br>
[ 9.625793 ]<br>
[ 8.764288 ]<br>
[ 7.902782 ]]<br>
Rmse: 5.683192e-06<br>
a = tensor([ 2.2871,  0.6413, -0.8615])<br>
b = tensor([-0.3649])<br>
Parameter containing:<br>
tensor([[ 4.0516, -9.3466,  6.8186]], requires_grad=True)<br>
Parameter containing:<br>
tensor([1.1346], requires_grad=True)</p>
</blockquote>

username: KFrank
cooked: <p>Hi Saif!</p>
<p>Probably the most practical next step for you to debug your issue<br>
would be for you to go through your code, systematically trimming<br>
out everything you can, while still preserving the issue.</p>
<p>Whittle it down to a short, self-contained, runnable script that<br>
reproduces your issue.  Not only is this a good approach to debugging;<br>
it lowers the barrier to people on the forum being able to help you<br>
if you have further questions.</p>
<aside class="quote no-group quote-modified" data-username="Saif_Ali_Khan" data-post="3" data-topic="118753" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saif_ali_khan/48/37319_2.png" class="avatar"> Saif_Ali_Khan:</div>
<blockquote>
<pre><code class="lang-auto">optimizer = torch.optim.SGD (model.parameters(), lr = 0.01)
...
prediction(model, x, t, 'regression')
print ('a =', a)
print ('b =', b)
for param in model_new.parameters():
    print(param)
</code></pre>
</blockquote>
</aside>
<p>We have no way to tell whether this is right or wrong.</p>
<p>You optimize <code>model</code>, but then you call something called <code>prediction()</code><br>
and print out the <code>parameters()</code> of something called <code>model_new</code>.</p>
<p>Who knows what <code>model_new</code> is or what its <code>parameters()</code> should<br>
be equal to?</p>
<p>Good luck.</p>
<p>K. Frank</p>