Getting incorrect subtraction result

Hiba_Ahsan: <p>I have a matrix X of shape (N,D) ans a vector mu of shape(D,K). I tried subtracting one column of mu from one row of X and am getting the wrong answer. When I convert it to numpy array, I am getting the right answer. What am I doing wrong?</p>
<p>print(X[0,:])<br>
print(mu[:,0])<br>
print(X[0,:]-mu[:, 0])<br>
print(X[0,:].data.numpy()-mu[:, 0].data.numpy())</p>
<p>tensor([-2.4682,  0.9872, -0.4661,  1.7443,  0.5375, -0.5948,  2.2457, -0.2980,<br>
1.2467,  0.2932], dtype=torch.float64)<br>
tensor([-1.5256, -0.6092, -0.7773,  0.4676,  0.8657, -0.1759, -0.7981, -0.4370,<br>
0.2152, -0.8696], grad_fn=)<br>
tensor([ -9.4257e-01,   1.1631e+00,  -4.6614e-01,   1.7443e+00,   5.3751e-01,<br>
-5.9480e-01,   2.2457e+00, -1.1146e+171,   2.1892e+00,   8.8797e-01],<br>
dtype=torch.float64, grad_fn=)<br>
[-0.94256972  1.59635256  0.31117838  1.27669038 -0.32818301 -0.41886666<br>
3.0437956   0.13902091  1.03149757  1.16275023]</p>

ptrblck: <p>Which PyTorch version are you using?<br>
Is this error reproducible? If so, could you please upload the tensors somewhere so that we could have a look?</p>

Hiba_Ahsan: <p>This was happening when I read data from a numpy array to a Variable X:<br>
X = Variable(torch.from_numpy(X))<br>
mu = Variable(torch.randn(D, K), requires_grad=True)</p>
<p>But when I changed it to X = torch.FloatTensor(X), subtraction was fine. What is the reason for this?</p>

ptrblck: <p><code>Variables</code> are deprecated since version <code>0.4.0</code>, so you shouldn’t use them anymore.<br>
Just out of curiosity, do you have a code snippet to reproduce this issue?</p>

Hiba_Ahsan: <p>X is a numpy array of shape (N,D)<br>
X = Variable(torch.from_numpy(X))<br>
mu = Variable(torch.randn(D, K), requires_grad=True)</p>
<p>print(X[0,:]-mu[:,0])</p>

beaupreda: <p>I think the problem comes from the different data types of <code>X</code> (float64) and <code>mu</code> (float32). When you do <code>X = torch.FloatTensor(X)</code>, you change the type of <code>X</code> to float32, hence the correct result. If you specify the type of <code>mu</code> to float64 and do not change <code>X</code>, everything should also work. As <a class="mention" href="/u/ptrblck">@ptrblck</a> said, Variable is deprecated so here is an example of what I have with PyTorch 1.0</p>
<pre><code class="lang-auto">    N = 10
    D = 5
    K = 2
    X = np.random.rand(N, D)
    X = torch.from_numpy(X)
    mu = torch.randn(D, K, dtype=torch.float64)
    res = X[0, :] - mu[:, 0]
    print(X[0, :])
    print(mu[:, 0])
    print(res)
</code></pre>

ptrblck: <p>That’s a good point, but the results of the PyTorch and numpy subtraction have a huge difference:</p>
<pre><code class="lang-python">a = torch.tensor([ -9.4257e-01, 1.1631e+00, -4.6614e-01, 1.7443e+00, 5.3751e-01,
                  -5.9480e-01, 2.2457e+00, -1.1146e+171, 2.1892e+00, 8.8797e-01],
                  dtype=torch.float64)
b = torch.tensor([-0.94256972, 1.59635256, 0.31117838, 1.27669038, -0.32818301,
                  -0.41886666, 3.0437956, 0.13902091, 1.03149757, 1.16275023],
                  dtype=torch.float64)

print(a - b)
&gt; tensor([ -2.8000e-07,  -4.3325e-01,  -7.7732e-01,   4.6761e-01,   8.6569e-01,
         -1.7593e-01,  -7.9810e-01, -1.1146e+171,   1.1577e+00,  -2.7478e-01],
       dtype=torch.float64)
</code></pre>
<p>Also, the PyTorch output looks a bit strange, as it contains <code>-1.1146e+171</code>.</p>
<p><a class="mention" href="/u/hiba_ahsan">@Hiba_Ahsan</a> Could you upload the input tensors somewhere, so that we can have a look?</p>

Hiba_Ahsan: <p>This is the kind of weird output I was getting as well. Huge powers. I don’t think I can upload data for X since it is someone else’s private dataset <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"> X is just a float64 type numpy array. I generated mu using torch.randn. You are able to reproduce the issue without my data anyway!</p>

ptrblck: <p>Unfortunately I cannot reproduce this issue.<br>
Neither with tensors (in FP32 and FP64) not with the deprecated <code>Variables</code> type. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

Hiba_Ahsan: <blockquote>
<blockquote>
<blockquote>
<p>X = Variable(torch.from_numpy(xtr))<br>
mu = Variable(torch.randn(10, 5), requires_grad=True)<br>
print(X[0,:]-mu[:,0])<br>
tensor([ -2.6652e+00,   2.0585e+00,  -4.6961e-01,   1.7443e+00, -4.7861e+180,<br>
-6.3082e+199, -1.6779e+243, -9.3102e+242,   1.2467e+00,   2.9316e-01],<br>
dtype=torch.float64, grad_fn=)<br>
print(X[0,:])<br>
tensor([-2.4682,  0.9872, -0.4661,  1.7443,  0.5375, -0.5948,  2.2457, -0.2980,<br>
1.2467,  0.2932], dtype=torch.float64)<br>
print(mu[:,0])<br>
tensor([ 0.1971,  1.0803, -0.8145,  0.9733, -1.4811, -1.0713,  0.3086,  0.6137,<br>
-0.6779,  0.3576], grad_fn=)<br>
print(X[0,:].data.numpy()-mu[:,0].data.numpy())<br>
[-2.66522103 -0.09317447  0.34836983  0.77099426  2.01856725  0.47648397<br>
1.93714274 -0.91173426  1.92456246 -0.06439294]</p>
</blockquote>
</blockquote>
</blockquote>

ptrblck: <pre><code class="lang-python">xtr = np.random.randn(5, 10)
xtr[0] = [-2.4682, 0.9872, -0.4661, 1.7443, 0.5375, -0.5948, 2.2457, -0.2980, 1.2467, 0.2932]
X = Variable(torch.from_numpy(xtr))
mu = Variable(torch.randn(10, 5), requires_grad=True)
mu[:, 0] = torch.tensor([ 0.1971, 1.0803, -0.8145, 0.9733, -1.4811, -1.0713, 0.3086, 0.6137, -0.6779, 0.3576])

res1 = X[0,:]-mu[:,0]
res2 = X[0,:].data.numpy()-mu[:,0].data.numpy()
print((res1.detach().numpy()-res2))
&gt; [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
</code></pre>

Hiba_Ahsan: <p>I don’t understand <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"> My version is 1.3. Also what is the snippet with tensors a and b that you had posted earlier that had  a huge difference?</p>

ptrblck: <p>For <code>a</code> and <code>b</code> I just initialized using your result to visualize the error in the results.<br>
So it wasn’t a reproduction of the bug, but rather just the information how large the difference in your code is.</p>

Hiba_Ahsan: <pre><code class="lang-auto">xtr = np.random.randn(5, 10)
xtr[0] = [-2.4682, 0.9872, -0.4661, 1.7443, 0.5375, -0.5948, 2.2457, -0.2980, 1.2467, 0.2932]
X = Variable(torch.from_numpy(xtr))
mu = Variable(torch.randn(10, 5), requires_grad=True)
mu[:, 0] = torch.tensor([ 0.1971, 1.0803, -0.8145, 0.9733, -1.4811, -1.0713, 0.3086, 0.6137, -0.6779, 0.3576])
&gt;&gt;&gt; X[0,:]-mu[:,0]
tensor([ -2.6653e+00,   2.0585e+00,  -4.6610e-01, -1.2407e+214,   5.3750e-01,
         -5.9480e-01,   2.2457e+00, -1.7297e+156,  -1.7331e+97,   2.9320e-01],
       dtype=torch.float64, grad_fn=&lt;SubBackward0&gt;)

&gt;&gt;&gt; X[0,:].data.numpy()-mu[:,0].data.numpy()
array([-2.6653    , -0.09309997,  0.34839997,  0.77100002,  2.01859996,
        0.47650003,  1.93709999, -0.91169997,  1.92460002, -0.0644    ])
</code></pre>
<p>I followed the same steps and am still facing this issue.</p>

ptrblck: <p>Great, it seems you are able to reproduce this issue!<br>
Back to my initial question: which PyTorch version are you currently using?</p>

Hiba_Ahsan: <p>My PyTorch version is 1.3.0.</p>

ptrblck: <p>Thanks for the version number!<br>
I could reproduce a high error (although not in the same value range).</p>
<p><a class="mention" href="/u/alband">@albanD</a> Could this be related to <a href="https://github.com/pytorch/pytorch/issues/28010" rel="nofollow noopener">28010</a>?</p>
<p><a class="mention" href="/u/hiba_ahsan">@Hiba_Ahsan</a> Could you update please to PyTorch <code>1.3.1</code>, as I cannot reproduce it in the latest release?</p>

albanD: <p>I think this is exactly the problem in 28010 and it was fixed in PR <span class="hashtag">#28231</span>. This should be fixed in <code>1.3.1</code>. Can you test and let us know <a class="mention" href="/u/hiba_ahsan">@Hiba_Ahsan</a>?</p>

Hiba_Ahsan: <p>I am trying to update but it says “Requirement already up-to-date: torch in \anaconda3\lib\site-packages (1.3.0)”.</p>

albanD: <p>Hi,</p>
<p>The packages are properly updated online: <a href="https://anaconda.org/pytorch/pytorch" rel="nofollow noopener">https://anaconda.org/pytorch/pytorch</a><br>
You might want to upgrade conda to make sure you can get the latest libraries?</p>