Pytorch Tensor and Numpy array show different result with same code

username: 20am847
cooked: <p>Hi, I was implementing a simple differentiation with Pytorch Tensor, and the results were strange compared to Numpy array. When I used Pytorch Tensor, I checked that the <strong>tmp_val value</strong> changes after calculating multi_func(x), while Numpy array did not.</p>
<p>Why is this happening and how do I modify the code to get the same results as the Numpy array?</p>
<pre><code class="lang-auto">def multi_func(x):
    return x[0]**2 + x[1]**2 


def derivative(f, x):
    h = 1e-4 # 0.0001
    grad = torch.zeros_like(x) 
    
    for idx in range(x.size()[0]):
        tmp_val = x[idx]
        
        # f(x+h)
        x[idx] = float(tmp_val) + h
        fxh1 = f(x)

        print(tmp_val)
        
        # f(x-h) 
        x[idx] = tmp_val - h 
        fxh2 = f(x) 
        
        grad[idx] = (fxh1 - fxh2) / (2*h)
        x[idx] = tmp_val
        
    return grad

print(derivative(multi_func, torch.tensor([3.0,4.0])))
</code></pre>
<p>tensor(3.0001)<br>
tensor(4.0001)<br>
tensor([2.9945, 4.0054])</p>
<p>=========================================================</p>
<pre><code class="lang-auto">def multi_func(x):
    return x[0]**2 + x[1]**2 

def derivative(f, x):
    h = 1e-4 # 0.0001
    grad = np.zeros_like(x) 
    
    for idx in range(x.size):
        tmp_val = x[idx]
        
        # f(x+h) 
        x[idx] = float(tmp_val) + h
        fxh1 = f(x)
        
        print(tmp_val)
        
        # f(x-h) 
        x[idx] = tmp_val - h 
        fxh2 = f(x) 
        
        grad[idx] = (fxh1 - fxh2) / (2*h)
        x[idx] = tmp_val
        
    return grad

print(derivative(multi_func, np.array([3.0,4.0])))
</code></pre>
<p>3.0<br>
4.0<br>
[6. 8.]</p>

username: Youyoun
cooked: <aside class="quote no-group" data-username="20am847" data-post="1" data-topic="127547">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/20am847/48/12561_2.png" class="avatar"> 20am847:</div>
<blockquote>
<pre><code class="lang-auto">       tmp_val = x[idx]
</code></pre>
</blockquote>
</aside>
<p>You are assigning x[idx] to tmp_val, but it’s not a copy, it’s simply a <a href="https://pytorch.org/docs/stable/tensor_view.html" rel="noopener nofollow ugc">view</a> of your tensor at that index. this means that tmp_val is referencing the same memory space as x[idx], which means that changing x[idx] changes tmp_val.</p>
<aside class="quote no-group" data-username="20am847" data-post="1" data-topic="127547">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/20am847/48/12561_2.png" class="avatar"> 20am847:</div>
<blockquote>
<p><code>x[idx] = float(tmp_val) + h</code></p>
</blockquote>
</aside>
<p>You’re adding h=1e-4 to x[idx], which means that tmp_val is now pointing to the value x[idx]+h which is 3.0001 and 4.0001</p>
<p>Numpy index selection on the other hand doesn’t seem to give a view, but instead a copy (I thought it also provides a view, but maybe for one item it gives the copy).</p>
<p>This is why your code with numpy works as intended. To remedy to the problem in pytorch, simply use  <code>tmp_val = x[idx].clone()</code> to copy the tensor. This should yield the result you wanted.</p>
<p>One more thing: numpy by default uses double (float64) for float values, while pytorch uses float (float32). This means that your operations might not give the same results in terms of precision. You can set the default precision as per <a href="https://pytorch.org/docs/stable/generated/torch.set_default_dtype.html" rel="noopener nofollow ugc">this</a></p>
<p>Hope this answers your question.</p>

username: 20am847
cooked: <p>Awesome!!! Thanks a lot <a class="mention" href="/u/youyoun">@Youyoun</a> !! The question has been completely resolved. Why didn’t I think that the mechanism would be different between the Pytorch and Numpy…? Thank you so much. I got a lot of inspiration.</p>