How does batch's element are processed by Pytorch?

username: Torcione
cooked: <p>I have a generic network without random element in his structure (e.g. no dropout) so that if I forward a given image input through the network, I put gradient to zero and repeat again the forward with the same image input I get the same result (same gradient vector, output,…)<br>
Now let’s say that we have a batch of N elements (data, label) and I perform the following experiment:</p>
<ol>
<li>forward the whole batch and store the gradient vector (using <code>reduction='sum'</code> in my criterion), use backward to generate the corresponding gradient, save it in a second object (that we’ll refer to as <code>Batch_Grad</code>)</li>
</ol>
<pre><code class="lang-auto">output = model(data)
loss = criterion(output,torch.reshape(label, (-1,)))
loss.backward() 

Batch_Grad= []
for p in model.parameters():     
    Batch_Grad.append(p.grad.clone()) 

</code></pre>
<ol start="2">
<li>reset the gradient</li>
</ol>
<pre><code class="lang-auto">optimizer.zero_grad()
</code></pre>
<ol start="3">
<li>repeat the first point giving in input batch’s elements one by one and collect after each backward the corresponding element’s gradient (resetting the gradient every time after that)</li>
</ol>
<pre><code class="lang-auto">for i in range(0, len(label)):
    #repeat the procedure of point 1. for each data[i] input
    #...
    optimizer.zero_grad()
</code></pre>
<ol start="4">
<li>
<p>Sum up togheter gradient vectors of the previous point corresponding to each element of the given batch in a single object (that we’ll refer to as Single_Grad)</p>
</li>
<li>
<p>compare the objects of point 4. and 1. (Batch_Grad and Single_Grad)</p>
</li>
</ol>
<p>Following the above procedure I find that tensor from point 1. and 5. are equal only if the batch size (N) is equal to 1, but they are different for N&gt;1.</p>
<p>With the method of point 3. and 4. I’m manually summing gradients associated to single image propagation (which as pointed in the above comment are equals to the ones calculated automatically by SGD, with N=1). Since automatic SGD approach (point 1.)is also expected to perform the same sum:<br>
<strong>Why do I observe this difference?</strong></p>
<p><strong>EDIT</strong><br>
From the answer to this <a href="https://stackoverflow.com/questions/70288949/how-does-batch-s-element-are-processed-by-pytorch" rel="noopener nofollow ugc">post</a> I understood that a possible reason is that proceeding with point 1. I could not be doing a mini batch forward but using, instead, an iterative method where the element of the batch are notprocessed in parallel but one after the other.<br>
I want to understand if it is my case; since I’m using <code>torch.optim.SGD()</code> does it perform mini-batch or the iterative method?</p>

username: ptrblck
cooked: <p>I don’t think the answer from the cross-post is correct, as your code doesn’t show any <code>optimizer.step()</code> usage so I assume you are purely comparing the gradients computed via using the entire dataset vs. single samples.<br>
If so, I would assume the final difference would show a relative error of ~1e-5 or ~1e-6, which is expected due to the limited floating point precision and which is caused by a different order of operations.</p>

username: Torcione
cooked: <p>Thank you for the answer! I have the same doubt indeed.</p>
<blockquote>
<p>I assume you are purely comparing the gradients computed via using the entire dataset vs. single samples.</p>
</blockquote>
<p>Exactly</p>
<p>if I consider a batch size of 2 element, for example, I get, for each single componenet of the gradient, an error of ~1e-5 or ~1e-6, as you predicted.<br>
May I ask you how did you guess the order of magnitude of the difference in the 2 cases? (Sorry for the stupid question)</p>
<p>Is it possible to pass to a double precision for the gradients component in order to verify that this is the right source of error? (in that case the order of magnitude of the difference should decrease and, in particular, it should be possible to predict the new order of magnitude as you just did for the floating precision)</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Torcione" data-post="3" data-topic="138927">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/torcione/48/52091_2.png" class="avatar"> Torcione:</div>
<blockquote>
<p>May I ask you how did you guess the order of magnitude of the difference in the 2 cases? (Sorry for the stupid question)</p>
</blockquote>
</aside>
<p>That’s the typical relative error in <code>float32</code>, so it was just my guess as you would see it in a lot of other operations as well.</p>
<aside class="quote no-group" data-username="Torcione" data-post="3" data-topic="138927">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/torcione/48/52091_2.png" class="avatar"> Torcione:</div>
<blockquote>
<p>Is it possible to pass to a double precision for the gradients component in order to verify that this is the right source of error? (in that case the order of magnitude of the difference should decrease and, in particular, it should be possible to predict the new order of magnitude as you just did for the floating precision)</p>
</blockquote>
</aside>
<p>Yes, absolutely! If you are using <code>float64</code> you should see a relative error of ~1e-13 or so (my gut feeling is way worse for <code>float64</code> <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> ).</p>

username: Torcione
cooked: <p>How can I set a <code>float64</code> precision for the gradient tensors computed by autograd (during the backward step)?</p>

username: ptrblck
cooked: <p>Call <code>.double()</code> on the model and input, which will transform all parameters to <code>float64</code>:</p>
<pre><code class="lang-python">model.double()
x = x.double()

# forward/backward pass as before
</code></pre>

username: Torcione
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="138927">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Call <code>.double()</code> on the model and input, which will transform all parameters to <code>float64</code>:</p>
</blockquote>
</aside>
<p>Thank you for the clarification!<br>
for the input should I call like this:</p>
<pre><code class="lang-auto">for x,label in train_loader:
    x = x.double()
    x = x.to(device)
    label = label.to(device) 
    # forward/backward pass as before
</code></pre>
<p>or is there a way to specify the double calling directly during the dataset creation (for example using the transform option in torchvision.dataset)</p>

username: ptrblck
cooked: <p>Your approach for testing should be fine. If you really want to use <code>double</code> during the entire training, you could certainly check if moving the transformation to the <code>Dataset</code> (or keeping the data directly in <code>float64</code>) could work.</p>