Difference between batch_input and "for loop"

jiang_ix: <p>I have a question.<br>
For pytorch0.3.0s.<br>
(I)I use the following code:<br>
total_loss = net([batch_size, input_size]) <span class="hashtag">#means</span> all batch_size train samples<br>
total_loss.backward()<br>
optimizer.step()<br>
(II)I use the following code:<br>
total_loss = 0<br>
for i in range(batch_size):<br>
loss = net([(i)input_size]) <span class="hashtag">#means</span> i train sample<br>
totle_loss += loss/batch_size<br>
total_loss.backward()<br>
optimizer.step()</p>
<p>Can you tell me any differences between (I) and (II), that means I use “for loop” to throw input to neural network step by step, and I want to know the final results is different? or maybe only training time is longer.</p>

ptrblck: <p>Usually you are using a criterion to calculate the loss. In the default setting the loss is averaged by the batch size, so that you don’t have to scale it.<br>
In your example the model returns directly the loss, which is also valid, but I’m not sure, if you need to scale it.<br>
You should definitely check if it’s right.</p>
<p>Back to your original question.<br>
In your first example you are using batch gradient descent, i.e. all training samples are used for one weight update. Each iteration is thus one epoch.<br>
This procedure might be OK, but it has been shown, that mini-batch gradient descent generalizes better in a lot of use cases.</p>
<p>I’m not sure, if your second example uses only one single example or a few.<br>
If it’s the former, you are applying stochastic gradient descent. You will have much more weight updates in your epoch, since each sample will give you gradients.<br>
The latter case is called mini-batch gradient descent and is usually the way to go.<br>
You use a specific batch size and update your model using the loss of this batch.<br>
Some layers like <code>BatchNorm</code> need a certain amount of samples to accurately estimate the running statistics, i.e. mean and var. It’s usually approx. 64 samples, but might differ based on your problem.</p>
<p>The final results of the training will most likely differ! The batch size is one hyperparameter which can change your model’s performance.</p>

jiang_ix: <p>Thanks a lot.<br>
And What I mean is for mini-batch gradient descent, My first code is to use batch_size examples, and I use the criterion to calculate average loss.<br>
My second code is also to use batch_size examples. and the difference is that every time i calculate loss of example of batch_size examples, (means loss_1, loss_2, and loss_batch_size), finally, I add them and average. Then I use backward and optimizer to update.<br>
If I do not use batch normalization or some other which may influence the loss. May be the result is the same?</p>
<p>Thanks for your reply</p>

jiang_ix: <p>Because the “for loop” code also calculate the average loss of batch_size examples. the gradient is depend on loss. the loss between code1 and code2 is the same? so the gradient is the same? I guess.</p>
<p>Thanks</p>

ptrblck: <p>Do you use the same batch size to calculate the different losses?<br>
If not, I’m not sure about the effects of this approach.</p>
<p>However, even without <code>BatchNorm</code>, your results will most likely depend on the batch size.<br>
Maybe the results will be approx. the same, but the loss curve will look different.</p>
<p>Your code is not formatted properly, but it seems you are trying to do the following:</p>
<pre><code class="lang-auto">criterion = nn.MSELoss()
model = nn.Linear(10, 1)
x = torch.randn(10, 10)
y = torch.randn(10, 1)

# Use whole batch
output1 = model(x)
loss1 = criterion(output1, y)
loss1.backward()
print(model.weight.grad)

model.zero_grad()

# Use mini-batches
loss2 = 0
for x_, y_ in zip(x, y):
    x_.unsqueeze(0)
    y_.unsqueeze(0)
    
    output2 = model(x_)
    loss2 += criterion(output2, y_)

# Scale the loss, since we accumulated it
loss2 /= len(x)
loss2.backward()
print(model.weight.grad)
</code></pre>
<p>Let me know, if this is your use case.</p>

jiang_ix: <p>Yeah, You are right, And it means the gradient is the same? I run it and it does</p>
<p>Thanks for your reply</p>

ptrblck: <p>Yes, the gradient should be the same.<br>
In my example I used single example in the for loop, so the scaling was straightforward.<br>
If you use a batch size &gt; 1, you would have to divide by the number of batches, if you use an averaged loss.<br>
Your loss might differ, if your batch size is not constant, because the last batch is smaller for example.<br>
Using a <code>DataLoader</code>, you could just drop the last batch, if it’s smaller, using <code>drop_last=True</code>.</p>

jiang_ix: <p>OK, the batch_size is the same. And I have know it does work if I use the “for loop”.<br>
Thanks a lot.</p>

seyeeet: <p>I thought we cannot do inplace loss update like <code>loss2 += criterion(output2, y_)</code> or am I wrong and I can do it in loop without any issue (I am not considering the speed efficiency)?</p>
<p>also just out of curiosity why the second case that you wrote is different from this</p>
<pre><code class="lang-auto"># Use mini-batches
loss2 = 0
for x_, y_ in zip(x, y):
    x_.unsqueeze(0)
    y_.unsqueeze(0)
    
    output2 = model(x_)
    loss2 = loss2 + criterion(output2, y_)
    loss2 /= len(x)
    loss2.backward(retain_graph=True)
# Scale the loss, since we accumulated it

print(model.weight.grad)
``</code></pre>

ptrblck: <p>You can add the losses inplace as long as no errors are raised, but would accumulate the computation graphs. <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">This post</a> explains it in more detail.</p>
<p>The losses wouldn’t be equal, since you are scaling <code>loss2</code> by <code>len(x)</code> in each iteration.</p>