Performing mini-batch gradient descent or stochastic gradient descent on a mini-batch

Joseph_Santarcangelo: <p>Hello, I have created a data-loader object, I set the parameter batch size equal to five and I run the following code. I would like some clarification, is the following code performing mini-batch gradient descent or stochastic gradient descent on a mini-batch.</p>
<pre><code class="lang-auto">from torch import nn
import torch
import numpy as np
import matplotlib.pyplot as plt
from torch import nn,optim
from torch.utils.data import Dataset, DataLoader
class Data(Dataset):
    def __init__(self):
        self.x=torch.arange(-3,3,0.1).view(-1, 1)
        self.y=-3*self.x+1+0.1*torch.randn(self.x.size())
        self.len=self.x.shape[0]
    def __getitem__(self,index):    
            
        return self.x[index],self.y[index]
    def __len__(self):
        return self.len
class linear_regression(nn.Module):
    def __init__(self,input_size,output_size):
        super(linear_regression,self).__init__()
        self.linear=nn.Linear(input_size,output_size)
    def forward(self,x):
        yhat=self.linear(x)
        return yhat

class linear_regression(nn.Module):
    def __init__(self,input_size,output_size):
        super(linear_regression,self).__init__()
        self.linear=nn.Linear(input_size,output_size)
    def forward(self,x):
        yhat=self.linear(x)
        return yhat
model=linear_regression(1,1)
optimizer = optim.SGD(model.parameters(), lr = 0.01)
criterion = nn.MSELoss()
dataset=Data()
trainloader=DataLoader(dataset=dataset,batch_size=5)
LOSS=[]

n=1;
for epoch in range(5):
    for x,y in trainloader:
        yhat=model(x)
        loss=criterion(yhat,y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        LOSS.append(loss)
</code></pre>
<p>i</p>

ptrblck: <aside class="quote no-group" data-username="Joseph_Santarcangelo" data-post="1" data-topic="21235">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joseph_santarcangelo/48/12526_2.png" class="avatar"> Joseph_Santarcangelo:</div>
<blockquote>
<p>mini-batch gradient descent or stochastic gradient descent on a mini-batch</p>
</blockquote>
</aside>
<p>I’m not sure what stochastic gradient descent on a mini-batch is, since as far as my understanding is, stochastic gradient descent uses only one sample by definition.<br>
Because you use a batch size of 5, your code applies mini-batch gradient descent.</p>

Joseph_Santarcangelo: <p>Thanks for the response, my confusion comes from the fact that the code included below calculates SGD by taking a tensor that is the size of my training set and performing at update one sample at the time. i.e for each iteration of the loop an update of SGD is performed for every sample in the tensor<br>
In the initial code in the second nested loop, the data loader provides a tensor the size of my mini-batch, then how come it does not follow the same procedure as above, i.e for each iteration performing an update on each sample in the tensor.</p>
<pre><code class="lang-auto">from torch import nn
import torch
import numpy as np
import matplotlib.pyplot as plt
from torch import nn,optim
from torch.utils.data import Dataset, DataLoader
class Data(Dataset):
   def __init__(self):
       self.x=torch.arange(-3,3,0.1).view(-1, 1)
       self.y=-3*self.x+1+0.1*torch.randn(self.x.size())
       self.len=self.x.shape[0]
   def __getitem__(self,index):    
           
       return self.x[index],self.y[index]
   def __len__(self):
       return self.len
class linear_regression(nn.Module):
   def __init__(self,input_size,output_size):
       super(linear_regression,self).__init__()
       self.linear=nn.Linear(input_size,output_size)
   def forward(self,x):
       yhat=self.linear(x)
       return yhat

class linear_regression(nn.Module):
   def __init__(self,input_size,output_size):
       super(linear_regression,self).__init__()
       self.linear=nn.Linear(input_size,output_size)
   def forward(self,x):
       yhat=self.linear(x)
       return yhat
model=linear_regression(1,1)
optimizer = optim.SGD(model.parameters(), lr = 0.01)
criterion = nn.MSELoss()
dataset=Data()


x,y=dataset[:]
LOSS=[]

n=1;
for epoch in range(5):
       yhat=model(x)
       loss=criterion(yhat,y)
       optimizer.zero_grad()
       loss.backward()
       optimizer.step()
       LOSS.append(loss)

</code></pre>

ptrblck: <p>In your current code snippet you are assigning <code>x</code> to your complete dataset, i.e. you are performing batch gradient descent.<br>
In the former code your <code>DataLoader</code> provided batches of size 5, so you used mini-batch gradient descent.<br>
If you use a dataloader with <code>batch_size=1</code> or slice each sample one by one, you would be applying stochastic gradient descent.</p>
<p>The averaged or summed loss will be computed based on your batch size. E.g. if your batch size is 5, and you are using your criterion with its default setting <code>size_average=True</code>, the average or the losses for each sample in the batch will be calculated and used to compute the gradients.</p>

Joseph_Santarcangelo: <p>Thanks for clearing that up! I was confused because I could not see the explicit relationship between the data loader object and the optimizer.</p>

jon: <p>Sorry to necro this post, but it bothers me: why is it named a stochastic gradient descent optimizer even if we are doing full batch gradient descent? Is there any sort of difference between what vanilla gradient descent would do vs. the sgd optimizer would do when we run it with the full batch?</p>

jon: <p>Ahh, actually sorry, it’s just a mismatch in terminology. The SGD optimizer <em>is</em> vanilla gradient descent (i.e. literally all it does is subtract the gradient * the learning rate from the weight, as expected). See here: <a href="https://discuss.pytorch.org/t/how-sgd-works-in-pytorch/8060">How SGD works in pytorch</a></p>

vinaykumar2491: <aside class="quote no-group" data-username="Joseph_Santarcangelo" data-post="1" data-topic="21235">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joseph_santarcangelo/48/12526_2.png" class="avatar"> Joseph_Santarcangelo:</div>
<blockquote>
<p>LOSS.append(loss)</p>
</blockquote>
</aside>
<p>If you want to store the whole computational graph then its okay to use <code>LOSS.append(loss)</code>;  but if you are just looking to store the loss value then use <code>LOSS.append(loss.item())</code>.</p>
<aside class="quote no-group" data-username="Joseph_Santarcangelo" data-post="1" data-topic="21235">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joseph_santarcangelo/48/12526_2.png" class="avatar"> Joseph_Santarcangelo:</div>
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">LOSS=[]

n=1;
for epoch in range(5):
    for x,y in trainloader:
        yhat=model(x)
        loss=criterion(yhat,y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        LOSS.append(loss)
</code></pre>
</blockquote>
</aside>
<p>Also, shouldn’t you be accumulating the loss (for each batch) within each epoch and then appending it to loss array like this (below):</p>
<pre data-code-wrap="python"><code class="lang-python">LOSS=[]
...
for epoch in range(5):
    running_loss = 0     #accumulates loss of each batch
    for x,y in trainloader:
        # do something......
        loss.backward()
        optimizer.step()
        running_loss += loss.item()   # accumulating loss
    
    LOSS.append(running_loss)   # saving final loss for each epoch
</code></pre>

Joseph_Santarcangelo: <p>Thanks <a href="https://discuss.pytorch.org/u/vinaykumar2491">Vinay Kumar</a>, I found out about Item it’s super useful.<br>
For the loss, I was just calculating the average loss for each batch.</p>

Jaddi_abd_elaziz: <p>just a question, if we write running_loss += loss.item() after an update of the parameters, I think like that we calculate these:<br>
for epoch in range(5):<br>
running_loss = 0     <span class="hashtag">#accumulates</span> loss of each batch<br>
for x,y in trainloader:<br>
running_loss += loss.item() &lt;=&gt;   F*  = F(x,y, w_{0})+ F(x,y, w_{1})+F(x,y, w_{2})+…</p>
<p>in an epoch  with F isobjective function.<br>
however our objective function defined by :<br>
F(data,w) = F(x,y, w)+ F(x,y,w)+F(x,y,w)+…  for (x,y) in data</p>