Unable to use iter and next on dataloader to process all batches

sparshgarg23: <p>I was experimenting with the iter and next functionality to iterate through my dataloader.</p>
<p>When I train the model using iter,it seems that I am only processing one batch of the trainloader<br>
as shown in the below log</p>
<pre><code class="lang-auto">torch.Size([16, 3, 512, 512])
Epoch : 1 Train Loss : 0.001343 
torch.Size([16, 3, 512, 512])
Epoch : 2 Train Loss : 0.002819 
torch.Size([16, 3, 512, 512])
Epoch : 3 Train Loss : 0.004004 
torch.Size([16, 3, 512, 512])
Epoch : 4 Train Loss : 0.005313 
torch.Size([16, 3, 512, 512])
Epoch : 5 Train Loss : 0.006345 
torch.Size([16, 3, 512, 512])
Epoch : 6 Train Loss : 0.007257 
torch.Size([16, 3, 512, 512])
Epoch : 7 Train Loss : 0.008262 
torch.Size([16, 3, 512, 512])
Epoch : 8 Train Loss : 0.009080 
torch.Size([16, 3, 512, 512])
Epoch : 9 Train Loss : 0.010034 
torch.Size([16, 3, 512, 512])
Epoch : 10 Train Loss : 0.011135 
</code></pre>
<p>My training code is as follows</p>
<pre><code class="lang-auto">epochs_a=10
criterion=nn.L1Loss()
optimizer=torch.optim.Adam(model.parameters(),lr = lr)
iter_source=iter(train_loader)
train_loss=0.0
for i in range(epochs_a):
  model.train()
  optimizer.zero_grad()
  images=iter_source.next()
  image=images[0].to(device)
  print(image.shape)
  labels=images[1].to(device)
  logits=model(image)
  loss=criterion(logits,labels)
  loss.backward()
  optimizer.step()
  train_loss+=loss.item()
  print("Epoch : {} Train Loss : {:.6f} ".format(i+1, train_loss/len(train_loader)))
</code></pre>
<p>But when I use a simple enumerate or tqdm to iterate through the trainloader as shown in the below code</p>
<pre><code class="lang-auto">criterion=nn.L1Loss()
optimizer=torch.optim.Adam(model.parameters(),lr = lr)
def train_batch_loop(model,trainloader):
        
        train_loss = 0.0
        train_acc = 0.0
       
        

        for images,labels in tqdm(trainloader): 
            
            # move the data to CPU
            images = images.to(device)
            labels = labels.to(device)
            
            logits = model(images)
            loss = criterion(logits,labels)
            
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            
            train_loss += loss.item()
            
            
        return train_loss / len(trainloader)
epochs_a=10
for i in range(epochs_a):
  model.train()
  avg_train_loss=train_batch_loop(model,train_loader)
  print("Epoch : {} Train Loss : {:.6f} ".format(i+1, avg_train_loss))
</code></pre>
<p>Then I am able to go through all 400 batches ,and the training log looks like this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aac567ffba584b95dfffd1bcadbf6d6ddd4c10df.png" alt="train_log" data-base62-sha1="omI6SkmhpfNzWwm27KXqeP8gslV" width="572" height="210"></p>
<p>So what is the difference between the two codes,and how can I use iter and next to go through all 400 batches rather than just one single batch</p>

ptrblck: <p>Both approaches should work as seen here:</p>
<pre><code class="lang-python">dataset = TensorDataset(torch.randn(100, 1))
train_loader = DataLoader(dataset, batch_size=10)

iter_source =iter(train_loader)

for i in range(len(dataset)//10):
    images = iter_source.next()
    print('iter {}, shape {}'.format(i, images[0].shape))
  
for i, images in enumerate(train_loader):
    print('iter {}, shape {}'.format(i, images[0].shape))
</code></pre>

sparshgarg23: <p>Thanks but why is the error in the first part different from the error in the second in my code<br>
Is it because it’s only considering one batch at a time?</p>

ptrblck: <p>I don’t know and would need an executable code snippet to reproduce and debug the issue.<br>
Using random data works, so I guess your dataset length is not what you would expect.</p>

sparshgarg23: <p>the length of train loader is 400,and when i use tqdm it sort of works.Shouldn’t it work for iter also?<br>
Would you like the entire code snippet to reproduce the error</p>

ptrblck: <p>Yes, it should work and also does work as seen in my code snippet.<br>
Yes, please post a minimal and executable code snippet, which would reproduce the issue.</p>

sparshgarg23: <p>I am actually using a custom dataset which I can’t share. But I can share train loader and other relevant details<br>
In all dataset has 8000 images where each image is of shape 512x512x3 and is associated with a lablel having 3 values [x,y,z]</p>