CNN Training Loop Not Working

username: ziad
cooked: <p>Hello!</p>
<p>I’ve built a CNN model which I’m now attempting to training using the conventional for-loop. I’ve been having an issue the past few days in which I’m not able to get the for-loop to work properly. Essentially, what happens is that the loop prints the epochs continuously until I have to interrupt.</p>
<p>This is the code I’m using:</p>
<pre><code class="lang-auto">epochs = 30
training_loss = []

for epoch in range(epochs):

    running_loss = 0

    #Train
    cnn.train()

    for images, labels in trainloader:
        if cuda:
            images, labels = images.cuda(), labels.cuda()

        optimiser.zero_grad()
        outputs = cnn(images)
        loss = criterion(outputs, labels)
        running_loss += loss.item() * images.size(0)
        loss.backward()
        optimiser.step()
    
        epochs_train_loss = running_loss / len(trainloader.dataset)
        print ("Epoch {}, Training Loss: {}".format(epoch, training_loss))
</code></pre>
<p>I’ve tried to place both</p>
<pre><code class="lang-auto">epochs_train_loss = running_loss / len(trainloader.dataset)
print ("Epoch {}, Training Loss: {}".format(epoch, training_loss)) 
</code></pre>
<p>out the for-loop but then nothing happens - the cell just runs indefinitely. I’m using a training set of 5000 images.</p>

username: ptrblck
cooked: <p>Could you check the length of the <code>trainloader</code> via <code>print(len(trainloader))</code>? This should return the number of batches and iterations in this loop.<br>
Assuming you are using the map-style <code>Dataset</code>, which implements the <code>__len__</code> method, each epoch should only yield the specified number of samples.</p>

username: ziad
cooked: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>thanks for the reply. <code>print(len(trainloader))</code> returned 163 - how does this impact my loop?</p>

username: ptrblck
cooked: <p>The <code>DataLoader</code> loop should exit after 163 iterations and would be executed <code>epochs</code> times.<br>
Add a print statement to the inner loop and check the current loop index.<br>
Since this loop apparently never exits, it would help to debug this issue.</p>

username: ziad
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>  - thanks for replying again.</p>
<p>I’ve tried update my code to the following:</p>
<pre><code class="lang-auto">num_epochs = 10
count = 0
loss_list = []
iteration_list = [] 
for epoch in range(num_epochs):
    for images, labels in enumerate(trainloader):
        
        # Clear gradients
        optimiser.zero_grad()
        
        # Forward propagation
        outputs = cnn(images)
        
        # Calculate loss
        loss = criterion(outputs, labels)
        
        # Calculating gradients
        loss.backward()
        
        # Update parameters
        optimiser.step()
        
        count += 1
            
            # store loss and iteration
        loss_list.append(loss.data)
        iteration_list.append(count)
            # accuracy_list.append(accuracy)
        if count % 12 == 0:
                # Print Loss
            print('Iteration: {}  Loss: {} %'.format(count, loss.data))
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">Iteration: 12  Loss: 0.3102647066116333 %
Iteration: 24  Loss: 0.5342930555343628 %
Iteration: 36  Loss: 0.13416573405265808 %
Iteration: 48  Loss: 0.05583428218960762 %
Iteration: 60  Loss: 0.3329312801361084 %
Iteration: 72  Loss: 0.1474495381116867 %
</code></pre>
<p>I made some changes so that my <code>len(trainloader)</code> is now 82. However, the cell keeps running and running after Iteration: 72 is achieved. I’m not too sure why. Is there some stoppage logic I could use perhaps?</p>
<p>Thanks again</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="ziad" data-post="5" data-topic="112166">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ziad/48/44608_2.png" class="avatar"> ziad:</div>
<blockquote>
<p>However, the cell keeps running and running after Iteration: 72 is achieved. I’m not too sure why. Is there some stoppage logic I could use perhaps?</p>
</blockquote>
</aside>
<p>It seems the cell is not running anymore, but hangs instead.<br>
Are you seeing the same issue in a terminal, if you execute the script directly without a notebook?</p>

username: ziad
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>You’re right in that it just hangs after it finishes executing.</p>
<p>I’m not too sure how to go about running this in terminal as I’ve only worked in Notebooks.  Interestingly enough, the same thing happens if I used Skorch’s <code>NeuralNetClassifier</code>:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a967938ed5e16d9a3f22d9812f174b48c663e038.png" data-download-href="https://discuss.pytorch.org/uploads/default/a967938ed5e16d9a3f22d9812f174b48c663e038" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a967938ed5e16d9a3f22d9812f174b48c663e038_2_277x375.png" alt="image" data-base62-sha1="oaCBrs17KYLOjbV5QYKw57ZZqY0" width="277" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a967938ed5e16d9a3f22d9812f174b48c663e038_2_277x375.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a967938ed5e16d9a3f22d9812f174b48c663e038_2_415x562.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a967938ed5e16d9a3f22d9812f174b48c663e038_2_554x750.png 2x" data-dominant-color="E9E7EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">592×799 77.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Lastly, could this be related in any way to how I defined the CNN or even to how I’ve pre-processed the images?</p>

username: ptrblck
cooked: <p>It sounds rather like a multiprocessing issue (or code parts are executed after the training, which are not shown in the posted code).<br>
You should be able to <code>export</code> the notebook as a Python script file and run it in a terminal.</p>