Training stopped at the end of the training phase of the first iteration

username: f10w
cooked: <p>Hello everybody,</p>
<p>Over the last days I have encountered a very strange problem: my training stopped at the end of the training phase of the first epoch (it did not perform the validation step), <strong>without any errors</strong>.</p>
<p>The main part of my training code is shown below. I have tested multiple times, the code works well on a subset of my dataset. However, whenever I run it on the <strong>full dataset</strong> (that has over 700000 images with train/val ratio is 9/1), it stopped at the first training phase and never entered the validation phase.</p>
<p>I remember having a similar issue some time ago: training did not enter the validation phase and produced a segmentation fault error. However, this time, there was no errors (the last lines of the terminal output are shown after the code).</p>
<p>Could you please help to to find out what happened?</p>
<p>Thank you so much in advance for any suggestions!</p>
<pre><code class="lang-auto">for phase in ['train', 'val']:
    print('Entering in phase:', phase)
    if phase == 'train':
        scheduler.step(best_acc)
        model.train()
    else:
        model.eval()

    running_loss = 0.0
    running_corrects = 0

    # Iterate over data.
    print('Iterating over data:')
    n_samples = 0
    for batch_idx, (inputs, labels) in enumerate(dataloaders[phase]):
        inputs = inputs.to(device)
        labels = labels.to(device)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
        # track history if only in train
        with torch.set_grad_enabled(phase == 'train'):
            outputs = model(inputs)
            _, preds = torch.max(outputs, 1)
            loss = criterion(outputs, labels)

            # backward + optimize only if in training phase
            if phase == 'train':
                loss.backward()
                optimizer.step()

        # statistics
        running_loss += loss.item() * inputs.size(0)
        running_corrects += torch.sum(preds == labels.data)
        n_samples += len(labels)

        if phase == 'train':
            print('{}/{} Avg Loss: {:.4f} Avg Acc: {:.4f}'.format(n_samples, dataset_sizes[phase],
                                    running_loss/n_samples, running_corrects.double()/n_samples))
    print('Done iterating over data')
    epoch_loss = running_loss / dataset_sizes[phase]
    epoch_acc = running_corrects.double() / dataset_sizes[phase]
    print('\t{} Loss: {:.4f}\t{} Acc: {:.4f}'.format(phase, epoch_loss, phase, epoch_acc))

# Here we are at the end of the 'val' phase
# check for improvement over the last epochs
if epoch_acc &gt; best_acc:
    print('Improved.')
    best_acc = epoch_acc
    try:
        state_dict = model.module.state_dict()
    except AttributeError:
        state_dict = model.state_dict()
    torch.save(state_dict, save_path)
</code></pre>
<blockquote>
<p>632736/633656 Avg Loss: 0.0436 Avg Acc: 0.9868<br>
632944/633656 Avg Loss: 0.0436 Avg Acc: 0.9868<br>
633152/633656 Avg Loss: 0.0435 Avg Acc: 0.9868<br>
633360/633656 Avg Loss: 0.0435 Avg Acc: 0.9868<br>
633568/633656 Avg Loss: 0.0435 Avg Acc: 0.9868<br>
633656/633656 Avg Loss: 0.0435 Avg Acc: 0.9868</p>
</blockquote>

username: f10w
cooked: <p>I tried with half the dataset and it works, but with the full dataset it did not work <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=6" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: ptrblck
cooked: <p>That’s really strange. How long did you wait?<br>
Could it be, that your memory is full and the swap is used for the validation step (which would take some time)?<br>
What stack trace do you get if you kill the process with CTRL+C?<br>
Are you using multiple workers? If so, does your code run using <code>num_workers=0</code>?</p>

username: f10w
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your reply. Yes I’m using multiple workers and I set the number of workers to be equal to the number of CPU processors (= 8). And I forgot to mention: training was run on multiple GPUs (4).<br>
It was stuck at the end of training for at least two hours before I shut it down (and this is reproducible, I ran it at least three times). Unfortunately I do not remember what stack trace I got, but I think it was related to dataloaders.<br>
Thanks again!</p>

username: ptrblck
cooked: <p>Thanks for the info! Is you code working with <code>num_workers=0</code>?<br>
If so, this might narrow down the possible source of your issue.</p>

username: f10w
cooked: <p>I have tested today with <code>num_workers=0</code> and it works! It finished the first epoch (training + validation) and now it’s running the second epoch.</p>
<p>D you have any suggestions to fix the problem with multiple workers? Because otherwise training is much slower (6 hours per epoch compared to 3 hours with <code>num_workers=8</code>) <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=6" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Thank you for your help!</p>

username: ptrblck
cooked: <p>Which PyTorch version are you using? We had some deadlock issues in older versions, which should be fixed by now in the latest release.</p>

username: f10w
cooked: <p>I’m using PyTorch 0.4.1. Since PyTorch 1.0 is a big update, I was afraid of upgrading because of potential incompatibility. However, after a quick search it seems that 0.4.1 code can run seamlessly in 1.0 so I think I will upgrade. Thank a lot!</p>

username: ptrblck
cooked: <p>The change should be pretty straightforward.<br>
However, if you encounter any issues, please let us know here and I’m sure we can fix it. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

username: f10w
cooked: <p>Thanks a lot! I will get back soon.</p>

username: chrisliu54
cooked: <p>Found similar issues.<br>
My program hung unexpectedly at some random point after several epochs of training. But I also set num_workers&gt;0 and I haven’t test using num_workers=0.</p>

username: f10w
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I confirm that upgrading to PyTorch 1.0 solves this issue. Thank you very much for your help!<br>
Surprisingly I didn’t have to change anything in my code (0.4.1), it just works!</p>
<p><a class="mention" href="/u/chrisliu54">@chrisliu54</a> Upgrade to PyTorch 1.0!</p>

username: hsah
cooked: <p>Hi, I am facing a similar issue where the training stopped at model.eval() after executing the training phase with no errors. I have not specified num_workers, so the default value is 0.</p>
<p>I am using torch==1.1.0 . Request any suggestions. Thanks.</p>

username: f10w
cooked: <p>Have you tried with version 1.0?</p>

username: hsah
cooked: <p>Sorry for the late response. There was no issue with torch as such but with pandas version. I confirm that this works fine with torch==1.1.0 too. Thanks.</p>

username: Tahlor
cooked: <p>I have experienced this issue intermittently using in version 1.1.0. Setting num_workers=0 seems to solved it for me, though it’s obviously not ideal.</p>