PyTorch on a Ubuntu 18.04.3 computer - CNN seems to crash the kernel

clived2: <p>I have been using PyTorch on my Ubuntu computer, latest anaconda3, Python 3.6.9, Pytorch 1.2.0.<br>
Linux kernel 5.24, 4 gigs of ram, for self training/personal interest. No GPU, running on CPU only</p>
<p>Most of the programs work, running linear regressions, logistic regressions, ANN models, CNN and RNN models. However when running a CNN model, they all seem to crash - The kernel appears to have died. It will restart automatically. Everything else seems to work fine. I have googled this and it seems a common issue, but no suggested solution which works for me.</p>
<p>Any comments, suggestions, folks ?</p>
<p>Thank you</p>

tom: <p><strong>Edit:</strong> So I had read too much LWN when looking at this. The kernel from the headline seems to be the Jupyter kernel rather than the Linux kernel that I had in mind.</p>
<p>It would seem impossible for PyTorch to crash a fully functional system (except by stalling it via swapping) in contrast to just PyTorch being terminated.<br>
As I would recommend to try other computationally intensive workloads to see if something is wrong with your computer.</p>
<p>Best regards</p>
<p>Thomas</p>

clived2: <p>Hi Tom, thanks for your response. It just crashes when I am running CNN training models with the message “the kernel appears to have died. It will restart automatically”. I use tensorflow on other other computationally intensive workloads, and they seem to work fine. I re-installed PyTorch to version 1.3.0 last night and the same thing thing keeps happening, except it seems that the script is still running on my hd. Anyway, I’ll keep experimenting , maybe its just because I am on an old comouter,  a Pentium based one, which seems to run everything else.</p>
<p>Anyway Thanks, Tom</p>
<p>Clive</p>

ptrblck: <p>Could you try to run your script in a terminal (export your notebook as a script, if necessary) and post the stack trace here, please?</p>

clived2: <p>Will do, Peter. Just trying to back up some of the stuff. Btw, I just ran the same script on my Win 10 laptop and it worked fine</p>

clived2: <h1><a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a></h1>
<pre><code class="lang-python">import torch
import torchvision
import torchvision.transforms as transforms
</code></pre>
<pre><code class="lang-python">transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
</code></pre>
<pre><code>Files already downloaded and verified
Files already downloaded and verified
</code></pre>
<pre><code class="lang-python">import matplotlib.pyplot as plt
import numpy as np

# functions to show an image


def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()


# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))
</code></pre>
<pre><code>&lt;Figure size 640x480 with 1 Axes&gt;


 frog plane  deer   car
</code></pre>
<h1>2. Define a Convolutional Neural Network</h1>
<pre><code class="lang-python">import torch.nn as nn
import torch.nn.functional as F
</code></pre>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


net = Net()
</code></pre>
<h1>3. Define a Loss function and optimizer</h1>
<pre><code class="lang-python">import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
</code></pre>
<h1>4. Train the network</h1>
<pre><code class="lang-python">for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')
</code></pre>
<h1>The kernel appears to have died. It will restart automatically.</h1>
<pre><code class="lang-python">
Here it is in Markdown format, bit of a mess, Peter. But everything worked until the training part where the kernel died. I might add that the same script ran perfectly on Google Colaboratory</code></pre>

ptrblck: <p>Thanks for the code!<br>
Could you nor rerun this code from a terminal via</p>
<pre><code class="lang-auto">python scripy.py
</code></pre>
<p>and check, if you get a valid stack stace? Jupyter kernels sometimes just get restarted without a proper error message (at least that’s my experience).</p>

clived2: <p>peter, i’ll give it a shot, not sure if I undestand how to run your suggested script with my code ?</p>

ptrblck: <p><code>script.py</code> is just a dummy name for your script.<br>
Just run the script in a terminal instead of a Jupyter notebook. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

clived2: <p>Gotcha. Peter, will do</p>

clived2: <p>Hi peter<br>
I took the original ipynb file and coded into a py file, ran it in spyder and had the same messesage re kernel restarting.</p>
<p>I then tried as suggested : python fifar10.py in a linux terminal and it just froze. I may just be because I am running it on an older (2009) desktop. it might be getting ready to die. I’m going to leave this for awhile.</p>
<p>Thanks for your help</p>
<p>Clive</p>

clived2: <p>Peter, I manged to do a backtrace using GDB ? makes no sense to me but here it is</p>
<p>gdb --args python CIFAR_10.py</p>
<p>(gdb) run<br>
Starting program: /home/clived/anaconda3/envs/tflow/bin/python CIFAR_10.py<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.<br>
Files already downloaded and verified<br>
[New Thread 0x7fffd6fa2780 (LWP 22610)]<br>
Files already downloaded and verified<br>
[New Thread 0x7fffcbf5d700 (LWP 22614)]<br>
[New Thread 0x7fffcb75c700 (LWP 22615)]<br>
[New Thread 0x7fffca315700 (LWP 22620)]<br>
[New Thread 0x7fffb9b79700 (LWP 22621)]<br>
^C<br>
Thread 1 “python” received signal SIGINT, Interrupt.</p>
<p>(gdb) backtrace<br>
<span class="hashtag">#0</span>  0x00007ffff78d9bf9 in _<em>GI___poll (fds=0x7fffc00072d0, nfds=1, timeout=-1) at …/sysdeps/unix/sysv/linux/poll.c:29<br>
<span class="hashtag">#1</span>  0x00007fffd4098cb3 in g_main_context_iterate.isra () from /home/clived/anaconda3/envs/tflow/lib/python3.6/site-packages/PyQt5/…/…/…/./libglib-2.0.so.0<br>
<span class="hashtag">#2</span>  0x00007fffd4098dce in g_main_context_iteration () from /home/clived/anaconda3/envs/tflow/lib/python3.6/site-packages/PyQt5/…/…/…/./libglib-2.0.so.0<br>
<span class="hashtag">#3</span>  0x00007fffd4b608df in QEventDispatcherGlib::processEvents(QFlags<a>QEventLoop::ProcessEventsFlag</a>) ()<br>
from /home/clived/anaconda3/envs/tflow/lib/python3.6/site-packages/PyQt5/…/…/…/libQt5Core.so.5<br>
<span class="hashtag">#4</span>  0x00007fffd4b30eb7 in QEventLoop::exec(QFlags<a>QEventLoop::ProcessEventsFlag</a>) ()<br>
from /home/clived/anaconda3/envs/tflow/lib/python3.6/site-packages/PyQt5/…/…/…/libQt5Core.so.5<br>
<span class="hashtag">#5</span>  0x00007fffd4b34aab in QCoreApplication::exec() () from /home/clived/anaconda3/envs/tflow/lib/python3.6/site-packages/PyQt5/…/…/…/libQt5Core.so.5<br>
<span class="hashtag">#6</span>  0x00007fffcc875970 in meth_QApplication_exec</em> () from /home/clived/anaconda3/envs/tflow/lib/python3.6/site-packages/PyQt5/QtWidgets.so<br>
<span class="hashtag">#7</span>  0x0000555555665b91 in _PyCFunction_FastCallDict () at /tmp/build/80754af9/python_1564510748219/work/Objects/methodobject.c:234<br>
<span class="hashtag">#8</span>  0x00005555556edabc in call_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4851<br>
<span class="hashtag">#9</span>  0x000055555571075a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:3335<br>
<span class="hashtag">#10</span> 0x00005555556e7c5b in _PyFunction_FastCall (globals=, nargs=0, args=, co=)<br>
at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4933<br>
<span class="hashtag">#11</span> fast_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4968<br>
<span class="hashtag">#12</span> 0x00005555556edb95 in call_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4872<br>
<span class="hashtag">#13</span> 0x000055555571075a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:3335<br>
<span class="hashtag">#14</span> 0x00005555556e89b9 in _PyEval_EvalCodeWithName (qualname=0x0, name=, closure=0x0, kwdefs=0x0, defcount=1, defs=0x7fffd5a47728, kwstep=2,<br>
kwcount=, kwargs=0x7ffff7f9c068, kwnames=0x7ffff7f9c060, argcount=, args=0x5555572f2410, locals=0x0, globals=,<br>
_co=0x7fffd5a489c0) at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4166<br>
<span class="hashtag">#15</span> PyEval_EvalCodeEx () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4187<br>
<span class="hashtag">#16</span> 0x00005555556e98e6 in function_call () at /tmp/build/80754af9/python_1564510748219/work/Objects/funcobject.c:604<br>
<span class="hashtag">#17</span> 0x0000555555665a5e in PyObject_Call () at /tmp/build/80754af9/python_1564510748219/work/Objects/abstract.c:2261<br>
—Type  to continue, or q  to quit—<br>
<span class="hashtag">#18</span> 0x0000555555711e37 in do_call_core (kwdict=0x7fffcbf91438, callargs=0x7fffc8f8b0f0, func=0x7fffd5a21a60)<br>
at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:5120<br>
<span class="hashtag">#19</span> _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:3404<br>
<span class="hashtag">#20</span> 0x00005555556e729e in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4166<br>
<span class="hashtag">#21</span> 0x00005555556e8598 in _PyFunction_FastCallDict () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:5084<br>
<span class="hashtag">#22</span> 0x000055555566601f in _PyObject_FastCallDict () at /tmp/build/80754af9/python_1564510748219/work/Objects/abstract.c:2310<br>
<span class="hashtag">#23</span> 0x000055555566aaa3 in _PyObject_Call_Prepend () at /tmp/build/80754af9/python_1564510748219/work/Objects/abstract.c:2373<br>
<span class="hashtag">#24</span> 0x0000555555665a5e in PyObject_Call () at /tmp/build/80754af9/python_1564510748219/work/Objects/abstract.c:2261<br>
<span class="hashtag">#25</span> 0x0000555555711e37 in do_call_core (kwdict=0x7fffcbf72480, callargs=0x7ffff7f9c048, func=0x7fffd5504e48)<br>
at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:5120<br>
<span class="hashtag">#26</span> _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:3404<br>
<span class="hashtag">#27</span> 0x00005555556e6e66 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4166<br>
<span class="hashtag">#28</span> 0x00005555556e7ed6 in fast_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4992<br>
<span class="hashtag">#29</span> 0x00005555556edb95 in call_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4872<br>
<span class="hashtag">#30</span> 0x000055555571075a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:3335<br>
<span class="hashtag">#31</span> 0x00005555556e7c5b in _PyFunction_FastCall (globals=, nargs=1, args=, co=)<br>
at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4933<br>
<span class="hashtag">#32</span> fast_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4968<br>
<span class="hashtag">#33</span> 0x00005555556edb95 in call_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4872<br>
<span class="hashtag">#34</span> 0x000055555571075a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:3335<br>
<span class="hashtag">#35</span> 0x00005555556e89b9 in _PyEval_EvalCodeWithName (qualname=0x0, name=, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2, kwcount=,<br>
kwargs=0x0, kwnames=0x0, argcount=0, args=0x0, locals=0x7ffff7f531b0, globals=0x7ffff7f531b0, _co=0x7ffff63f5a50)<br>
at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4166<br>
—Type  to continue, or q  to quit—<br>
<span class="hashtag">#36</span> PyEval_EvalCodeEx () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4187<br>
<span class="hashtag">#37</span> 0x00005555556e975c in PyEval_EvalCode (co=co@entry=0x7ffff63f5a50, globals=globals@entry=0x7ffff7f531b0, locals=locals@entry=0x7ffff7f531b0)<br>
at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:731<br>
<span class="hashtag">#38</span> 0x0000555555769744 in run_mod () at /tmp/build/80754af9/python_1564510748219/work/Python/pythonrun.c:1025<br>
<span class="hashtag">#39</span> 0x0000555555769b41 in PyRun_FileExFlags () at /tmp/build/80754af9/python_1564510748219/work/Python/pythonrun.c:978<br>
<span class="hashtag">#40</span> 0x0000555555769d43 in PyRun_SimpleFileExFlags () at /tmp/build/80754af9/python_1564510748219/work/Python/pythonrun.c:419<br>
<span class="hashtag">#41</span> 0x0000555555769e4d in PyRun_AnyFileExFlags () at /tmp/build/80754af9/python_1564510748219/work/Python/pythonrun.c:81<br>
<span class="hashtag">#42</span> 0x000055555576d833 in run_file (p_cf=0x7fffffffdc0c, filename=0x5555558a86c0 L"CIFAR_10.py", fp=0x55555593f120)<br>
at /tmp/build/80754af9/python_1564510748219/work/Modules/main.c:340<br>
<span class="hashtag">#43</span> Py_Main () at /tmp/build/80754af9/python_1564510748219/work/Modules/main.c:811<br>
<span class="hashtag">#44</span> 0x000055555563788e in main () at /tmp/build/80754af9/python_1564510748219/work/Programs/python.c:69<br>
<span class="hashtag">#45</span> 0x00007ffff77e6b97 in __libc_start_main (main=0x5555556377a0 , argc=2, argv=0x7fffffffde18, init=, fini=,<br>
rtld_fini=, stack_end=0x7fffffffde08) at …/csu/libc-start.c:310<br>
<span class="hashtag">#46</span> 0x0000555555717160 in _start () at …/sysdeps/x86_64/elf/start.S:103<br>
(gdb)</p>

ptrblck: <p>Thanks for the information. Since the script execution just hangs and you had to kill it via CTRL+C, you won’t get a proper backtrace.</p>
<p>Anyway, since the code runs fine on another machine, we should focus on your current setup.<br>
If you are using conda, could you create a clean new virtual environment via</p>
<pre><code class="lang-auto">conda create -n pytorch_stable Python=3.7 Anaconda
...
conda activate pytorch_stable
</code></pre>
<p>And install the latest stable release via the command from <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">the website</a>?</p>
<p>Also, some specs about the laptop could be helpful.<br>
E.g. which CPU is it using etc.<br>
I’ve seen some issues with old CPUs, which didn’t support some instructions (I think it was related to AVX instructions, but am not sure).</p>

clived2: <p>Hi Peter. Thats a good idea, my computer is a Dell Optiplex 70 wiith the following specs:<br>
OS: Ubuntu 18.04 (bionic) kernel = 5.2.14-050214-generic<br>
CPU: Pentium® Dual-Core  CPU E5200  @ 2.50GHz Frequency = 2227.079 MHz L2 cache = 2048 KB</p>
<p>The Anaconda3 install is using Python 3.7 but most of my dev work is on an env built for Python 3.6<br>
I’ll go ahead with your suggestion and see what happens, I’ll let you know how things go</p>
<p>Thanks<br>
Clive</p>

clived2: <p>Hi Peter, with your help, I figured this out. I remembered way back that I started having this problem with training a CNN when the PyTorch  1.2.0 was introduced. Anyway I came across a line of code which seemed to implement the appropriate version of Torch and Torchvision to install in my case on my older Dell Optiplex desktop, which was PyTorch 1.1.0. The code is as follows:<br>
(tflow) clived@clived-OptiPlex-760:~$ conda install pytorch-cpu torchvision-cpu -c pytorch.<br>
I ran this in my tflow env (Python 3.6) and in the Python environment (Python 3.7) as you suggested, and in both of these situations, my training a CNN model worked fine.</p>
<p>Glad that this has been resolved, as I had this in my head for like the last two weeks.</p>
<p>Thank you</p>
<p>Clive</p>