How to solve pytorch freezing issue?

username: mobassir94
cooked: <p>i am trying to train tf_efficientnet_b0_ns model from popular timm repository for kaggle g2net gravitation wave detection challenge : <a href="https://www.kaggle.com/c/g2net-gravitational-wave-detection" class="inline-onebox" rel="noopener nofollow ugc">G2Net Gravitational Wave Detection | Kaggle</a></p>
<p>during training i checked nvidia-smi and i was using around 19gb vram out of 24(i am using rtx 3090)<br>
my problem is during 7th epoch computer freezes and i am forced to restart the pc.<br>
here is the code and log attached : <a href="https://pastebin.com/XEusHppZ" class="inline-onebox" rel="noopener nofollow ugc">torch_freezing_pc - Pastebin.com</a><br>
how to troubleshoot this annoying freezing issue?<br>
thank you</p>

username: ptrblck
cooked: <p>Once the code freezes you could attach <code>gdb</code> to the process and check the backtrace to see where the code is frozen.</p>

username: mobassir94
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> how to use gdb and backtrace?i never used it,sorry.i checked kern log and found nothing problematic,is there any memory leak in my code?<br>
or is it related to pytorch version?<br>
i am using torch ‘1.10.0.dev20210623+cu111’<br>
i am using these packages :</p>
<h1>
<a name="name-version-build-channel-1" class="anchor" href="#name-version-build-channel-1"></a>Name                    Version                   Build  Channel</h1>
<p>_libgcc_mutex             0.1                        main<br>
_openmp_mutex             4.5                       1_gnu<br>
addict                    2.4.0                    pypi_0    pypi<br>
alabaster                 0.7.12                     py_0    conda-forge<br>
albumentations            1.0.3                    pypi_0    pypi<br>
alsa-lib                  1.2.3                h516909a_0    conda-forge<br>
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge<br>
argh                      0.26.2          pyh9f0ad1d_1002    conda-forge<br>
astroid                   2.6.0            py38h578d9bd_0    conda-forge<br>
async_generator           1.10                       py_0    conda-forge<br>
asynctest                 0.13.0                   pypi_0    pypi<br>
atomicwrites              1.4.0              pyh9f0ad1d_0    conda-forge<br>
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge<br>
autopep8                  1.5.6              pyhd8ed1ab_0    conda-forge<br>
babel                     2.9.1              pyh44b312d_0    conda-forge<br>
backcall                  0.2.0              pyhd3eb1b0_0<br>
black                     21.5b2             pyhd8ed1ab_0    conda-forge<br>
bleach                    3.3.0              pyh44b312d_0    conda-forge<br>
brotlipy                  0.7.0           py38h497a2fe_1001    conda-forge<br>
ca-certificates           2021.5.30            ha878542_0    conda-forge<br>
certifi                   2021.5.30        py38h578d9bd_0    conda-forge<br>
cffi                      1.14.5           py38ha65f79e_0    conda-forge<br>
cfgv                      3.3.0                    pypi_0    pypi<br>
chardet                   4.0.0            py38h578d9bd_1    conda-forge<br>
click                     8.0.1            py38h578d9bd_0    conda-forge<br>
cloudpickle               1.6.0                      py_0    conda-forge<br>
codecov                   2.1.11                   pypi_0    pypi<br>
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge<br>
configparser              5.0.2                    pypi_0    pypi<br>
coverage                  5.5                      pypi_0    pypi<br>
cryptography              3.4.7            py38ha5dfef3_0    conda-forge<br>
cudatoolkit               11.1.74              h6bb024c_0    nvidia<br>
cycler                    0.10.0                   pypi_0    pypi<br>
cython                    0.29.23                  pypi_0    pypi<br>
dataclasses               0.8                pyhc8e2a94_1    conda-forge<br>
dbus                      1.13.6               h48d8840_2    conda-forge<br>
decorator                 4.4.2                    pypi_0    pypi<br>
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge<br>
diff-match-patch          20200713           pyh9f0ad1d_0    conda-forge<br>
dill                      0.3.4                    pypi_0    pypi<br>
distlib                   0.3.2                    pypi_0    pypi<br>
docker-pycreds            0.4.0                    pypi_0    pypi<br>
docutils                  0.17.1           py38h578d9bd_0    conda-forge<br>
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge<br>
expat                     2.4.1                h9c3ff4c_0    conda-forge<br>
filelock                  3.0.12                   pypi_0    pypi<br>
flake8                    3.9.2                    pypi_0    pypi<br>
fontconfig                2.13.1            hba837de_1005    conda-forge<br>
freetype                  2.10.4               h0708190_1    conda-forge<br>
future                    0.18.2           py38h578d9bd_3    conda-forge<br>
gast                      0.3.3                    pypi_0    pypi<br>
gettext                   0.19.8.1          h0b5b191_1005    conda-forge<br>
gitdb                     4.0.7                    pypi_0    pypi<br>
gitpython                 3.1.18                   pypi_0    pypi<br>
glib                      2.68.3               h9c3ff4c_0    conda-forge<br>
glib-tools                2.68.3               h9c3ff4c_0    conda-forge<br>
googleapis-common-protos  1.53.0                   pypi_0    pypi<br>
grad-cam                  1.3.1                    pypi_0    pypi<br>
grpcio                    1.32.0                   pypi_0    pypi<br>
gst-plugins-base          1.18.4               hf529b03_2    conda-forge<br>
gstreamer                 1.18.4               h76c114f_2    conda-forge<br>
h5py                      2.10.0                   pypi_0    pypi<br>
helpdev                   0.7.1              pyhd8ed1ab_0    conda-forge<br>
icu                       68.1                 h58526e2_0    conda-forge<br>
identify                  2.2.10                   pypi_0    pypi<br>
idna                      2.10               pyh9f0ad1d_0    conda-forge<br>
imageio                   2.9.0                    pypi_0    pypi<br>
imagesize                 1.2.0                      py_0    conda-forge<br>
imgaug                    0.4.0                    pypi_0    pypi<br>
importlib-metadata        4.5.0            py38h578d9bd_0    conda-forge<br>
importlib-resources       5.2.2                    pypi_0    pypi<br>
importlib_metadata        4.5.0                hd8ed1ab_0    conda-forge<br>
iniconfig                 1.1.1                    pypi_0    pypi<br>
intervaltree              3.0.2                      py_0    conda-forge<br>
ipykernel                 5.5.5            py38hd0cf306_0    conda-forge<br>
ipython                   7.18.1           py38h5ca1d4c_0    anaconda<br>
ipython_genutils          0.2.0              pyhd3eb1b0_1<br>
ipywidgets                7.6.3                    pypi_0    pypi<br>
isort                     5.8.0                    pypi_0    pypi<br>
jedi                      0.17.2           py38h578d9bd_1    conda-forge<br>
jeepney                   0.6.0              pyhd8ed1ab_0    conda-forge<br>
jinja2                    3.0.1              pyhd8ed1ab_0    conda-forge<br>
jpeg                      9d                   h36c2ea0_0    conda-forge<br>
jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge<br>
jupyter_client            6.1.12             pyhd8ed1ab_0    conda-forge<br>
jupyter_core              4.7.1            py38h578d9bd_0    conda-forge<br>
jupyterlab-widgets        1.0.0                    pypi_0    pypi<br>
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge<br>
kaggle                    1.5.12                   pypi_0    pypi<br>
kaggledatasets            0.0.1                    pypi_0    pypi<br>
keyring                   23.0.1           py38h578d9bd_0    conda-forge<br>
kiwisolver                1.3.1                    pypi_0    pypi<br>
krb5                      1.19.1               hcc1bbae_0    conda-forge<br>
kwarray                   0.5.19                   pypi_0    pypi<br>
lanms-proper              1.0.1                    pypi_0    pypi<br>
lazy-object-proxy         1.6.0            py38h497a2fe_0    conda-forge<br>
ld_impl_linux-64          2.35.1               h7274673_9<br>
libclang                  11.1.0          default_ha53f305_1    conda-forge<br>
libedit                   3.1.20191231         he28a2e2_2    conda-forge<br>
libevent                  2.1.10               hcdb4288_3    conda-forge<br>
libffi                    3.3                  he6710b0_2<br>
libgcc-ng                 9.3.0               h5101ec6_17<br>
libglib                   2.68.3               h3e27bee_0    conda-forge<br>
libgomp                   9.3.0               h5101ec6_17<br>
libiconv                  1.16                 h516909a_0    conda-forge<br>
libllvm11                 11.1.0               hf817b99_2    conda-forge<br>
libogg                    1.3.4                h7f98852_1    conda-forge<br>
libopus                   1.3.1                h7f98852_1    conda-forge<br>
libpng                    1.6.37               h21135ba_2    conda-forge<br>
libpq                     13.3                 hd57d9b9_0    conda-forge<br>
libsodium                 1.0.18               h36c2ea0_1    conda-forge<br>
libspatialindex           1.9.3                h9c3ff4c_3    conda-forge<br>
libstdcxx-ng              9.3.0               hd4cf53a_17<br>
libuuid                   2.32.1            h7f98852_1000    conda-forge<br>
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge<br>
libxcb                    1.13              h7f98852_1003    conda-forge<br>
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge<br>
libxml2                   2.9.12               h72842e0_0    conda-forge<br>
llvmlite                  0.36.0                   pypi_0    pypi<br>
lmdb                      1.2.1                    pypi_0    pypi<br>
lz4-c                     1.9.3                h9c3ff4c_0    conda-forge<br>
markupsafe                2.0.1            py38h497a2fe_0    conda-forge<br>
matplotlib                3.4.2                    pypi_0    pypi<br>
mccabe                    0.6.1                    pypi_0    pypi<br>
mistune                   0.8.4           py38h497a2fe_1003    conda-forge<br>
mmcv-full                 1.3.7                     dev_0    <br>
mmdet                     2.11.0                   pypi_0    pypi<br>
mmocr                     0.2.0                     dev_0    <br>
mmpycocotools             12.0.3                   pypi_0    pypi<br>
mypy_extensions           0.4.3            py38h578d9bd_3    conda-forge<br>
mysql-common              8.0.25               ha770c72_2    conda-forge<br>
mysql-libs                8.0.25               hfa10184_2    conda-forge<br>
nbclient                  0.5.3              pyhd8ed1ab_0    conda-forge<br>
nbconvert                 6.0.7                    pypi_0    pypi<br>
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge<br>
ncurses                   6.2                  he6710b0_1<br>
nest-asyncio              1.5.1              pyhd8ed1ab_0    conda-forge<br>
networkx                  2.5.1                    pypi_0    pypi<br>
nnaudio                   0.2.5                    pypi_0    pypi<br>
nodeenv                   1.6.0                    pypi_0    pypi<br>
nspr                      4.30                 h9c3ff4c_0    conda-forge<br>
nss                       3.64                 hb5efdd6_0    conda-forge<br>
numba                     0.53.1                   pypi_0    pypi<br>
numpy                     1.19.5                   pypi_0    pypi<br>
numpydoc                  1.1.0                      py_1    conda-forge<br>
oauthlib                  3.1.1                    pypi_0    pypi<br>
opencv-python             4.5.2.54                 pypi_0    pypi<br>
opencv-python-headless    4.5.3.56                 pypi_0    pypi<br>
openssl                   1.1.1k               h7f98852_0    conda-forge<br>
ordered-set               4.0.2                    pypi_0    pypi<br>
packaging                 20.9               pyh44b312d_0    conda-forge<br>
pandas                    1.3.0                    pypi_0    pypi<br>
pandoc                    2.14.0.3             h7f98852_0    conda-forge<br>
pandocfilters             1.4.3                    pypi_0    pypi<br>
parso                     0.7.0              pyh9f0ad1d_0    conda-forge<br>
pathspec                  0.8.1              pyhd3deb0d_0    conda-forge<br>
pathtools                 0.1.2                    pypi_0    pypi<br>
pcre                      8.45                 h9c3ff4c_0    conda-forge<br>
pexpect                   4.8.0              pyhd3eb1b0_3<br>
pickleshare               0.7.5           pyhd3eb1b0_1003<br>
pillow                    8.2.0                    pypi_0    pypi<br>
pip                       21.1.2           py38h06a4308_0<br>
pluggy                    0.13.1           py38h578d9bd_4    conda-forge<br>
polygon3                  3.0.9.1                  pypi_0    pypi<br>
pre-commit                2.13.0                   pypi_0    pypi<br>
promise                   2.3                      pypi_0    pypi<br>
prompt-toolkit            3.0.18                   pypi_0    pypi<br>
protobuf                  3.17.3                   pypi_0    pypi<br>
psutil                    5.8.0            py38h497a2fe_1    conda-forge<br>
pthread-stubs             0.4               h36c2ea0_1001    conda-forge<br>
ptyprocess                0.7.0              pyhd3eb1b0_2<br>
py                        1.10.0                   pypi_0    pypi<br>
pyclipper                 1.2.1                    pypi_0    pypi<br>
pycocotools               2.0.2                    pypi_0    pypi<br>
pycodestyle               2.7.0                    pypi_0    pypi<br>
pycparser                 2.20               pyh9f0ad1d_2    conda-forge<br>
pydocstyle                6.1.1              pyhd8ed1ab_0    conda-forge<br>
pyflakes                  2.3.1                    pypi_0    pypi<br>
pygments                  2.9.0              pyhd3eb1b0_0<br>
pylint                    2.8.2              pyhd8ed1ab_0    conda-forge<br>
pyls-black                0.4.6              pyh9f0ad1d_0    conda-forge<br>
pyls-spyder               0.3.2              pyhd8ed1ab_0    conda-forge<br>
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge<br>
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge<br>
pyqt                      5.12.3           py38h578d9bd_7    conda-forge<br>
pyqt-impl                 5.12.3           py38h7400c14_7    conda-forge<br>
pyqt5-sip                 4.19.18          py38h709712a_7    conda-forge<br>
pyqtchart                 5.12             py38h7400c14_7    conda-forge<br>
pyqtwebengine             5.12.1           py38h7400c14_7    conda-forge<br>
pyrsistent                0.17.3           py38h497a2fe_2    conda-forge<br>
pysocks                   1.7.1            py38h578d9bd_3    conda-forge<br>
pytest                    6.2.4                    pypi_0    pypi<br>
pytest-cov                2.12.1                   pypi_0    pypi<br>
pytest-runner             5.3.1                    pypi_0    pypi<br>
python                    3.8.5                h7579374_1<br>
python-dateutil           2.8.1                      py_0    conda-forge<br>
python-jsonrpc-server     0.4.0              pyh9f0ad1d_0    conda-forge<br>
python-language-server    0.36.2             pyhd8ed1ab_0    conda-forge<br>
python-slugify            5.0.2                    pypi_0    pypi<br>
python_abi                3.8                      2_cp38    conda-forge<br>
pytz                      2021.1             pyhd8ed1ab_0    conda-forge<br>
pywavelets                1.1.1                    pypi_0    pypi<br>
pyxdg                     0.27               pyhd8ed1ab_0    conda-forge<br>
pyyaml                    5.4.1                    pypi_0    pypi<br>
pyzmq                     22.1.0           py38h2035c66_0    conda-forge<br>
qdarkstyle                2.8.1              pyhd8ed1ab_2    conda-forge<br>
qt                        5.12.9               hda022c4_4    conda-forge<br>
qtawesome                 1.0.3              pyhd8ed1ab_0    conda-forge<br>
qtconsole                 5.1.0              pyhd8ed1ab_0    conda-forge<br>
qtpy                      1.9.0                      py_0    conda-forge<br>
qudida                    0.0.4                    pypi_0    pypi<br>
rapidfuzz                 1.4.1                    pypi_0    pypi<br>
readline                  8.1                  h27cfd23_0<br>
regex                     2021.4.4         py38h497a2fe_0    conda-forge<br>
requests                  2.25.1             pyhd3deb0d_0    conda-forge<br>
rope                      0.19.0             pyhd8ed1ab_0    conda-forge<br>
rtree                     0.9.7            py38h02d302b_1    conda-forge<br>
scikit-image              0.18.1                   pypi_0    pypi<br>
scipy                     1.6.3                    pypi_0    pypi<br>
seaborn                   0.11.2                   pypi_0    pypi<br>
secretstorage             3.3.1            py38h578d9bd_0    conda-forge<br>
send2trash                1.5.0                    pypi_0    pypi<br>
sentry-sdk                1.3.1                    pypi_0    pypi<br>
setuptools                52.0.0           py38h06a4308_0<br>
shapely                   1.7.1                    pypi_0    pypi<br>
shortuuid                 1.0.1                    pypi_0    pypi<br>
six                       1.16.0             pyh6c4a22f_0    conda-forge<br>
smmap                     4.0.0                    pypi_0    pypi<br>
snowballstemmer           2.1.0              pyhd8ed1ab_0    conda-forge<br>
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge<br>
sphinx                    4.0.2              pyh6c4a22f_1    conda-forge<br>
sphinxcontrib-applehelp   1.0.2                      py_0    conda-forge<br>
sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge<br>
sphinxcontrib-htmlhelp    2.0.0              pyhd8ed1ab_0    conda-forge<br>
sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge<br>
sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge<br>
sphinxcontrib-serializinghtml 1.1.5              pyhd8ed1ab_0    conda-forge<br>
spyder                    4.2.5            py38h578d9bd_0    conda-forge<br>
spyder-kernels            1.10.2           py38h578d9bd_0    conda-forge<br>
sqlite                    3.35.4               hdfb4753_0<br>
subprocess32              3.5.4                    pypi_0    pypi<br>
tensorboard               2.6.0                    pypi_0    pypi<br>
tensorflow-datasets       4.4.0                    pypi_0    pypi<br>
tensorflow-gpu            2.4.0                    pypi_0    pypi<br>
tensorflow-metadata       1.2.0                    pypi_0    pypi<br>
terminado                 0.10.0                   pypi_0    pypi<br>
terminaltables            3.1.0                    pypi_0    pypi<br>
testpath                  0.5.0              pyhd8ed1ab_0    conda-forge<br>
text-unidecode            1.3                      pypi_0    pypi<br>
textdistance              4.2.1              pyhd8ed1ab_0    conda-forge<br>
three-merge               0.1.1              pyh9f0ad1d_0    conda-forge<br>
tifffile                  2021.6.6                 pypi_0    pypi<br>
timm                      0.4.13                   pypi_0    pypi<br>
tk                        8.6.10               hbc83047_0<br>
toml                      0.10.2             pyhd8ed1ab_0    conda-forge<br>
torch                     1.10.0.dev20210623+cu111          pypi_0    pypi<br>
torchaudio                0.8.1                    pypi_0    pypi<br>
torchvision               0.11.0.dev20210623+cu111          pypi_0    pypi<br>
tornado                   6.1              py38h497a2fe_1    conda-forge<br>
traitlets                 5.0.5              pyhd3eb1b0_0<br>
ttach                     0.0.3                    pypi_0    pypi<br>
typed-ast                 1.4.3            py38h497a2fe_0    conda-forge<br>
typing_extensions         3.10.0.0           pyha770c72_0    conda-forge<br>
ubelt                     0.9.5                    pypi_0    pypi<br>
ujson                     4.0.2            py38h709712a_0    conda-forge<br>
urllib3                   1.26.5             pyhd8ed1ab_0    conda-forge<br>
virtualenv                20.4.7                   pypi_0    pypi<br>
wandb                     0.12.1                   pypi_0    pypi<br>
watchdog                  1.0.2            py38h578d9bd_1    conda-forge<br>
wcwidth                   0.2.5                      py_0<br>
webencodings              0.5.1                    pypi_0    pypi<br>
wheel                     0.36.2             pyhd3eb1b0_0<br>
widgetsnbextension        3.5.1                    pypi_0    pypi<br>
wrapt                     1.12.1           py38h497a2fe_3    conda-forge<br>
wurlitzer                 2.1.0            py38h578d9bd_0    conda-forge<br>
xdoctest                  0.15.4                   pypi_0    pypi<br>
xorg-libxau               1.0.9                h7f98852_0    conda-forge<br>
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge<br>
xz                        5.2.5                h7b6447c_0<br>
yaml                      0.2.5                h516909a_0    conda-forge<br>
yapf                      0.31.0             pyhd8ed1ab_0    conda-forge<br>
zeromq                    4.3.4                h9c3ff4c_0    conda-forge<br>
zipp                      3.4.1              pyhd8ed1ab_0    conda-forge<br>
zlib                      1.2.11               h7b6447c_3<br>
zstd                      1.5.0                ha95c52a_0    conda-forge</p>

username: ptrblck
cooked: <p><code>gdb attach pid</code> should work as described <a href="https://sourceware.org/gdb/onlinedocs/gdb/Attach.html">here</a>.</p>

username: mobassir94
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> i track system metric using wandb during training and before freezing here are the results :</p>
<ol>
<li><a href="https://i.ibb.co/Jzs0J2C/1.png" rel="noopener nofollow ugc">https://i.ibb.co/Jzs0J2C/1.png</a></li>
<li><a href="https://i.ibb.co/cttKxNk/2.png" rel="noopener nofollow ugc">https://i.ibb.co/cttKxNk/2.png</a></li>
<li><a href="https://i.ibb.co/q5xGTsN/3.png" rel="noopener nofollow ugc">https://i.ibb.co/q5xGTsN/3.png</a></li>
<li><a href="https://i.ibb.co/hWyW5bx/4.png" rel="noopener nofollow ugc">https://i.ibb.co/hWyW5bx/4.png</a></li>
</ol>

username: ptrblck
cooked: <p>The graphs show a constant usage and I don’t see any hang or how this could be used for debugging the issue. Were you able to check the backtrace via <code>gdb</code>?</p>

username: mobassir94
cooked: <p>sorry  i am not understanding how to use gdb tracking.is there any tutorial to follow? for example i want to train this model : <a href="https://www.kaggle.com/yasufuminakama/g2net-efficientnet-b7-baseline-training" class="inline-onebox" rel="noopener nofollow ugc">G2Net / efficientnet_b7 / baseline [training] | Kaggle</a><br>
what are the step by step procedure for gdb tracking? need a tutorial</p>

username: ptrblck
cooked: <p><a href="https://www.cs.cmu.edu/~gilpin/tutorial/">This tutorial</a> might be a good starter.</p>

username: mobassir94
cooked: <p>so i have my training code saved in trainer.py file and for gdb backtracking i am planning to execute this command from terminal :<br>
gdb -batch --ex “run” -ex “bt” --args python trainer.py 2&gt;&amp;1 | grep -v ^“No stack.” &gt; log.txt</p>
<p>it should train my model using trainer.py file and save log in log.txt file,am i on track? or i am making mistake? is the command right for saving backtracking log?</p>