RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THC/THCBlas.cu:259

Mona_Jalal: <p>Do you know how I can fix this error? I am following all the instructions as shown in this git repo:<br>
</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/yinyunie/Total3DUnderstanding" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:400/400;"><img src="https://avatars0.githubusercontent.com/u/25686434?s=400&amp;amp;v=4" class="thumbnail" width="400" height="400"></div>

<h3><a href="https://github.com/yinyunie/Total3DUnderstanding" target="_blank" rel="noopener nofollow ugc">yinyunie/Total3DUnderstanding</a></h3>


  <p><span class="label1">Implementation of CVPR'20 Oral: Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>full log accessible here <a href="https://pastebin.com/raw/TCGxksDt" rel="noopener nofollow ugc">https://pastebin.com/raw/TCGxksDt</a><br>
also <a href="https://github.com/pytorch/pytorch/issues/49116" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/49116</a></p>
<pre><code class="lang-auto">(base) mona@mona:~/research$ git clone https://github.com/yinyunie/Total3DUnderstanding.git
Cloning into 'Total3DUnderstanding'...
remote: Enumerating objects: 206, done.
remote: Counting objects: 100% (206/206), done.
remote: Compressing objects: 100% (181/181), done.
remote: Total 206 (delta 31), reused 192 (delta 20), pack-reused 0
Receiving objects: 100% (206/206), 4.23 MiB | 19.59 MiB/s, done.
Resolving deltas: 100% (31/31), done.
(base) mona@mona:~/research$ cd Total3DUnderstanding/
(basRuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THC/THCBlas.cu:259e) mona@mona:~/research/Total3DUnderstanding$ conda env create -f environment.yml
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages
Installing pip dependencies: / Ran pip subprocess with arguments:
['/home/mona/anaconda3/envs/Total3D/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/mona/research/Total3DUnderstanding/condaenv.usbz06he.requirements.txt']
Pip subprocess output:

done
#
# To activate this environment, use
#
#     $ conda activate Total3D
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) mona@mona:~/research/Total3DUnderstanding$ conda activate Total3D
(Total3D) mona@mona:~/research/Total3DUnderstanding$ python main.py configs/total3d.yaml --mode demo --demo_path demo/inputs/1
Loading configurations.
{'method': 'TOTAL3D', 'resume': False, 'finetune': True, 'weight': ['out/pretrained_models/pretrained_model.pth'], 'seed': 123, 'device': {'use_gpu': True, 'gpu_ids': '0', 'num_workers': 2}, 'data': {'dataset': 'sunrgbd', 'split': 'data/sunrgbd/splits', 'tmn_subnetworks': 2, 'face_samples': 1, 'with_edge_classifier': True}, 'model': {'layout_estimation': {'method': 'PoseNet', 'loss': 'PoseLoss'}, 'object_detection': {'method': 'Bdb3DNet', 'loss': 'DetLoss'}, 'mesh_reconstruction': {'method': 'DensTMNet', 'loss': 'ReconLoss'}}, 'optimizer': {'method': 'Adam', 'lr': '1e-4', 'betas': [0.9, 0.999], 'eps': '1e-08', 'weight_decay': '1e-04'}, 'scheduler': {'patience': 5, 'factor': 0.5, 'threshold': 0.01}, 'train': {'epochs': 400, 'phase': 'joint', 'freeze': ['mesh_reconstruction'], 'batch_size': 2}, 'test': {'phase': 'joint', 'batch_size': 2}, 'demo': {'phase': 'joint'}, 'log': {'vis_path': 'out/total3d/2020-12-09T15:00:36.822598/visualization', 'save_results': True, 'vis_step': 100, 'print_step': 50, 'path': 'out/total3d/2020-12-09T15:00:36.822598'}, 'config': 'configs/total3d.yaml', 'mode': 'demo', 'demo_path': 'demo/inputs/1'}
Data save path: out/total3d/2020-12-09T15:00:36.822598
Loading device settings.
GPU mode is on.
GPU Ids: 0 used.
Loading model.
Downloading: "https://download.pytorch.org/models/resnet34-333f7ec4.pth" to /home/mona/.cache/torch/checkpoints/resnet34-333f7ec4.pth
100.0%
Downloading: "https://download.pytorch.org/models/resnet18-5c106cde.pth" to /home/mona/.cache/torch/checkpoints/resnet18-5c106cde.pth
100.0%
Begin to finetune from the existing weight.
Loading checkpoint from out/pretrained_models/pretrained_model.pth.
set() subnet missed.
Weights for finetuning loaded.
----------------------------------------------------------------------------------------------------
Loading data.
Traceback (most recent call last):
  File "main.py", line 38, in &lt;module&gt;
    demo.run(cfg)
  File "/home/mona/research/Total3DUnderstanding/demo.py", line 147, in run
    est_data = net(data)
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mona/research/Total3DUnderstanding/models/total3d/modules/network.py", line 67, in forward
    lo_centroid_result, lo_coeffs_result = self.layout_estimation(data['image'])
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mona/research/Total3DUnderstanding/models/total3d/modules/layout_estimation.py", line 63, in forward
    cam = self.fc_1(x)
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 92, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/mona/anaconda3/envs/Total3D/lib/python3.6/site-packages/torch/nn/functional.py", line 1406, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THC/THCBlas.cu:259
(Total3D) mona@mona:~/research/Total3DUnderstanding$ python
Python 3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.__version__
'1.1.0'
&gt;&gt;&gt; quit()
(Total3D) mona@mona:~/research/Total3DUnderstanding$ bat environment.yml 
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: environment.yml
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ name: Total3D
   2   │ channels:
   3   │   - pytorch
   4   │   - defaults
   5   │ dependencies:
   6   │   - _libgcc_mutex=0.1
   7   │   - blas=1.0
   8   │   - bzip2=1.0.8
   9   │   - ca-certificates=2020.1.1
  10   │   - certifi=2020.6.20
  11   │   - cffi=1.14.0
  12   │   - cudatoolkit=9.0
  13   │   - curl=7.69.1
  14   │   - expat=2.2.6
  15   │   - freetype=2.9.1
  16   │   - future=0.18.2
  17   │   - geos=3.8.0
  18   │   - hdf4=4.2.13
  19   │   - hdf5=1.10.4
  20   │   - icu=58.2
  21   │   - intel-openmp=2020.1
  22   │   - jpeg=9b
  23   │   - jsoncpp=1.8.4
  24   │   - krb5=1.17.1
  25   │   - ld_impl_linux-64=2.33.1
  26   │   - libcurl=7.69.1
  27   │   - libedit=3.1.20181209
  28   │   - libffi=3.3
  29   │   - libgcc-ng=9.1.0
  30   │   - libgfortran-ng=7.3.0
  31   │   - libnetcdf=4.6.1
  32   │   - libogg=1.3.2
  33   │   - libpng=1.6.37
  34   │   - libssh2=1.9.0
  35   │   - libstdcxx-ng=9.1.0
  36   │   - libtheora=1.1.1
  37   │   - libtiff=4.1.0
  38   │   - libvorbis=1.3.6
  39   │   - libxml2=2.9.10
  40   │   - lz4-c=1.8.1.2
  41   │   - mkl=2020.1
  42   │   - mkl-service=2.3.0
  43   │   - mkl_fft=1.0.15
  44   │   - mkl_random=1.1.1
  45   │   - ncurses=6.2
  46   │   - ninja=1.9.0
  47   │   - numpy=1.18.1
  48   │   - numpy-base=1.18.1
  49   │   - olefile=0.46
  50   │   - openssl=1.1.1g
  51   │   - pandas=1.0.5
  52   │   - pillow=7.1.2
  53   │   - pip=20.0.2
  54   │   - pycparser=2.20
  55   │   - python=3.6.10
  56   │   - python-dateutil=2.8.1
  57   │   - pytorch=1.1.0
  58   │   - pytz=2020.1
  59   │   - pyyaml=5.3.1
  60   │   - readline=8.0
  61   │   - scipy=1.4.1
  62   │   - setuptools=47.1.1
  63   │   - shapely=1.7.0
  64   │   - six=1.15.0
  65   │   - sqlite=3.31.1
  66   │   - tbb=2020.0
  67   │   - tk=8.6.8
  68   │   - torchvision=0.3.0
  69   │   - vtk=8.2.0
  70   │   - wheel=0.34.2
  71   │   - xz=5.2.5
  72   │   - yaml=0.1.7
  73   │   - zlib=1.2.11
  74   │   - zstd=1.3.7
  75   │   - pip:
  76   │     - cycler==0.10.0
  77   │     - jellyfish==0.8.2
  78   │     - kiwisolver==1.2.0
  79   │     - matplotlib==3.2.2
  80   │     - opencv-python==4.2.0.34
  81   │     - pyparsing==2.4.7
  82   │     - seaborn==0.10.1
  83   │
</code></pre>
<p>I have:</p>
<pre><code class="lang-auto">$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243



$ nvidia-smi
Wed Dec  9 15:14:27 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 2070    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   49C    P8    10W /  N/A |   3121MiB /  7982MiB |     11%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1364      G   /usr/lib/xorg/Xorg                816MiB |
|    0   N/A  N/A      1797      G   /usr/bin/gnome-shell              516MiB |
|    0   N/A  N/A      3284      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      3506      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      4545      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      7443      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     37638      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     37787      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     69220      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     74559      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     77168      G   ...AAAAAAAAA= --shared-files      136MiB |
|    0   N/A  N/A     77506      C   ...mona/anaconda3/bin/python     1621MiB |
+-----------------------------------------------------------------------------+
</code></pre>

ptrblck: <p>Double post from <a href="https://github.com/pytorch/pytorch/issues/49116">here</a>.</p>

sandipan211: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , I am using a code for which I am getting the same error. Here is my environment:</p>
<pre><code class="lang-auto">(zsd_env2) root@2d3b19630ea3:/workspace/sandipan/sushil/zsd# conda info

     active environment : zsd_env2
    active env location : /opt/conda/envs/zsd_env2
            shell level : 2
       user config file : /root/.condarc
 populated config files :
          conda version : 4.14.0
    conda-build version : not installed
         python version : 3.7.0.final.0
       virtual packages : __cuda=11.4=0
                          __linux=5.4.0=0
                          __glibc=2.23=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.14.0 requests/2.28.1 CPython/3.7.0 Linux/5.4.0-113-generic ubuntu/16.04.4 glibc/2.23
                UID:GID : 0:0
             netrc file : None
           offline mode : False
</code></pre>
<p>I am using Pytorch 1.1.0. The environment is as follows:</p>
<pre><code class="lang-auto">(zsd_env2) root@2d3b19630ea3:/workspace/sandipan/sushil/zsd# conda list -n zsd_env2
# packages in environment at /opt/conda/envs/zsd_env2:
#
# Name                    Version                   Build  Channel
_anaconda_depends         2019.10                  py37_0
_ipyw_jlab_nb_ext_conf    0.1.0                    py37_0
_libgcc_mutex             0.1                        main    conda-forge
addict                    2.2.1                    pypi_0    pypi
alabaster                 0.7.12                   py37_0
albumentations            0.5.0                    pypi_0    pypi
anaconda                  custom                   py37_1
anaconda-client           1.7.2                    py37_0
anaconda-navigator        1.9.7                    py37_0
anaconda-project          0.8.2                    py37_0
asn1crypto                0.24.0                   py37_0
astroid                   2.2.5                    py37_0    conda-forge
astropy                   3.1.2            py37h7b6447c_0
atomicwrites              1.3.0                    py37_1
attrs                     19.1.0                   py37_1
babel                     2.6.0                    py37_0
backcall                  0.1.0                    py37_0
backports                 1.0                      py37_1
backports.os              0.1.1                    py37_0
backports.shutil_get_terminal_size 1.0.0                    py37_2
beautifulsoup4            4.7.1                    py37_1
bitarray                  0.8.3            py37h14c3975_0
bkcharts                  0.2                      py37_0
blas                      1.0                         mkl    conda-forge
bleach                    3.1.0                    py37_0
blosc                     1.15.0               hd408876_0
bokeh                     1.0.4                    py37_0
boto                      2.49.0                   py37_0
bottleneck                1.2.1            py37h035aef0_1
bzip2                     1.0.6                h14c3975_5
ca-certificates           2020.6.20            hecda079_0    conda-forge
cairo                     1.14.12              h8948797_3
certifi                   2020.6.20        py37he5f6b98_2    conda-forge
cffi                      1.12.2           py37h2e261b9_1
chardet                   3.0.4                    py37_1
click                     7.0                      py37_0
cloudpickle               0.8.0                    py37_0
clyent                    1.2.2                    py37_1
cmake                     3.14.5               hf94ab9c_0    conda-forge
colorama                  0.4.1                    py37_0
conda                     4.9.0            py37he5f6b98_1    conda-forge
conda-build               3.17.8                   py37_0    conda-forge
conda-env                 2.6.0                         1    conda-forge
conda-package-handling    1.3.11                   py37_0    conda-forge
conda-verify              3.1.1                    py37_0
contextlib2               0.5.5                    py37_0
cryptography              2.6.1            py37h1ba5d50_0
cudatoolkit               9.0                  h13b8566_0
curl                      7.64.0               hbc83047_2
cycler                    0.10.0                   py37_0
cython                    0.29.6           py37he6710b0_0
cytoolz                   0.9.0.1          py37h14c3975_1
dask                      1.1.4                    py37_1
dask-core                 1.1.4                    py37_1
dbus                      1.13.6               h746ee38_0
decorator                 4.4.0                    py37_1
defusedxml                0.5.0                    py37_1
distributed               1.26.0                   py37_1    conda-forge
distro                    1.5.0              pyh9f0ad1d_0    conda-forge
docutils                  0.14                     py37_0
entrypoints               0.3                      py37_0
et_xmlfile                1.0.1                    py37_0
expat                     2.2.6                he6710b0_0
fastcache                 1.0.2            py37h14c3975_2
ffmpeg                    4.0.2                ha0c5888_2    conda-forge
filelock                  3.0.10                   py37_0
flask                     1.0.2                    py37_1
fontconfig                2.13.0               h9420a91_0
freeglut                  3.0.0             hf484d3e_1005    conda-forge
freetype                  2.9.1                h8a8886c_1
fribidi                   1.0.5                h7b6447c_0
fsspec                    0.8.4                      py_0    conda-forge
future                    0.17.1                   py37_0
geos                      3.8.1                he1b5a44_0    conda-forge
get_terminal_size         1.0.0                haa9412d_0
gevent                    1.4.0            py37h7b6447c_0
glib                      2.56.2               hd408876_0
glob2                     0.6                      py37_1
gmp                       6.1.2                h6c8ec71_1
gmpy2                     2.0.8            py37h10f8cd9_2
gnutls                    3.5.19               h2a4e5f8_1    conda-forge
graphite2                 1.3.13               h23475e2_0
greenlet                  0.4.15           py37h7b6447c_0
gst-plugins-base          1.14.0               hbbd80ab_1
gstreamer                 1.14.0               hb453b48_1
h5py                      2.8.0           py37h3010b51_1003    conda-forge
harfbuzz                  1.8.8                hffaf4a1_0
hdf5                      1.10.2               hc401514_3    conda-forge
heapdict                  1.0.0                    py37_2
html5lib                  1.0.1                    py37_0
icu                       58.2                 h9c2bf20_1
idna                      2.8                      py37_0
imagecorruptions          1.1.0                    pypi_0    pypi
imageio                   2.5.0                    py37_0    conda-forge
imagesize                 1.1.0                    py37_0
imgaug                    0.4.0                      py_0    conda-forge
importlib-metadata        4.12.0                   pypi_0    pypi
intel-openmp              2019.3                      199
ipykernel                 5.1.0            py37h39e3cac_0
ipython                   7.4.0            py37h39e3cac_0
ipython_genutils          0.2.0                    py37_0
ipywidgets                7.4.2                    py37_0
isort                     4.3.16                   py37_0    conda-forge
itsdangerous              1.1.0                    py37_0
jasper                    2.0.14               h07fcdf6_1
jbig                      2.1                  hdba287a_0
jdcal                     1.4                      py37_0
jedi                      0.13.3                   py37_0    conda-forge
jeepney                   0.4                      py37_0
jinja2                    2.10                     py37_0
joblib                    0.17.0                     py_0    conda-forge
jpeg                      9b                   h024ee3a_2
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsonschema                3.0.1                    py37_0    conda-forge
jupyter                   1.0.0                    py37_7
jupyter_client            5.2.4                    py37_0
jupyter_console           6.0.0                    py37_0
jupyter_core              4.4.0                    py37_0
jupyterlab                0.35.4           py37hf63ae98_0
jupyterlab_server         0.2.0                    py37_0
keyring                   18.0.0                   py37_0    conda-forge
kiwisolver                1.0.1            py37hf484d3e_0
kmeans-pytorch            0.3                      pypi_0    pypi
krb5                      1.16.1               h173b8e3_7
lazy-object-proxy         1.3.1            py37h14c3975_2
libarchive                3.3.3                h5d8350f_5
libcurl                   7.64.0               h20c2e04_2
libedit                   3.1.20181209         hc058e9b_0
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.3.0                hdf63c60_0
libglu                    9.0.0             he1b5a44_1001    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblief                   0.9.0                h7725739_2
libopencv                 3.4.2                hb342d67_1
libpng                    1.6.36               hbc83047_0
libsodium                 1.0.16               h1bed415_0
libssh2                   1.8.0                h1ba5d50_4
libstdcxx-ng              8.2.0                hdf63c60_1
libtiff                   4.0.10               h2733197_2
libtool                   2.4.6                h7b6447c_5
libuuid                   1.0.3                h1bed415_2
libuv                     1.40.0               hd18ef5c_0    conda-forge
libxcb                    1.13                 h1bed415_1
libxml2                   2.9.9                he19cac6_0
libxslt                   1.1.33               h7d1a2b0_0
llvmlite                  0.28.0           py37hd408876_0
locket                    0.2.0                    py37_1
lxml                      4.3.2            py37hefd8a0e_0
lz4-c                     1.8.1.2              h14c3975_0
lzo                       2.10                 h49e0be7_2
markupsafe                1.1.1            py37h7b6447c_0
matplotlib                3.0.3            py37h5429711_0
mccabe                    0.6.1                    py37_1
mistune                   0.8.4            py37h7b6447c_0
mkl                       2019.3                      199
mkl-service               1.1.2            py37he904b0f_5
mkl_fft                   1.0.10           py37ha843d7b_0
mkl_random                1.0.2            py37hd81dba3_0
mmcv                      0.2.10                   pypi_0    pypi
mmdet                     1.0rc0+1c9afec            dev_0    &lt;develop&gt;
mock                      4.0.2            py37hc8dfbb8_1    conda-forge
more-itertools            6.0.0                    py37_0
mpc                       1.1.0                h10f8cd9_1
mpfr                      4.0.1                hdf1c602_3
mpmath                    1.1.0                    py37_0
msgpack-python            0.6.1            py37hfd86e86_1
multipledispatch          0.6.0                    py37_0
navigator-updater         0.2.1                    py37_0
nbconvert                 5.4.1                    py37_3
nbformat                  4.4.0                    py37_0
ncurses                   6.1                  he6710b0_1
nettle                    3.3                           0    conda-forge
networkx                  2.2                      py37_1
ninja                     1.9.0            py37hfd86e86_0
nltk                      3.4                      py37_1
nose                      1.3.7                    py37_2    conda-forge
notebook                  5.7.8                    py37_0    conda-forge
numba                     0.43.1           py37h962f231_0
numexpr                   2.6.9            py37h9e4a6bb_0
numpy                     1.21.6                   pypi_0    pypi
numpydoc                  0.8.0                    py37_0
olefile                   0.46                     py37_0
opencv                    3.4.2            py37h6fd60c2_1
opencv-python             4.6.0.66                 pypi_0    pypi
opencv-python-headless    4.6.0.66                 pypi_0    pypi
openh264                  1.8.0             hdbcaa40_1000    conda-forge
openpyxl                  2.6.1                    py37_1
openssl                   1.1.1h               h516909a_0    conda-forge
packaging                 21.3                     pypi_0    pypi
pandas                    0.24.2           py37he6710b0_0
pandoc                    2.2.3.2                       0
pandocfilters             1.4.2                    py37_1
pango                     1.42.4               h049681c_0
parso                     0.3.4                    py37_0
partd                     0.3.10                   py37_1
patchelf                  0.9                  he6710b0_3
path.py                   11.5.0                   py37_0
pathlib2                  2.3.3                    py37_0
patsy                     0.5.1                    py37_0
pcre                      8.43                 he6710b0_0
pep8                      1.7.1                    py37_0
pexpect                   4.6.0                    py37_0    conda-forge
pickleshare               0.7.5                    py37_0
pillow                    9.2.0                    pypi_0    pypi
pip                       22.2.2                   pypi_0    pypi
pixman                    0.38.0               h7b6447c_0
pkginfo                   1.5.0.1                  py37_0
pluggy                    0.9.0                    py37_0
ply                       3.11                     py37_0
progressbar2              4.0.0                    pypi_0    pypi
prometheus_client         0.6.0                    py37_0
prompt_toolkit            2.0.9                    py37_0
psutil                    5.6.1            py37h7b6447c_0
ptyprocess                0.6.0                    py37_0    conda-forge
py                        1.8.0                    py37_0
py-lief                   0.9.0            py37h7725739_2
py-opencv                 3.4.2            py37hb342d67_1
pycocotools               2.0.2                    pypi_0    pypi
pycodestyle               2.5.0                    py37_0
pycosat                   0.6.3            py37h14c3975_0
pycparser                 2.19                     py37_0
pycrypto                  2.6.1            py37h14c3975_9
pycurl                    7.43.0.2         py37h1ba5d50_0
pyflakes                  2.1.1                    py37_0
pygments                  2.3.1                    py37_0
pylint                    2.3.1                    py37_0    conda-forge
pyodbc                    4.0.26           py37he6710b0_0
pyopenssl                 19.0.0                   py37_0    conda-forge
pyparsing                 2.3.1                    py37_0
pyqt                      5.9.2            py37h05f1152_2
pyrsistent                0.14.11          py37h7b6447c_0
pysocks                   1.6.8                    py37_0
pytables                  3.4.4            py37ha205bf6_0
pytest                    4.3.1                    py37_0    conda-forge
pytest-arraydiff          0.3              py37h39e3cac_0
pytest-astropy            0.5.0                    py37_0
pytest-doctestplus        0.3.0                    py37_0
pytest-openfiles          0.3.2                    py37_0
pytest-remotedata         0.3.1                    py37_0
python                    3.7.3                h0371630_0
python-dateutil           2.8.0                    py37_0
python-libarchive-c       2.8                      py37_6
python-utils              3.1.0                    pypi_0    pypi
python_abi                3.7                     1_cp37m    conda-forge
pytorch                   1.1.0           py3.7_cuda9.0.176_cudnn7.5.1_0    pytorch
pytz                      2018.9                   py37_0
pywavelets                1.1.1            py37h03ebfcd_1    conda-forge
pyyaml                    5.1              py37h7b6447c_0
pyzmq                     18.0.0           py37he6710b0_0
qt                        5.9.7                h5867ecd_1
qtawesome                 0.5.7                    py37_1
qtconsole                 4.4.3                    py37_0
qtpy                      1.7.0                    py37_1
readline                  7.0                  h7b6447c_5
requests                  2.21.0                   py37_0
rhash                     1.3.6             h516909a_1001    conda-forge
ripgrep                   12.1.1               h516909a_1    conda-forge
rope                      0.12.0                   py37_0
ruamel_yaml               0.15.46          py37h14c3975_0
scikit-build              0.11.1           py37h3340039_2    conda-forge
scikit-image              0.19.3                   pypi_0    pypi
scikit-learn              0.20.3           py37hd81dba3_0
scipy                     1.7.3                    pypi_0    pypi
seaborn                   0.9.0                    py37_0
secretstorage             3.1.1                    py37_0    conda-forge
send2trash                1.5.0                    py37_0
setuptools                65.3.0                   pypi_0    pypi
shapely                   1.7.0            py37hc88ce51_3    conda-forge
simplegeneric             0.8.1                    py37_2
singledispatch            3.4.0.3                  py37_0
sip                       4.19.8           py37hf484d3e_0
six                       1.12.0                   py37_0
snappy                    1.1.7                hbae5bb6_3
snowballstemmer           1.2.1                    py37_0
sortedcollections         1.1.2                    py37_0
sortedcontainers          2.1.0                    py37_0
soupsieve                 1.8                      py37_0    conda-forge
sphinx                    1.8.5                    py37_0    conda-forge
sphinxcontrib             1.0                      py37_1
sphinxcontrib-applehelp   1.0.2                      py_0    conda-forge
sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge
sphinxcontrib-htmlhelp    1.0.3                      py_0    conda-forge
sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge
sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
sphinxcontrib-serializinghtml 1.1.4                      py_0    conda-forge
sphinxcontrib-websupport  1.1.0                    py37_1
spyder                    3.3.3                    py37_0
spyder-kernels            0.4.2                    py37_0    conda-forge
sqlalchemy                1.3.1            py37h7b6447c_0
sqlite                    3.27.2               h7b6447c_0
statsmodels               0.9.0            py37h035aef0_0
sympy                     1.3                      py37_0    conda-forge
tbb                       2020.2               hc9558a2_0    conda-forge
tblib                     1.3.2                    py37_0
terminado                 0.8.1                    py37_1    conda-forge
terminaltables            3.1.10                   pypi_0    pypi
testpath                  0.4.2                    py37_0
tifffile                  2021.11.2                pypi_0    pypi
tk                        8.6.8                hbc83047_0
toolz                     0.9.0                    py37_0
torchvision               0.3.0           py37_cu9.0.176_1    pytorch
tornado                   6.0.2            py37h7b6447c_0
tqdm                      4.31.1                   py37_1
traitlets                 4.3.2                    py37_0    conda-forge
typing-extensions         4.3.0                    pypi_0    pypi
unicodecsv                0.14.1                   py37_0
unixodbc                  2.3.7                h14c3975_0
urllib3                   1.24.1                   py37_0
wcwidth                   0.1.7                    py37_0
webencodings              0.5.1                    py37_1
werkzeug                  0.14.1                   py37_0
wheel                     0.37.1                   pypi_0    pypi
widgetsnbextension        3.4.2                    py37_0
wrapt                     1.11.1           py37h7b6447c_0
wurlitzer                 1.0.2                    py37_0
x264                      1!152.20180806       h14c3975_0    conda-forge
xlrd                      1.2.0                    py37_0
xlsxwriter                1.1.5                    py37_0
xlwt                      1.3.0                    py37_0
xorg-fixesproto           5.0               h14c3975_1002    conda-forge
xorg-inputproto           2.3.2             h14c3975_1002    conda-forge
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libx11               1.6.12               h516909a_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxfixes            5.0.3             h516909a_1004    conda-forge
xorg-libxi                1.7.10               h516909a_0    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge
xz                        5.2.4                h14c3975_4
yaml                      0.1.7                had09818_2
zeromq                    4.3.1                he6710b0_3
zict                      0.1.4                    py37_0
zipp                      3.8.1                    pypi_0    pypi
zlib                      1.2.11               h7b6447c_3
zstd                      1.3.7                h0b5b093_0
</code></pre>
<pre><code class="lang-auto">(zsd_env2) root@2d3b19630ea3:/workspace/sandipan/sushil/zsd# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
</code></pre>
<p>I am getting the error during backward() call:</p>
<pre><code class="lang-auto">MLP_CRITIC(
  (fc1): Linear(in_features=1324, out_features=4096, bias=True)
  (fc2): Linear(in_features=4096, out_features=1, bias=True)
  (lrelu): LeakyReLU(negative_slope=0.2, inplace)
)

#############################################################


Traceback (most recent call last):
  File "trainer.py", line 81, in &lt;module&gt;
    trainFGGAN(epoch, features, labels)
  File "/workspace/sandipan/sushil/zsd/train_gan.py", line 146, in __call__
    self.trainEpoch()
  File "/workspace/sandipan/sushil/zsd/train_gan.py", line 283, in trainEpoch
    criticD_real.backward(self.mone)
  File "/opt/conda/envs/zsd_env2/lib/python3.7/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/envs/zsd_env2/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653215914/work/aten/src/THC/THCBlas.cu:259
</code></pre>
<p>Changing pytorch version for me is infeasible as its showing lot of package incompatibilities if I change the pytorch version in my environment.yml. What should i do then?</p>

ptrblck: <p>I’ve answered in the cross-post.<br>
Fixes won’t be backported to old PyTorch releases (in this case <code>1.1.0</code>) so you should try to update to the latest release and check if this issue is fixed.</p>