.cuda() freezes the python

username: bmkor
cooked: <p>Hi guys. I am a newbie in <code>pytorch</code>.</p>
<p>My Ubuntu 16.04.02 LTS got python2.7 and cuda 9.0 installed (<code>torch.cuda.is_available()</code> returns <code>True</code>). However, when I tried run the following, my python just got frozen and I can only type <code>ctrl+z</code> to force exit.</p>
<p><code>import torch</code><br>
<code>torch.randn(10).cuda()</code></p>
<p>Then freeze until <code>ctrl+z</code>.</p>
<p>Did I do something wrong? Anyone could help? Thanks a lot.</p>

username: Spencer_ML
cooked: <p>Hi, i see similar issue during use CUDA.</p>
<p>And I try to check GPU memory, it will goes up until ~229MB. Python then can work normally.</p>
<p>I am not sure why. If you find a way to fix it, please tell me. thanks</p>

username: lgvaz
cooked: <p>I’m facing the same issue.<br>
I’m running on jupyter notebook, the first call to <code>torch.randn(10).cuda()</code> takes a long loooong time to complete (~7 min), but subsequent calls behave normally.</p>
<p>I’m posting a screenshoot of the stack when interrupting the kernel.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/88ce7398ef06f0af2fdef0ac3660390e2461059c.png" data-download-href="https://discuss.pytorch.org/uploads/default/88ce7398ef06f0af2fdef0ac3660390e2461059c" title="cuda_slow"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/88ce7398ef06f0af2fdef0ac3660390e2461059c_2_690x278.png" alt="cuda_slow" data-base62-sha1="jwffTzo7g8Rb3DfSn5XmURgcku8" width="690" height="278" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/88ce7398ef06f0af2fdef0ac3660390e2461059c_2_690x278.png, https://discuss.pytorch.org/uploads/default/optimized/2X/8/88ce7398ef06f0af2fdef0ac3660390e2461059c_2_1035x417.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/8/88ce7398ef06f0af2fdef0ac3660390e2461059c.png 2x" data-dominant-color="F9F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cuda_slow</span><span class="informations">1071×433 44 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: richard
cooked: <p>What is the output of your nvidia-smi when this is happening?<br>
Could you try building PyTorch from source and seeing if the long cuda() calls still occur?</p>

username: lgvaz
cooked: <p>This is the output of nvidia-smi (fastai/bin/python is the process related to the notebook):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/4/4df3235ff923e84c03f13738bcd64b8ca91e2d2c.png" alt="image" data-base62-sha1="b7zGyrtpRnlWIcZMeWkUnmnTo4s" width="640" height="359"><br>
I will try to build from source.</p>

username: lgvaz
cooked: <p>I couldn’t manage to build from source, first I was getting <a href="https://discuss.pytorch.org/t/error-expected-gloo-use-cuda-to-be-defined/5333">this</a> error, I followed the workaround but then got an error regarding cuDNN version, my cuDNN installation is a mess because of tensorflow, so I didn’t wanted to mess with that afraid that it could mess my tensorflow installation. So I switched to docker, and the first run of <code>torch.randn(10).cuda()</code> is taking ~2 sec, which I think is expected.</p>

username: smth
cooked: <p><a class="mention" href="/u/lgvaz">@lgvaz</a> you have a 1070 GPU, but you must’ve installed pytorch for CUDA 7.5. What it is doing is recompiling pytorch for your GPU (cuda8 supported GPU).<br>
If you reinstall the CUDA8 version of pytorch, the freezing wont happen. Select the right option in our version selector on <a href="http://pytorch.org">http://pytorch.org</a></p>

username: Spencer_ML
cooked: <p>Hi smth,</p>
<p>i also see the same issue even with CUDA8.</p>

username: richard
cooked: <p><a class="mention" href="/u/spencer_ml">@Spencer_ML</a> It’s possible that there’s a mismatch between the CUDA version your pytorch was compiled with, and the CUDA version you’re running your computer on. How did you install pytorch?</p>

username: bmkor
cooked: <p>Thanks all for the discussion.<br>
Have successfully tried cuda() but on the first run of .cuda(), I found that I have to wait for 5 to 10 minutes. Is it normal or should I do some config first?</p>

username: QuantScientist
cooked: <p>Can you run the following and report the output?</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/QuantScientist/Deep-Learning-Boot-Camp/blob/master/day02-PyTORCH-and-PyCUDA/PyCUDA/01%20PyCUDA%20verify%20CUDA%208.0.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/QuantScientist/Deep-Learning-Boot-Camp/blob/master/day02-PyTORCH-and-PyCUDA/PyCUDA/01%20PyCUDA%20verify%20CUDA%208.0.ipynb" target="_blank" rel="nofollow noopener">QuantScientist/Deep-Learning-Boot-Camp/blob/master/day02-PyTORCH-and-PyCUDA/PyCUDA/01 PyCUDA verify CUDA 8.0.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Deep Learning Bootcamp November 2017, GPU Computing for Data Scientists\n",
    "\n",
    "&lt;img src=\"../images/bcamp.png\" align=\"center\"&gt;\n",
    "\n",
    "## Using CUDA, Jupyter, PyCUDA and PyTorch\n",
    "\n",
    "### 01 PyCUDA verify CUDA 8.0\n",
    "\n",
    "Web: https://www.meetup.com/Tel-Aviv-Deep-Learning-Bootcamp/events/241762893/\n",
    "\n",
</code></pre>

  This file has been truncated. <a href="https://github.com/QuantScientist/Deep-Learning-Boot-Camp/blob/master/day02-PyTORCH-and-PyCUDA/PyCUDA/01%20PyCUDA%20verify%20CUDA%208.0.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/QuantScientist/Deep-Learning-Boot-Camp/blob/master/day02-PyTORCH-and-PyCUDA/PyTorch/01%20PyTorch%20GPU%20support%20test.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/QuantScientist/Deep-Learning-Boot-Camp/blob/master/day02-PyTORCH-and-PyCUDA/PyTorch/01%20PyTorch%20GPU%20support%20test.ipynb" target="_blank" rel="nofollow noopener">QuantScientist/Deep-Learning-Boot-Camp/blob/master/day02-PyTORCH-and-PyCUDA/PyTorch/01 PyTorch GPU support test.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "% reset -f\n",
    "from __future__ import print_function\n",
    "from __future__ import division\n",
    "import math\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "# ! pip install torchvision"
   ]
  },
</code></pre>

  This file has been truncated. <a href="https://github.com/QuantScientist/Deep-Learning-Boot-Camp/blob/master/day02-PyTORCH-and-PyCUDA/PyTorch/01%20PyTorch%20GPU%20support%20test.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: bmkor
cooked: <p><a class="mention" href="/u/quantscientist">@QuantScientist</a>  Sure. Here are my outputs:<br>
##############################################<br>
#########<span class="hashtag">#01</span> PyCUDA verify CUDA 8.0.ipynb########</p>
<p>1 device(s) found.<br>
Device <span class="hashtag">#0:</span> GeForce GTX 1080 Ti<br>
Compute Capability: 6.1<br>
Total Memory: 11440512 KB<br>
ASYNC_ENGINE_COUNT: 2<br>
CAN_MAP_HOST_MEMORY: 1<br>
CLOCK_RATE: 1582000<br>
COMPUTE_CAPABILITY_MAJOR: 6<br>
COMPUTE_CAPABILITY_MINOR: 1<br>
COMPUTE_MODE: DEFAULT<br>
CONCURRENT_KERNELS: 1<br>
ECC_ENABLED: 0<br>
GLOBAL_L1_CACHE_SUPPORTED: 1<br>
GLOBAL_MEMORY_BUS_WIDTH: 352<br>
GPU_OVERLAP: 1<br>
INTEGRATED: 0<br>
KERNEL_EXEC_TIMEOUT: 1<br>
L2_CACHE_SIZE: 2883584<br>
LOCAL_L1_CACHE_SUPPORTED: 1<br>
MANAGED_MEMORY: 1<br>
MAXIMUM_SURFACE1D_LAYERED_LAYERS: 2048<br>
MAXIMUM_SURFACE1D_LAYERED_WIDTH: 32768<br>
MAXIMUM_SURFACE1D_WIDTH: 32768<br>
MAXIMUM_SURFACE2D_HEIGHT: 65536<br>
MAXIMUM_SURFACE2D_LAYERED_HEIGHT: 32768<br>
MAXIMUM_SURFACE2D_LAYERED_LAYERS: 2048<br>
MAXIMUM_SURFACE2D_LAYERED_WIDTH: 32768<br>
MAXIMUM_SURFACE2D_WIDTH: 131072<br>
MAXIMUM_SURFACE3D_DEPTH: 16384<br>
MAXIMUM_SURFACE3D_HEIGHT: 16384<br>
MAXIMUM_SURFACE3D_WIDTH: 16384<br>
MAXIMUM_SURFACECUBEMAP_LAYERED_LAYERS: 2046<br>
MAXIMUM_SURFACECUBEMAP_LAYERED_WIDTH: 32768<br>
MAXIMUM_SURFACECUBEMAP_WIDTH: 32768<br>
MAXIMUM_TEXTURE1D_LAYERED_LAYERS: 2048<br>
MAXIMUM_TEXTURE1D_LAYERED_WIDTH: 32768<br>
MAXIMUM_TEXTURE1D_LINEAR_WIDTH: 134217728<br>
MAXIMUM_TEXTURE1D_MIPMAPPED_WIDTH: 16384<br>
MAXIMUM_TEXTURE1D_WIDTH: 131072<br>
MAXIMUM_TEXTURE2D_ARRAY_HEIGHT: 32768<br>
MAXIMUM_TEXTURE2D_ARRAY_NUMSLICES: 2048<br>
MAXIMUM_TEXTURE2D_ARRAY_WIDTH: 32768<br>
MAXIMUM_TEXTURE2D_GATHER_HEIGHT: 32768<br>
MAXIMUM_TEXTURE2D_GATHER_WIDTH: 32768<br>
MAXIMUM_TEXTURE2D_HEIGHT: 65536<br>
MAXIMUM_TEXTURE2D_LINEAR_HEIGHT: 65000<br>
MAXIMUM_TEXTURE2D_LINEAR_PITCH: 2097120<br>
MAXIMUM_TEXTURE2D_LINEAR_WIDTH: 131072<br>
MAXIMUM_TEXTURE2D_MIPMAPPED_HEIGHT: 32768<br>
MAXIMUM_TEXTURE2D_MIPMAPPED_WIDTH: 32768<br>
MAXIMUM_TEXTURE2D_WIDTH: 131072<br>
MAXIMUM_TEXTURE3D_DEPTH: 16384<br>
MAXIMUM_TEXTURE3D_DEPTH_ALTERNATE: 32768<br>
MAXIMUM_TEXTURE3D_HEIGHT: 16384<br>
MAXIMUM_TEXTURE3D_HEIGHT_ALTERNATE: 8192<br>
MAXIMUM_TEXTURE3D_WIDTH: 16384<br>
MAXIMUM_TEXTURE3D_WIDTH_ALTERNATE: 8192<br>
MAXIMUM_TEXTURECUBEMAP_LAYERED_LAYERS: 2046<br>
MAXIMUM_TEXTURECUBEMAP_LAYERED_WIDTH: 32768<br>
MAXIMUM_TEXTURECUBEMAP_WIDTH: 32768<br>
MAX_BLOCK_DIM_X: 1024<br>
MAX_BLOCK_DIM_Y: 1024<br>
MAX_BLOCK_DIM_Z: 64<br>
MAX_GRID_DIM_X: 2147483647<br>
MAX_GRID_DIM_Y: 65535<br>
MAX_GRID_DIM_Z: 65535<br>
MAX_PITCH: 2147483647<br>
MAX_REGISTERS_PER_BLOCK: 65536<br>
MAX_REGISTERS_PER_MULTIPROCESSOR: 65536<br>
MAX_SHARED_MEMORY_PER_BLOCK: 49152<br>
MAX_SHARED_MEMORY_PER_MULTIPROCESSOR: 98304<br>
MAX_THREADS_PER_BLOCK: 1024<br>
MAX_THREADS_PER_MULTIPROCESSOR: 2048<br>
MEMORY_CLOCK_RATE: 5505000<br>
MULTIPROCESSOR_COUNT: 28<br>
MULTI_GPU_BOARD: 0<br>
MULTI_GPU_BOARD_GROUP_ID: 0<br>
PCI_BUS_ID: 1<br>
PCI_DEVICE_ID: 0<br>
PCI_DOMAIN_ID: 0<br>
STREAM_PRIORITIES_SUPPORTED: 1<br>
SURFACE_ALIGNMENT: 512<br>
TCC_DRIVER: 0<br>
TEXTURE_ALIGNMENT: 512<br>
TEXTURE_PITCH_ALIGNMENT: 32<br>
TOTAL_CONSTANT_MEMORY: 65536<br>
UNIFIED_ADDRESSING: 1<br>
WARP_SIZE: 32</p>
<p>##############################################</p>
<p>Global memory occupancy:98.000000% free</p>
<p>===Attributes for device 0<br>
MAX_THREADS_PER_BLOCK:1024<br>
MAX_BLOCK_DIM_X:1024<br>
MAX_BLOCK_DIM_Y:1024<br>
MAX_BLOCK_DIM_Z:64<br>
MAX_GRID_DIM_X:2147483647<br>
MAX_GRID_DIM_Y:65535<br>
MAX_GRID_DIM_Z:65535<br>
MAX_SHARED_MEMORY_PER_BLOCK:49152<br>
TOTAL_CONSTANT_MEMORY:65536<br>
WARP_SIZE:32<br>
MAX_PITCH:2147483647<br>
MAX_REGISTERS_PER_BLOCK:65536<br>
CLOCK_RATE:1582000<br>
TEXTURE_ALIGNMENT:512<br>
GPU_OVERLAP:1<br>
MULTIPROCESSOR_COUNT:28<br>
KERNEL_EXEC_TIMEOUT:1<br>
INTEGRATED:0<br>
CAN_MAP_HOST_MEMORY:1<br>
COMPUTE_MODE:DEFAULT<br>
MAXIMUM_TEXTURE1D_WIDTH:131072<br>
MAXIMUM_TEXTURE2D_WIDTH:131072<br>
MAXIMUM_TEXTURE2D_HEIGHT:65536<br>
MAXIMUM_TEXTURE3D_WIDTH:16384<br>
MAXIMUM_TEXTURE3D_HEIGHT:16384<br>
MAXIMUM_TEXTURE3D_DEPTH:16384<br>
MAXIMUM_TEXTURE2D_ARRAY_WIDTH:32768<br>
MAXIMUM_TEXTURE2D_ARRAY_HEIGHT:32768<br>
MAXIMUM_TEXTURE2D_ARRAY_NUMSLICES:2048<br>
SURFACE_ALIGNMENT:512<br>
CONCURRENT_KERNELS:1<br>
ECC_ENABLED:0<br>
PCI_BUS_ID:1<br>
PCI_DEVICE_ID:0<br>
TCC_DRIVER:0<br>
MEMORY_CLOCK_RATE:5505000<br>
GLOBAL_MEMORY_BUS_WIDTH:352<br>
L2_CACHE_SIZE:2883584<br>
MAX_THREADS_PER_MULTIPROCESSOR:2048<br>
ASYNC_ENGINE_COUNT:2<br>
UNIFIED_ADDRESSING:1<br>
MAXIMUM_TEXTURE1D_LAYERED_WIDTH:32768<br>
MAXIMUM_TEXTURE1D_LAYERED_LAYERS:2048<br>
MAXIMUM_TEXTURE2D_GATHER_WIDTH:32768<br>
MAXIMUM_TEXTURE2D_GATHER_HEIGHT:32768<br>
MAXIMUM_TEXTURE3D_WIDTH_ALTERNATE:8192<br>
MAXIMUM_TEXTURE3D_HEIGHT_ALTERNATE:8192<br>
MAXIMUM_TEXTURE3D_DEPTH_ALTERNATE:32768<br>
PCI_DOMAIN_ID:0<br>
TEXTURE_PITCH_ALIGNMENT:32<br>
MAXIMUM_TEXTURECUBEMAP_WIDTH:32768<br>
MAXIMUM_TEXTURECUBEMAP_LAYERED_WIDTH:32768<br>
MAXIMUM_TEXTURECUBEMAP_LAYERED_LAYERS:2046<br>
MAXIMUM_SURFACE1D_WIDTH:32768<br>
MAXIMUM_SURFACE2D_WIDTH:131072<br>
MAXIMUM_SURFACE2D_HEIGHT:65536<br>
MAXIMUM_SURFACE3D_WIDTH:16384<br>
MAXIMUM_SURFACE3D_HEIGHT:16384<br>
MAXIMUM_SURFACE3D_DEPTH:16384<br>
MAXIMUM_SURFACE1D_LAYERED_WIDTH:32768<br>
MAXIMUM_SURFACE1D_LAYERED_LAYERS:2048<br>
MAXIMUM_SURFACE2D_LAYERED_WIDTH:32768<br>
MAXIMUM_SURFACE2D_LAYERED_HEIGHT:32768<br>
MAXIMUM_SURFACE2D_LAYERED_LAYERS:2048<br>
MAXIMUM_SURFACECUBEMAP_WIDTH:32768<br>
MAXIMUM_SURFACECUBEMAP_LAYERED_WIDTH:32768<br>
MAXIMUM_SURFACECUBEMAP_LAYERED_LAYERS:2046<br>
MAXIMUM_TEXTURE1D_LINEAR_WIDTH:134217728<br>
MAXIMUM_TEXTURE2D_LINEAR_WIDTH:131072<br>
MAXIMUM_TEXTURE2D_LINEAR_HEIGHT:65000<br>
MAXIMUM_TEXTURE2D_LINEAR_PITCH:2097120<br>
MAXIMUM_TEXTURE2D_MIPMAPPED_WIDTH:32768<br>
MAXIMUM_TEXTURE2D_MIPMAPPED_HEIGHT:32768<br>
COMPUTE_CAPABILITY_MAJOR:6<br>
COMPUTE_CAPABILITY_MINOR:1<br>
MAXIMUM_TEXTURE1D_MIPMAPPED_WIDTH:16384<br>
STREAM_PRIORITIES_SUPPORTED:1<br>
GLOBAL_L1_CACHE_SUPPORTED:1<br>
LOCAL_L1_CACHE_SUPPORTED:1<br>
MAX_SHARED_MEMORY_PER_MULTIPROCESSOR:98304<br>
MAX_REGISTERS_PER_MULTIPROCESSOR:65536<br>
MANAGED_MEMORY:1<br>
MULTI_GPU_BOARD:0<br>
MULTI_GPU_BOARD_GROUP_ID:0</p>
<p>##############################################</p>
<p>##############################################<br>
########<span class="hashtag">#01</span> PyTorch GPU support test.ipynb#########</p>
<p>__Python VERSION:’, ‘2.7.12 (default, Nov 19 2016, 06:48:10) \n[GCC 5.4.0 20160609]<br>
__pyTorch VERSION:’, ‘0.1.12_2<br>
__CUDA VERSION<br>
nvcc: NVIDIA ® Cuda compiler driver<br>
Copyright © 2005-2017 NVIDIA Corporation<br>
Built on Fri_Sep__1_21:08:03_CDT_2017<br>
Cuda compilation tools, release 9.0, V9.0.176<br>
__CUDNN VERSION:’, 6021<br>
__Number CUDA Devices:’, 1L<br>
Active CUDA Device: GPU’, 0L<br>
Available devices ', 1L<br>
Current cuda device ', 0L</p>
<p>##############################################<br>
1 device(s) found.<br>
(0, ‘GeForce GTX 1080 Ti’)</p>
<p>##############################################<br>
True<br>
&lt;class ‘pycuda.gpuarray.GPUArray’&gt;<br>
##############################################</p>

username: smth
cooked: <p><a class="mention" href="/u/bmkor">@bmkor</a> please make sure you install the CUDA8 version of pytorch from the version selector. if you install version of 7.5 then you will have to wait a minute for startup.</p>

username: bmkor
cooked: <p><a class="mention" href="/u/smth">@smth</a> Thanks. Should I down my cuda from 9 to 8 in this case? I am not sure if my cuda is 9 in fact. Execute command <code>nvcc —version</code> gives:</p>
<blockquote>
<p>nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2017 NVIDIA Corporation<br>
Built on Fri_Sep__1_21:08:03_CDT_2017<br>
Cuda compilation tools, release 9.0, V9.0.176</p>
</blockquote>

username: smth
cooked: <p><a class="mention" href="/u/bmkor">@bmkor</a> cuda &gt;=8 should be fine. So if you have cuda9, then you dont need to downgrade.</p>

username: Spencer_ML
cooked: <p>i  install pytorch through anaconda.</p>

username: Lacrymology
cooked: <p>I see a similar issue when running <code>mytensor.to(torch.device('cuda'))</code></p>
<p>here’s the output to several calls to nvidia-smi, notice how the memory usage goes up until ~340MiB and then the call returns</p>
<pre><code class="lang-auto">Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:13 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   43C    P8    N/A /  N/A |      0MiB /   983MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:15 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |     33MiB /   983MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python    22MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:16 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |     94MiB /   983MiB |     15%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python    83MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:17 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |    140MiB /   983MiB |      7%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   130MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:18 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |    173MiB /   983MiB |      5%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   162MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:19 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   44C    P0    N/A /  N/A |    216MiB /   983MiB |      5%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   205MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:20 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   45C    P0    N/A /  N/A |    249MiB /   983MiB |      5%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   238MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:21 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   45C    P0    N/A /  N/A |    292MiB /   983MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   281MiB |
+-----------------------------------------------------------------------------+

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:23 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   45C    P0    N/A /  N/A |    333MiB /   983MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   322MiB |
+-----------------------------------------------------------------------------+
^[[A                                                                                                                                                                                                              

Shiphero-API on  feature/tomas/OP-11-mergeable-order-large-accounts [$?] 
➜ nvidia-smi
Thu Oct 18 19:45:25 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.57                 Driver Version: 410.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940M        Off  | 00000000:04:00.0 Off |                  N/A |
| N/A   45C    P0    N/A /  N/A |    353MiB /   983MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15774      C   ...ymology/.virtualenvs/pytorch/bin/python   342MiB |
+-----------------------------------------------------------------------------+
</code></pre>

username: ghasemikasra39
cooked: <p>I am using <code>Driver Version: 515.43.04</code> with <code>CUDA Version: 11.7</code> and with <code>torch 1.12.1</code> on <code>ubuntu Ubuntu 18.04.6 LTS</code>.</p>
<p>I am facing the same issue. For me, if I reboot the system, it will work only once after that. The second time I try, it does not work.</p>