Cuda Kernel loaded in memory for processes not using GPU

username: alikmeta
cooked: <p>I am training my models in a local machine with a GPU available. My particular application is in RL, where I have multiple workers working in parallel (implemented with python3 multiprocessing) collecting experience using my torch models on the cpu since only forward passes are needed. Periodically I also have a main process, that trains an instance of the model on the GPU and then updates the weights of the worker models.<br>
The problem that I  face is that, also the worker processes take around 800 MBs of memory on the GPU (I assume for the CUDA kernels) even though they never access the GPU.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f569dde0bbff54b68ca340ec253f114ca6038c7.png" data-download-href="https://discuss.pytorch.org/uploads/default/5f569dde0bbff54b68ca340ec253f114ca6038c7" title="Screenshot from 2021-11-04 12-39-45"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f569dde0bbff54b68ca340ec253f114ca6038c7.png" alt="Screenshot from 2021-11-04 12-39-45" data-base62-sha1="dBoVYkPA4nlActV7jgT7IPqu0oD" width="636" height="500" data-dominant-color="39172F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-11-04 12-39-45</span><span class="informations">722×567 69.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The screenshot shown, presents an exemple view of nvidia-smi. In this example I am using a simple MLP model (2MB of memory needed), and I am using 10 worker processes, which can be seen to take 797 MBs of memory each, and the master process (id 1707716) that takes 799 ( 2 MB more because of the model memory).</p>
<p>I am not adding portions of the code, as It is pretty complex and spread around multiple files but I wanted to ask if there is a way to avoid loading the kernel in the GPU memory for every process using torch or if there is a way to free it.</p>

username: alikmeta
cooked: <p>I am also uploading a simplified scheme of my program, that underlines that torch is imported and used after creating the worker processes.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b78f97f59f40de2168f615fb42a0a029e7a97de.png" alt="Screenshot from 2021-11-04 13-02-25" data-base62-sha1="3V24S5hkU6s6cGUhtkpzCr8ClUa" width="434" height="482"></p>

username: ptrblck
cooked: <p>Based on the output of <code>nvidia-smi</code> you are initializing a CUDA context in each worker. I don’t know how you are spawning them but you might want to check if you are (accidentally) initializing it or could also try to mask the device in the workers.</p>

username: alikmeta
cooked: <p>Thank you for your reply and sorry for the delayed response!<br>
I have tried to mask the device in the worker processes, in their <strong>init</strong> method of the worker processes, by setting “CUDA_VISIBLE_DEVICES”="" in the environment, but this has also not worked. I have also checked for accidental references to cuda without a finding any.<br>
I produced a minimum working example of the problem in this repo: <a href="https://github.com/amarildolikmeta/torch_workers_gpu" class="inline-onebox" rel="noopener nofollow ugc">GitHub - amarildolikmeta/torch_workers_gpu</a><br>
It requires open ai gym (it includes a simple test environment).</p>
<p>The main script initializes the worker processes then runs a loop alternating between the main process running some episodes in the gpu, followed by 2 workers processes running some episodes with the models in the CPU.</p>
<p>This minimum example reproduces this behaviour in my ubuntu 18.04, with python 3.6 and torch 1.10</p>

username: alikmeta
cooked: <p>I found the problem, and as you sad it was an “accidental” access to the cuda context, done when I copied the parameters between the model of the main process (in gpu) to the models of the workers processes (in cpu). This was done through the state_dict method, without mapping the tensors in the state_dict to the CPU before calling load_state_dict in the worker models.<br>
Thanks again for the help!</p>