Multiprocessing cuda tensor proliferation on 'cuda:0'

bapi: <p>Hi,</p>
<p>My program implements sharing GPU models/tensors in a class by multiple processes. Below is an MWP:</p>
<pre><code class="lang-auto">import time
import torch
import multiprocessing as mp
import torch.multiprocessing as tmp


def work(rank, t):
    print("Working")
    time.sleep(10)
    return


class Tester(object):
    def __init__(self, num_proc):
        self.device = 'cuda:1'
        self.tensor = torch.zeros(1).to(self.device)
        self.np = num_proc

    def run(self):
        procs = [mp.Process(target=work, args=(rank, self,)) for rank in range(self.np)]
        [p.start() for p in procs]
        [p.join() for p in procs]
        # tmp.spawn(work, nprocs=self.np, args=(self,))
        

def main():
    t = Tester(4)
    t.run()


if __name__ == '__main__':
    mp.set_start_method('spawn', force=True)
    main()
</code></pre>
<p>Essentially, the processes are sharing a tensor which is on ‘cuda:1’. On starting the program, gpustat looks like the following:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6a2fc9c78557355e8ab18747ce8808de97661ab.png" data-download-href="https://discuss.pytorch.org/uploads/default/d6a2fc9c78557355e8ab18747ce8808de97661ab" title="Screenshot from 2020-02-13 16-08-40"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6a2fc9c78557355e8ab18747ce8808de97661ab_2_690x122.png" alt="Screenshot from 2020-02-13 16-08-40" data-base62-sha1="uCLnk826rT6mIMFT22oYKfxuaQH" width="690" height="122" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6a2fc9c78557355e8ab18747ce8808de97661ab_2_690x122.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6a2fc9c78557355e8ab18747ce8808de97661ab_2_1035x183.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6a2fc9c78557355e8ab18747ce8808de97661ab_2_1380x244.png 2x" data-dominant-color="391F32"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-02-13 16-08-40</span><span class="informations">2086×370 72.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
However, just before the exit, gpustat shows the following memory use:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/91b085e9055649fe407b5845f5d530bbe55c92d5.png" data-download-href="https://discuss.pytorch.org/uploads/default/91b085e9055649fe407b5845f5d530bbe55c92d5" title="Screenshot from 2020-02-13 16-08-46"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/91b085e9055649fe407b5845f5d530bbe55c92d5_2_690x146.png" alt="Screenshot from 2020-02-13 16-08-46" data-base62-sha1="kMPqNjFd007RaPdaXEV0HsfShW5" width="690" height="146" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/91b085e9055649fe407b5845f5d530bbe55c92d5_2_690x146.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/91b085e9055649fe407b5845f5d530bbe55c92d5_2_1035x219.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/91b085e9055649fe407b5845f5d530bbe55c92d5_2_1380x292.png 2x" data-dominant-color="361E32"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-02-13 16-08-46</span><span class="informations">2122×451 122 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Now, in my actual implementation, wherein I have multiple processes sharing models on each of the GPUs in the system, this results in not enough memory on ‘cuda:0’ and some cudNN errors.</p>
<p>I checked if torch.multiprocessing wrapper, instead of python multiprocessing, has the same issue; and, it does.</p>
<p>What exactly is going wrong here?</p>
<p>Thank you.</p>

bapi: <p>Furthermore, it seems that the memory is actually filled with garbage; notice the size occupied (&gt;2100MB) for just a torch.zeros(1)!</p>
<p>Moreover, I notice that when the device on which processes are run is set as ‘cuda:0’, ‘cuda:1’ gets memory allocated over as 12 MB.</p>
<p>What is exactly happening here?</p>

albanD: <p>So you means that, on exit, we initialize cuda on device 0?</p>
<p>Note that one way to avoid this is to use <code>CUDA_VISIBLE_DEVICES=1</code> to make sure you never use other GPUs.</p>

bapi: <p>Thanks for the response.<br>
However, my requirement is not running the code on one of the GPUs; indeed, I need to run the code on all the GPUs available.</p>

albanD: <p>But this env variable can be set on a per-process basis. From what I understand above, a process should only use one GPU no?</p>
<p>The problem you’re seeing is that a worker that should run only on GPU1 actually use GPU0 right?</p>

bapi: <p>Perfect! Thanks for this idea. It worked.</p>
<p>However, the other issue is still persisting. Only a torch.zeros(1), or for that matter, any small tensor takes up that huge sized memory of roughly 2000 MB. Any idea why that is happening?</p>
<p>Thanks.</p>

albanD: <p>Yes, that memory is used by the cuda driver and runtime. Just doing a CudaInit eats up all that memory :’(<br>
This is why we try to only init on the GPU that are actually used.</p>

bapi: <p>Thanks for this info.<br>
So, in actuality, I should safely leave out that much memory from my overall requirement?</p>

albanD: <p>Yes you need to leave out this much memory for each process.</p>