Cuda/c++ pybind preprocess cost huge on gpu 0

username: Lin1
cooked: <p>hello everyone,</p>
<p>A cuda preprocessor with pybind11 is implemented and imported in Dataloader. DDP with multi-process is used for multi-gpus training(one machine). I find once I increase the num_workers(keep everything else frozen), the gpu 0 is occupied more, leading to gpu 0’s capabilty become the limit of the overall training speed.</p>
<p>Is it possible to move some cuda calculations within the preprocessor to other gpus?</p>
<p>Thanks,<br>
Lin</p>

username: ptrblck
cooked: <p>I don’t know what exactly the “cuda preprocessor with pybind11” is doing in the <code>DataLoader</code> and how it’s implemented, but I would guess you are recreating new CUDA contexts on the default device (GPU0) and are thus increasing the memory usage. Try to use the corresponding GPU used in the DDP process to perform your GPU calculations in the <code>DataLoader</code>.</p>

username: Lin1
cooked: <p>hello <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Please let me provide more details:<br>
The cuda preprocessor is built with cmake as follows<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2f1f9de3c559eac2e064f92cecf8caedb230da0.png" data-download-href="https://discuss.pytorch.org/uploads/default/e2f1f9de3c559eac2e064f92cecf8caedb230da0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2f1f9de3c559eac2e064f92cecf8caedb230da0_2_690x21.png" alt="image" data-base62-sha1="wnEl3t0FO2ITljhMU2exgBzvm0M" width="690" height="21" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2f1f9de3c559eac2e064f92cecf8caedb230da0_2_690x21.png, https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2f1f9de3c559eac2e064f92cecf8caedb230da0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2f1f9de3c559eac2e064f92cecf8caedb230da0.png 2x" data-dominant-color="101311"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">850×27 6.63 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The implementation is following pybind11’s official doc <a href="https://pybind11.readthedocs.io/en/stable/classes.html" rel="noopener nofollow ugc">here</a> and all the cuda kernel funcs are called in the “generateFeatureLabel” func.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba4277b0e904cbd439437126dd5abff85ec5ff04.png" data-download-href="https://discuss.pytorch.org/uploads/default/ba4277b0e904cbd439437126dd5abff85ec5ff04" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba4277b0e904cbd439437126dd5abff85ec5ff04.png" alt="image" data-base62-sha1="qzJdmBvSnSoOUKi5id0yHgGrFsg" width="690" height="245" data-dominant-color="2B2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">729×259 48.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Finnally, The preprocessorcpp.so is imported and initialized in Dataset and  its member func is called in <em>getitem</em><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/59a858a95d3ed3002a54cc61c8f5786c9db5a1dc.png" alt="image" data-base62-sha1="cN9aRzG5Tf8IrUWz2bS7O9Nn6PO" width="491" height="98"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/3/633b28e5f7204e8abfe2cb3903d2591e3fdf6e3e.png" alt="image" data-base62-sha1="e9Q1aag5phnIdykUJmU7VbmsTHo" width="451" height="50"><br>
The DDP is used and it seems all the preprocessor’s cuda calculations is applied on gpu0:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33b22ea6ed0b7fc8bfe2e65607d34ab1b89b667d.png" alt="image" data-base62-sha1="7nk6ulLh9zQHluedNrL15zmuLRr" width="562" height="415"></p>
<p>GPU0’s memory is heavily used and some of the GPUs(e.g., gppu4) is not 100% of the GPU-Util collumn.</p>
<p>The DDP part is basically following the tutorial <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">here</a>. I do use rank to specify GPU ID for model and input data’s GPU, but it seems not working within the dataloader?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a98a4ff920773fc02a2597b341dfb3f50492c70.png" data-download-href="https://discuss.pytorch.org/uploads/default/6a98a4ff920773fc02a2597b341dfb3f50492c70" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a98a4ff920773fc02a2597b341dfb3f50492c70.png" alt="image" data-base62-sha1="fcZEnpW5vBE8AxSXXetNJqwO99K" width="690" height="278" data-dominant-color="252626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">734×296 28.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is it normal or actually can be optimized?<br>
Is there any doc/link that I can check for the mentioned “use the corresponding GPU used in the DDP”?</p>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>No, it’s not normal or expected and I would still guess that your extension is initializing a new CUDA context in each process on the default device.</p>
<p>Each rank will use a corresponding device as explained in the linked tutorial.<br>
(E.g. see the <code>print(f"Running basic DDP example on rank {rank}.")</code> statement which prints the current rank) You should check what your extension is exactly doing and try to avoid initializing the default GPU0.</p>

username: Lin1
cooked: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>You are correct. It seems pytorch’s DDP cant take care of the c++/cuda extensions’ device id.<br>
So I pass the rank as a param to the c++/cuda preprocessor and mannuly set the cuda device ID for each instance of the c++/cuda preprocessor class.</p>
<p>Following Nvidia’s official multi-gpu example: <a href="https://github.com/NVIDIA/cuda-samples" rel="noopener nofollow ugc">here</a>, everything seems fine now.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d5ca3267c519de913dbeb4911fb7ae4088cfac3.png" data-download-href="https://discuss.pytorch.org/uploads/default/5d5ca3267c519de913dbeb4911fb7ae4088cfac3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d5ca3267c519de913dbeb4911fb7ae4088cfac3.png" alt="image" data-base62-sha1="djUSvTlFRT7dzH4VLwVAqxunMaf" width="464" height="500" data-dominant-color="0A0A0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">478×515 43.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>With cuda kenels evenly executed on each GPU, 5~10 times’ faster training speed is gained in my case compared with pure python+numba preprocessor.</p>
<p>Thanks again for your help.</p>