Is it possible to execute the script module in C++ on multiple GPUs in parallel?

rosaline1995: <p>Hi, everyone!</p>
<p>I have a PyTorch model generated by <code>torch.jit.trace().save()</code>. According to the information I found online, it didn’t seem to support torch.nn.parallel when the model is generated and saved.<br>
Then, I converted it to TorchScript, loaded, and finally executed the script module in C++.<br>
The whole program is used to detect objects in a video.</p>
<p>But I am not satisfied enough with its runtime (represented by FPS). Since I have two GPUs, I am thinking if it’s possible to execute on multiple GPUs (for example, load 32 frames from the video and let each GPU handle 16). But still, it’s impossible to load the script module directly to multiple GPUs using <code>torch::jit::load(modelPath.c_str())</code>. In this way, I am curious about if it’s possible to use <code>torch::nn::parallel</code>.</p>
<p>Could anyone please give me a clue on how to use <code>torch::nn::parallel</code> in my case? If it’s not possible, are there any other possible ways of executing the script module in C++ on multiple GPUs in parallel? Can I only turn to multiple-GPU programming with CUDA?</p>
<p>Thanks a lot for any possible help!</p>

albanD: <p>Hi,</p>
<p>Doesn’t <code>nn::parallel::DataParallel</code> work just like the python version (doc <a href="https://pytorch.org/cppdocs/api/function_namespacetorch_1_1nn_1_1parallel_1afcf04d9b87a197c3a734bfb721dd67b0.html?highlight=nn%20parallel#_CPPv4I0EN5torch2nn8parallel13data_parallelE6Tensor10ModuleType6Tensor8optionalINSt6vectorI6DeviceEEE8optionalI6DeviceE7int64_t">here</a>) ?<br>
You can wrap your module into it after loading.</p>

rosaline1995: <p>Hi, thanks a lot for your reply! But according to this post <a href="https://discuss.pytorch.org/t/how-to-trace-jit-in-evaluation-mode-using-multi-gpu-learned-model/33280" class="inline-onebox">How to trace jit in evaluation mode using multi-gpu learned model?</a>, if my model is generated in Python using <code>torch.jit.trace().save()</code>, will it still work to load the model and parallelize it in C++? (Maybe I mixed those concepts…)</p>
<p>Furthermore, I tried so but keep encountering data type errors, may I ask if the following is what you mean?</p>
<pre><code class="lang-auto">torch::jit::Module model = torch::jit::load(modelPath.c_str());
input_tensor = input_tensor.to(at::kCUDA).permute({0, 3, 1, 2}).contiguous();
torch::Tensor outputs = torch::nn::parallel::data_parallel(model, input_tensor);
</code></pre>
<p>But I encountered the error saying that</p>
<pre><code class="lang-auto">libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:267:5: error: base operand of ‘-&gt;’ has non-pointer type ‘torch::jit::Module’
     module-&gt;to(devices-&gt;front());
     ^~~~~~
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:269:12: error: base operand of ‘-&gt;’ has non-pointer type ‘torch::jit::Module’
     return module-&gt;forward(std::move(input)).to(*output_device);
            ^~~~~~
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:279:28: error: no matching function for call to ‘replicate(torch::jit::Module&amp;, std::vector&lt;c10::Device&gt;&amp;)’
   auto replicas = replicate(module, *devices);
                   ~~~~~~~~~^~~~~~~~~~~~~~~~~~
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:142:42: note: candidate: template&lt;class ModuleType&gt; std::vector&lt;std::shared_ptr&lt;_Tp&gt; &gt; torch::nn::parallel::replicate(const std::shared_ptr&lt;_Tp&gt;&amp;, const std::vector&lt;c10::Device&gt;&amp;)
 std::vector&lt;std::shared_ptr&lt;ModuleType&gt;&gt; replicate(
                                          ^~~~~~~~~
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:142:42: note:   template argument deduction/substitution failed:
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:279:28: note:   ‘torch::jit::Module’ is not derived from ‘const std::shared_ptr&lt;_Tp&gt;’
   auto replicas = replicate(module, *devices);
                   ~~~~~~~~~^~~~~~~~~~~~~~~~~~
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:161:39: note: candidate: template&lt;class ModuleType&gt; std::vector&lt;torch::nn::ModuleHolder&lt;ModuleType&gt; &gt; torch::nn::parallel::replicate(const torch::nn::ModuleHolder&lt;ModuleType&gt;&amp;, const std::vector&lt;c10::Device&gt;&amp;)
 std::vector&lt;ModuleHolder&lt;ModuleType&gt;&gt; replicate(
                                       ^~~~~~~~~
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:161:39: note:   template argument deduction/substitution failed:
libtorch/include/torch/csrc/api/include/torch/nn/parallel/data_parallel.h:279:28: note:   ‘torch::jit::Module’ is not derived from ‘const torch::nn::ModuleHolder&lt;ModuleType&gt;’
   auto replicas = replicate(module, *devices);
                   ~~~~~~~~~^~~~~~~~~~~~~~~~~~
</code></pre>
<p>I tried to use <code>_net = std::make_shared&lt;torch::jit::Module&gt;(model);</code> or <code>torch::Tensor outputs = torch::nn::parallel::data_parallel(&amp;model, input_tensor);</code> but none of them work.</p>
<p>Do I misunderstand what you said?</p>

albanD: <p>Hi,</p>
<blockquote>
<p>if my model is generated in Python using <code>torch.jit.trace().save()</code>, will it still work to load the model and parallelize it in C++</p>
</blockquote>
<p>The difference with this other post is that you don’t save the DataParallel mode, you want to apply it afterward.</p>
<blockquote>
<p>Do I misunderstand what you said?</p>
</blockquote>
<p>No that was my suggestion. But it looks like the <code>jit::Module</code> is not working fine to replace a <code>nn::Module</code> here <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"><br>
Could you open an issue on github about that please?</p>

rosaline1995: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="101222">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>You can wrap your module into it after loading.</p>
</blockquote>
</aside>
<p>Sure, thanks a lot for your quick response!</p>