How to use multi gpu inference in libtorch?

username: lcxywfe
cooked: <p>I want to use libtorch for multi gpu inference, is there any example or tutorial?<br>
Should I clone multi <code>jit::script::Module</code> and move them to different gpu?</p>

username: ptrblck
cooked: <p>If I’m not mistaken <code>torch::nn::parallel::data_parallel</code> would be the equivalent to <code>nn.DataParallel</code> in the Python frontend and in case you would like to use <code>DistributedDataParallel</code> feel free to add your use case in <a href="https://github.com/pytorch/pytorch/issues/48959">this poll</a>.</p>

username: lcxywfe
cooked: <p>I want to use multi gpu manually, because the input data size is different.</p>
<pre><code class="lang-auto">for (int i = 0; i &lt; param_.gpu_size(); ++i) {
    torch::Device device(torch::kCUDA, i);
    models.push_back(torch::jit::load(MODEL_PATH, device));
    models.back()-&gt;eval();
}
</code></pre>
<p>And I will create <code>gpu_size</code> threads to run the inference.<br>
Is it the correct way?</p>

username: lcxywfe
cooked: <p>I used 2 threads to run the model on 2 gpus, like:</p>
<pre><code class="lang-auto">      std::vector&lt;std::thread*&gt; threads(0);
      int parallel = std::min(param_.gpu_size(),
                              static_cast&lt;int&gt;(batched_samples.size()));
      batches.resize(parallel);
      for (int i = 0; i &lt; parallel; ++i) {
          batches[i].resize(0);
          for (int j = i; j &lt; betched_samples.size(); j += parallel) {
              batches[i].push_back(std::move(batched_samples[j]));
          }
          LOG(ERROR) &lt;&lt; "==== predict level-" &lt;&lt; l &lt;&lt; " batch-"
                     &lt;&lt; batches[i].size();

          threads.push_back(new std::thread(
                  &amp;predict_torch, models_[i],
                  std::ref(batches[i]), param_.gpu(i));
      }
      for (int i = 0; i &lt; threads.size(); ++i) {
          threads[i]-&gt;join();
          delete threads[i];
      }
</code></pre>
<p>But I got the:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'c10::Error'
  what():  r INTERNAL ASSERT FAILED at "../aten/src/ATen/core/jit_type_base.h":172, please report a bug to PyTorch. 
Exception raised from expect at ../aten/src/ATen/core/jit_type_base.h:172 (most recent call first):
frame #0: &lt;unknown function&gt; + 0x101f9b (0x7fa6e14e6f9b in /root/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: std::function&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; ()&gt;::operator()() const + 0x4c (0x7fa6e14e7e20 in /root/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #2: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0x40 (0x7fa6e14e6030 in /root/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3: std::shared_ptr&lt;c10::ClassType&gt; c10::Type::expect&lt;c10::ClassType&gt;() + 0xbb (0x7fa6c2ae3b1b in /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #4: c10::ivalue::Object::type() const + 0x41 (0x7fa6c2ad10f1 in /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #5: &lt;unknown function&gt; + 0x5ec04cf (0x7fa6c69c14cf in /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #6: torch::jit::Object::find_method(std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;) const + 0x37 (0x7fa6c69d48c1 in /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.s
o)
frame #7: torch::jit::Object::get_method(std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;) const + 0x50 (0x7fa6d7179414 in /root/ecopia-weaver-multi-scale-framework/src/build/libecopia_weaver.
so)
frame #8: torch::jit::Module::forward(std::vector&lt;c10::IValue, std::allocator&lt;c10::IValue&gt; &gt;) + 0xba (0x7fa6d717979a in /root/ecopia-weaver-multi-scale-framework/src/build/libecopia_weaver.so)
frame #9: ecopia::ml::CaffeForwardMultiScale::predict_torch(torch::jit::Module*, std::vector&lt;std::vector&lt;ecopia::ml::Sample*, std::allocator&lt;ecopia::ml::Sample*&gt; &gt;, std::allocator&lt;std::vector&lt;ecopia::ml::Sample*, std::allocator&lt;ec
opia::ml::Sample*&gt; &gt; &gt; &gt; const&amp;, int, int, int, int, ecopia::ml::LevelInfo const&amp;, std::unordered_map&lt;int, MultiChannelRasterData&lt;float&gt;*, std::hash&lt;int&gt;, std::equal_to&lt;int&gt;, std::allocator&lt;std::pair&lt;int const, MultiChannelRasterD
ata&lt;float&gt;*&gt; &gt; &gt;&amp;, std::unordered_map&lt;int, MultiChannelLabelMap&lt;unsigned char&gt;*, std::hash&lt;int&gt;, std::equal_to&lt;int&gt;, std::allocator&lt;std::pair&lt;int const, MultiChannelLabelMap&lt;unsigned char&gt;*&gt; &gt; &gt;&amp;, std::unordered_map&lt;int, SingleCha
nnelLabelMap&lt;unsigned char&gt;*, std::hash&lt;int&gt;, std::equal_to&lt;int&gt;, std::allocator&lt;std::pair&lt;int const, SingleChannelLabelMap&lt;unsigned char&gt;*&gt; &gt; &gt;&amp;) + 0x139c (0x7fa6d7172224 in /root/ecopia-weaver-multi-scale-framework/src/build/lib
ecopia_weaver.so)
</code></pre>
<p>Anyone have ideas?</p>

username: lcxywfe
cooked: <p>If I run the model on one of the two gpus, both can work.</p>