Model running in CPU instead of GPU when module->to(torch::kCUDA) with C++ API

username: OleguerCanal
cooked: <p>Hi, I am trying to run a pytorch trained model following the instructions here: <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_export.html</a></p>
<p>I want to run the model in the GPU if I run:</p>
<pre><code class="lang-auto">    torch::DeviceType device_type;
    if (torch::cuda::is_available()) {
        std::cout &lt;&lt; "Cuda available, running on GPU" &lt;&lt; std::endl;
        device_type = torch::kCUDA;
    } else {
        std::cout &lt;&lt; "Cuda NOT available, running on CPU" &lt;&lt; std::endl;
        device_type = torch::kCPU;
    }
    torch::Device device(device_type);
    module-&gt;to(torch::Device(device));
</code></pre>
<p>I get the message: Cuda available, running on GPU. Nevertheless, when I execute:<br>
<code>at::Tensor output = module-&gt;forward(inputs).toTensor();</code><br>
The command consumes way to much time and I can see at the system monitor how all cores are being used and how no new GPU process is opened in the nvidia-smi tool.<br>
I guess I am doing something wrong but I can’t find what. Any ideas?</p>
<p>I am using Cuda 9.0 and built the project from libcudnn7_7.4.2.24-1<br>
I imported:</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; 
#include &lt;torch/cuda.h&gt;
</code></pre>
<p>Thanks a lot and sorry for the inconvenience.</p>

username: yf225
cooked: <p>This seems weird. Can you try and see if <code>module-&gt;to(torch::Device(torch::kCUDA, 0))</code> works?</p>

username: OleguerCanal
cooked: <p>Sorry, I discovered that before tracing the model in python I didn’t move it to cuda. If I move it, trace it, and reload it in c++, it correctly runs on the GPU.</p>
<p>Still I can’t copy a cv::Mat into a torch::Tensor and move it to the GPU</p>
<pre><code class="lang-auto">at::Tensor output = torch::from_blob(img.data, {1, 3, img.rows, img.cols}, torch::kFloat32).clone();
output.to(torch::Device(torch::kCUDA, 0));
output.to(torch::kCUDA); //trying both ways just in case
assert(output.device().type() == torch::kCUDA); // Assertation failed
</code></pre>
<p>After, I tried to copy the data manually like so:</p>
<pre><code class="lang-auto">auto options = torch::TensorOptions().dtype(torch::kFloat32).device(torch::kCUDA, 0);
at::Tensor output = torch::rand({1, 3, img.rows, img.cols}, options); // maybe cols rows
auto foo_a = output.accessor&lt;float, 4&gt;(); // define accessor    
for(int i = 0; i &lt; img.rows; i++){
    for(int j = 0; j &lt; img.cols; j++){
        foo_a[0][0][i][j] = img.at&lt;cv::Vec3f&gt;(i, j)[0]; //Segmentation Fault here when accessing foo_a
        foo_a[0][1][i][j] = img.at&lt;cv::Vec3f&gt;(i, j)[1];
        foo_a[0][2][i][j] = img.at&lt;cv::Vec3f&gt;(i, j)[2];
    }
}
assert(output.device().type() == torch::kCUDA);
</code></pre>
<p>but I get a segmentation fault (line with the comment).</p>
<p>Any ideas on what could be causing these errors?</p>

username: yf225
cooked: <p>First I’d recommend using <code>torch::Tensor</code> instead of <code>at::Tensor</code>, as <code>at::Tensor</code> is now an implementation detail and <code>torch::Tensor</code> is the user-facing tensor class.</p>
<p>For the first example, we can try <code>output = output.to(torch::Device(torch::kCUDA, 0));</code> to see if it works.</p>
<p>For the second example, this will work:</p>
<pre><code class="lang-auto">auto options = torch::TensorOptions().dtype(torch::kFloat32).device(torch::kCUDA, 0);
auto output = torch::rand({1, 3, img.rows, img.cols}, options); // maybe cols rows
for(int i = 0; i &lt; img.rows; i++){
    for(int j = 0; j &lt; img.cols; j++){
        output[0][0][i][j] = img.at&lt;cv::Vec3f&gt;(i, j)[0];
        output[0][1][i][j] = img.at&lt;cv::Vec3f&gt;(i, j)[1];
        output[0][2][i][j] = img.at&lt;cv::Vec3f&gt;(i, j)[2];
    }
}
assert(output.device().type() == torch::kCUDA);
</code></pre>

username: OleguerCanal
cooked: <p>Thank you a lot Will Feng! Both solutions work!!</p>
<p>Is it normal that running the forward method on the GPU consumes more time than using the CPU? I wrote a simplified version of my code and this still occurs:</p>
<pre><code class="lang-auto">#include &lt;chrono&gt;
#include &lt;ctime&gt;
#include &lt;iostream&gt;

#include &lt;torch/cuda.h&gt;
#include &lt;torch/script.h&gt;

bool use_cuda = true;

class Timer{
  private:
    std::chrono::system_clock::time_point t_start;

  public:
    void start(){
        t_start = std::chrono::system_clock::now();
    }

    void stop(std::string msg){
        std::chrono::system_clock::time_point end = std::chrono::system_clock::now();
        std::chrono::duration&lt;double&gt; elapsed_seconds = end - t_start;
        std::cout &lt;&lt; msg &lt;&lt; ": " &lt;&lt; elapsed_seconds.count() &lt;&lt; "s\n";
    }
};

int main(int argc, const char* argv[]) {
    if (argc != 2) {
        std::cerr &lt;&lt; "usage: example-app &lt;path-to-exported-script-module&gt;\n";
        return -1;
    }

    // Deserialize model
    std::shared_ptr&lt;torch::jit::script::Module&gt; module = torch::jit::load(argv[1]);
    assert(module != nullptr);

    // Set CPU/GPU
    torch::DeviceType device_type = torch::kCPU;
    if(use_cuda) device_type = torch::kCUDA;
    module-&gt;to(device_type);

    // Create input
    auto options = torch::TensorOptions().dtype(torch::kFloat32).device(device_type);
    auto inp = torch::zeros({2, 3, 720, 1280}, options);
    std::vector&lt;torch::jit::IValue&gt; inputs;
    inputs.push_back(inp);

    // Run model
    Timer t;
    t.start();
    torch::Tensor output = module-&gt;forward(inputs).toTensor(); 
    t.stop("forward");
}
</code></pre>
<p>Time in CPU: ~3s<br>
Time in GPU: ~7s!!!</p>
<p>I’m surely missing something, I am using a Nvidia GTX 1080 and the model can be evaluated under 0.1s in Python.</p>
<p>Again, thank you a lot for your help!!</p>

username: yf225
cooked: <p>No problem. For the GPU slowness, is the JIT model originally trained with GPU?</p>

username: OleguerCanal
cooked: <p>Yes.<br>
I use a pre-trained ResNet101 with custom layers at the end that are trained in GPU.</p>

username: yf225
cooked: <p>Do you mind sharing your JIT model so I can reproduce?</p>

username: OleguerCanal
cooked: <p>Hi Will, sorry for the late reply. Turns out that the first time I run forward it takes ~7s, but if I run it in a loop all the others take ~0.09s (still not as fast as running it from Python which is ~0.02s, but way more decent)</p>
<p>Any idea on what may be causing this? I guess I could just do a warm-up at the beginning and run it once but I am curious.</p>

username: FantasyJXF
cooked: <p>warmup is necessary.</p>
<p>I try to move tensor from cpu to gpu, the first time costs a lot time, but the next times it went fast as I wish.</p>

username: 111257
cooked: <p>Hello,Could you please tell me how to warm up?Thank you.</p>

username: 111257
cooked: <p>Hello,I don’t know how to warm up,could you please tell me ? Thank you.</p>

username: FantasyJXF
cooked: <p>Just inference once after you load the model</p>

username: cvchanghao
cooked: <p>Hi. I have a question. How to move the data  to CPU?</p>
<pre><code class="lang-auto">module.forward(inputs).to(torch::kCPU)
</code></pre>
<p>I tried this but failed. Do you have any ideas?</p>