Segmentation fault in the following code (using C++ API for prediction)

username: reddy
cooked: <p>Using CUDA 10 on linux</p>
<p>Segv occurs at line ‘int idx = top_idxs_a[i];’ in the code below</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include "utils.hpp"
#include &lt;opencv2/opencv.hpp&gt;

int main(int argc, char** argv)
{
    if (argc != 4) {
    std::cerr &lt;&lt; "usage: test_PyTorch_Cpp_Prediction_GPU &lt;path-to-image&gt; &lt;path-to-exported-script-module&gt; &lt;path-to-labels-file&gt; \n";
    return -1;
  } 

  std::string image_path = argv[1];
  std::string model_path = argv[2];
  std::string labels_path = argv[3];

  int image_height = 224;
  int image_width = 224;
  int n_channels = 3;

  // Read labels
  std::vector&lt;std::string&gt; labels;
  std::string label;
  std::ifstream labelsfile (labels_path);
  if (labelsfile.is_open())
  {
    while (getline(labelsfile, label))
    {
      labels.push_back(label);
    }
    labelsfile.close();
  }

  std::vector&lt;double&gt; mean = {0.485, 0.456, 0.406};
  std::vector&lt;double&gt; std = {0.229, 0.224, 0.225};

  cv::Mat image = cv::imread(image_path, 1);

  // Deserialize the ScriptModule from a file using torch::jit::load().
  std::shared_ptr&lt;torch::jit::script::Module&gt; module = torch::jit::load(model_path);
  assert(module != nullptr);

  module-&gt;to(torch::kCUDA);

  // Preprocess image
  image = preprocess(image, image_height, image_width, mean, std);

  std::vector&lt;int64_t&gt; dims = {1, image_height, image_width, n_channels};
  std::vector&lt;int64_t&gt; permute_dims = {0, 3, 1, 2};
  torch::TensorOptions options(torch::kFloat32);
  torch::Tensor image_as_tensor = torch::from_blob(image.data, torch::IntList(dims), options).clone();
  image_as_tensor = image_as_tensor.permute(torch::IntList(permute_dims));
  image_as_tensor = image_as_tensor.toType(torch::kFloat32);
  
  // image_as_tensor = image_as_tensor.to(torch::kCUDA);
  std::vector&lt;torch::jit::IValue&gt; inputs;
  inputs.push_back((image_as_tensor).cuda());
  torch::Tensor output = module-&gt;forward(inputs).toTensor();
  // torch::Tensor output = module-&gt;forward({image_as_tensor}).toTensor();
  output = output.to(torch::kCPU, false);
  std::cout &lt;&lt; output.sizes() &lt;&lt; "\n";

  // out tensor sort, print the first 2 category
  std::tuple&lt;torch::Tensor,torch::Tensor&gt; result = output.sort(-1, true);
  torch::Tensor top_scores = std::get&lt;0&gt;(result)[0];
  torch::Tensor top_idxs = std::get&lt;1&gt;(result)[0].toType(torch::kInt32);

  auto top_scores_a = top_scores.accessor&lt;float,1&gt;();
  auto top_idxs_a = top_idxs.accessor&lt;int,1&gt;();
  // std::cout &lt;&lt; top_scores_a &lt;&lt; "\n";
  // std::cout &lt;&lt; top_idxs_a &lt;&lt; "\n";
  std::cout &lt;&lt; labels.size() &lt;&lt; "\n";
  for (int i = 0; i &lt; 5; ++i) {
    int idx = top_idxs_a[i];
    std::cout &lt;&lt;idx &lt;&lt; labels[idx] &lt;&lt;"\n";
    std::cout &lt;&lt; "top " &lt;&lt; i+1 &lt;&lt; ", label: "&lt;&lt; labels[idx] &lt;&lt; " ";
    std::cout &lt;&lt; ", score: " &lt;&lt; top_scores_a[i] &lt;&lt; std::endl;
    }
}
</code></pre>

username: LeviViana
cooked: <p>I think it happens because you are trying to access a value of a vector stored on the device from the host. Run the same operation inside a kernel or pass the <code>top_idxs</code> to the <code>cpu</code> and tell me what happens please.</p>

username: reddy
cooked: <p>Thanks <a class="mention" href="/u/leviviana">@LeviViana</a>.  That does fix the segv.  But, the output classification result seem way off</p>
<p>$ ./test_PyTorch_Cpp_Prediction_GPU example.JPEG ./…/…/model.pt labels.txt<br>
[1, 1000]<br>
1000<br>
238Greater_Swiss_Mountain_dog<br>
, score: 1.54363ater_Swiss_Mountain_dog<br>
777scabbard<br>
, score: 1.54137bbard<br>
859toaster<br>
, score: 1.47798ster<br>
454bookshop<br>
, score: 1.42607kshop<br>
219cocker_spaniel<br>
, score: 1.4023cker_spaniel</p>
<p>This is the input image:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/e/ef9ea8af3438315aaf94a05ab3c6820d904251f2.jpeg" alt="example" data-base62-sha1="ybM2w7dcMP4i5idtJos8i0n60WC" width="224" height="224"></p>

username: LeviViana
cooked: <p>Are you sure that you are loading a pretrained model ? And that you are properly loading its weights ?</p>
<p>Another side question ? Why are you using the C++ API ? If you want to do inference with deep CNN models, the python overhead may not be significant, since the bottleneck will probably be the convolutions, which are done by cuDNN in both API’s (assuming that you use a GPU for inference).</p>

username: reddy
cooked: <ol>
<li>
<p>I am using pretrained models from torchvision.models.  So, I believe it should do the right thing</p>
</li>
<li>
<p>I have been using PyTorch python API for the past few months.  Two reasons why I am exploring C++ API:<br>
(a) I was under the impression it will be faster.  But, based on your comments it shouldn’t be that different compared to python API<br>
(b) For deployment in production</p>
</li>
</ol>
<p>In-fact, from my experiments I see C++ API is slower than python API for some models I tried:</p>
<h3>C++ API, torch JIT</h3>
<p>./test_PyTorch_JIT_Cmp<br>
Network: alexnet, Batch-size: 1, Images/Sec: 633.132<br>
Network: vgg-16, Batch-size: 1, Images/Sec: 208.806<br>
Network: resnet-50, Batch-size: 1, Images/Sec: 128.854</p>
<h3>Python</h3>
<p>$ python test_PyTorch_Cmp.py<br>
Network: alexnet, Batch-size: 1, Images/Sec: 717.5638130431807<br>
Network: vgg-16, Batch-size: 1, Images/Sec: 236.11391642601006<br>
Network: resnet-50, Batch-size: 1, Images/Sec: 127.30998980136953</p>
<p>For reference (partial code below)</p>
<h3>Python</h3>
<pre><code class="lang-auto">dry_run = 5 # use 5 iterations to warm up
    num_batches = 10

    for i in range(dry_run+num_batches):
        if i == dry_run:
            tic = time.time()
        batch_TensorTemp = torch.autograd.Variable(batch_Tensor)
        if dev == "gpu":
            # move tensor to GPU
            batch_TensorTemp = batch_TensorTemp.cuda()
        output_batch = model(batch_TensorTemp)
        if dev == "gpu":
            # move ouput to cpu
            output_batch = output_batch.data.cpu()
    end = time.time()

    print("Network: {}, Batch-size: {}, Images/Sec: {}\n".format(network, batch_size, (num_batches*batch_size/(end - tic))))
</code></pre>
<h3>C++</h3>
<pre><code class="lang-auto">int dry_run = 5;
        int num_batches = 10;

        for(int iter = 0; iter &lt; dry_run + num_batches; ++iter){
          if(iter == dry_run){
            cudaEventRecord(start);
          }
          std::vector&lt;torch::jit::IValue&gt; inputs;
          inputs.push_back((image_as_tensor).cuda());
          torch::Tensor output = module-&gt;forward(inputs).toTensor();
          output = output.to(torch::kCPU);    
        }
        cudaEventRecord(stop);
    
        cudaEventSynchronize(stop);
        cudaEventElapsedTime(&amp;milliseconds, start, stop);
        seconds = milliseconds/1000;
        float throughPut = num_batches*batchSize/seconds;
        std::cout &lt;&lt; "Network: " &lt;&lt; networkName &lt;&lt; ", Batch-size: " &lt;&lt; batchSize &lt;&lt; ", Images/Sec: " &lt;&lt; throughPut &lt;&lt; "\n";
</code></pre>

username: dambo
cooked: <aside class="quote no-group quote-modified" data-username="reddy" data-post="1" data-topic="39539">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/reddy/48/9845_2.png" class="avatar"> reddy:</div>
<blockquote>
<p>output = output.to(torch::kCPU, false); std::cout &lt;&lt; output.sizes() &lt;&lt; “\n”;</p>
</blockquote>
</aside>
<p>Did you try using batches when comparing the results between C++ and Python?</p>