faster-RCNN to libtorch issue

username: Nisan_Aryal
cooked: <p>I have finetuned a faster-RCNN model and am trying to shift the code to c++ using the jit.script.</p>
<p>I tried with jit.trace as instructed in the pytorch website. However, there was error. Upon searching in the internet I found the torch vision models are scriptable not traceable (i dont  know the difference).</p>
<p>thus I have created a .pt file using the torch.jit.script(model)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/4468b63e8ed5489a45294d7fefc3b72a50911a6b.png" alt="image" data-base62-sha1="9LaOm2kDDYu3BaYINVwCwakGQ7V" width="385" height="56"></p>
<p>I setup the c++ environment with libtorch, opencv etc and am trying to run the code. However, i am running with two problem.</p>
<p>problem 1: I trained the model in python with shape (720,1280) and when i run the code on the c++. i received this error…Note: the code runs fine in the python version with the same size<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec537f314895ab9c80877e26162f777963bb557c.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec537f314895ab9c80877e26162f777963bb557c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec537f314895ab9c80877e26162f777963bb557c_2_690x462.png" alt="image" data-base62-sha1="xIDzBcgj925viZ5wXqscOeAxWDG" width="690" height="462" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec537f314895ab9c80877e26162f777963bb557c_2_690x462.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec537f314895ab9c80877e26162f777963bb557c_2_1035x693.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec537f314895ab9c80877e26162f777963bb557c.png 2x" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1053×706 73.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Problem 2:<br>
I then changed the shape of the image (resize) to (275,256) by looking at this example code</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/vision/blob/main/test/tracing/frcnn/test_frcnn_tracing.cpp">
  <header class="source">

      <a href="https://github.com/pytorch/vision/blob/main/test/tracing/frcnn/test_frcnn_tracing.cpp" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/main/test/tracing/frcnn/test_frcnn_tracing.cpp" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/main/test/tracing/frcnn/test_frcnn_tracing.cpp</a></h4>


      <pre><code class="lang-cpp">#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#include &lt;torchvision/vision.h&gt;
#include &lt;torchvision/ops/nms.h&gt;


int main() {
  torch::DeviceType device_type;
  device_type = torch::kCPU;

  torch::jit::script::Module module;
  try {
    std::cout &lt;&lt; "Loading model\n";
    // Deserialize the ScriptModule from a file using torch::jit::load().
    module = torch::jit::load("fasterrcnn_resnet50_fpn.pt");
    std::cout &lt;&lt; "Model loaded\n";
  } catch (const torch::Error&amp; e) {
    std::cout &lt;&lt; "error loading the model\n";
    return -1;
  } catch (const std::exception&amp; e) {
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/vision/blob/main/test/tracing/frcnn/test_frcnn_tracing.cpp" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4ebd18427324bc560894eb4971868e787fe7fb95.png" data-download-href="https://discuss.pytorch.org/uploads/default/4ebd18427324bc560894eb4971868e787fe7fb95" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4ebd18427324bc560894eb4971868e787fe7fb95.png" alt="image" data-base62-sha1="beynkIIXFF7g62WcTxSZ82e0IIt" width="398" height="499" data-dominant-color="FAFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">555×696 13.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Then<br>
I am receiving this output<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8bfdbe731cd132a17c1c96e3546923c5ac0f02e.png" data-download-href="https://discuss.pytorch.org/uploads/default/c8bfdbe731cd132a17c1c96e3546923c5ac0f02e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8bfdbe731cd132a17c1c96e3546923c5ac0f02e_2_690x69.png" alt="image" data-base62-sha1="sDUy6yOlNrp9LXoPSfrV6OnjQUS" width="690" height="69" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8bfdbe731cd132a17c1c96e3546923c5ac0f02e_2_690x69.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8bfdbe731cd132a17c1c96e3546923c5ac0f02e_2_1035x103.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8bfdbe731cd132a17c1c96e3546923c5ac0f02e.png 2x" data-dominant-color="353535"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1120×113 22.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The prediction is null<br>
(to be sure that the open cv is extracting frames. I saved the frames as video in c++. The output videos were fine.)<br>
Since the predicyion is null, I am doubting that my scripting method might have been wrong in the python.</p>
<p>The c++ code is given below.(writing a code in c++ after a very very very long time so there might be a bug causing the error. Please review it.)</p>
<pre><code class="lang-auto">int main(){

  // Create a VideoCapture object and use camera to capture the video
  VideoCapture cap("/workspace/cpp/input.mp4");

  torch::jit::script::Module module;
  module = torch::jit::load("/workspace/cpp/script_blurr_model.pt");
  // Default resolutions of the frame are obtained.The default resolutions are system dependent.
  int frame_width = cap.get(cv::CAP_PROP_FRAME_WIDTH);
  int frame_height = cap.get(cv::CAP_PROP_FRAME_HEIGHT);

  // Define the codec and create VideoWriter object.The output is stored in 'outcpp.avi' file.
//  VideoWriter video("outcpp.avi", cv::VideoWriter::fourcc('M','J','P','G'), 30, Size(frame_width,frame_height));
  VideoWriter video("outcpp.avi", cv::VideoWriter::fourcc('M','J','P','G'), 30, Size(275,256));

  while(1){

    Mat frame;

    // Capture frame-by-frame
    torch::TensorOptions options = torch::TensorOptions{torch::kCUDA};
    torch::TensorOptions options1 = torch::TensorOptions{torch::kCPU};
    cap &gt;&gt; frame;
    // If the frame is empty, break immediately
    if (frame.empty())
      break;
//    cv::cvtColor(frame, frame, CV_BGR2RGB);
    resize(frame, frame, Size(275, 256), CV_INTER_CUBIC);
    auto tensor_image = torch::from_blob(frame.data, { frame.rows, frame.cols, frame.channels() }, at::kByte);
    cout&lt;&lt;tensor_image.sizes();
    tensor_image = tensor_image.permute({ 2,0,1 });
    module.eval();
    module.to(torch::kCUDA);
    std::vector&lt;torch::jit::IValue&gt; inputs;
    std::vector&lt;torch::Tensor&gt; images;

    images.push_back(tensor_image.to(torch::Device(torch::kCUDA)));
//    images.push_back(torch::rand({3, 256, 275}, options));
//    images.push_back(torch::rand({3, 256, 275}, options));
//    images.push_back(tensor_image.to(torch::Device(torch::kCUDA)));
    inputs.push_back(images);
//    cout&lt;&lt;inputs[0].shape();
    auto output = module.forward(inputs);
//    at::Tensor output = module.forward(tensor_image).toTensor();
    auto detections = output.toTuple()-&gt;elements().at(1).toList().get(0).toGenericDict();

    auto boxes=  detections.at("boxes");
    cout&lt;&lt;  detections&lt;&lt;endl;


//    break;
//    cout&lt;&lt;tensor_image.sizes();


    // Write the frame into the file 'outcpp.avi'
    video.write(frame);

    // Display the resulting frame
    //imshow( "Frame", frame );

    // Press  ESC on keyboard to  exit

  }

  // When everything done, release the video capture and write object
  cap.release();
  video.release();

  // Closes all the frames
  destroyAllWindows();
  return 0;
}
</code></pre>
<p>I tested the size (256,275) in the python. It worked fine with the prediction.</p>
<p>the rough code used for the python is below (model and scripting part):</p>
<pre><code class="lang-auto">def get_model_instance_segmentation(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    return model

model = get_model_instance_segmentation(4)

model.load_state_dict(torch.load('model_face_organ_screen_test_4.pt'))
model.cuda()
model.eval()

traced_script_module = torch.jit.script(model)
traced_script_module.save("trace_test.pt")
</code></pre>
<p>This is my first attempt of using libtorch and jit.script. Please review it and give suggestions to run the model successfully.</p>