Model.forward() with same input size as in pytorch leads to dimension error in libtorch

birds_are_drones: <p>hi !<br>
I have this code that is a libtorch cpp inference class for models trained on various platforms. I load my .pth model and try to use it to predict output data but i quickly encounter an error :</p>
<pre><code class="lang-auto">tensor sent to cuda
tensor sent to half
leaving toTensor
input_.at(0).size() = [1, 3, 384, 288]
entring inference
terminate called after throwing an instance of 'std::runtime_error'
  what():  The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/models/yolo.py", line 46, in forward
    _35 = (_4).forward(_34, )
    _36 = (_2).forward((_3).forward(_35, ), _29, )
    _37 = (_0).forward(_33, _35, (_1).forward(_36, ), )
           ~~~~~~~~~~~ &lt;--- HERE
    _38, _39, _40, _41, = _37
    return (_41, [_38, _39, _40])
  File "code/__torch__/models/yolo.py", line 75, in forward
    y = torch.sigmoid(_50)
    _51 = torch.mul(torch.slice(y, 4, 0, 2), CONSTANTS.c0)
    _52 = torch.add(torch.sub(_51, CONSTANTS.c1), CONSTANTS.c2)
          ~~~~~~~~~ &lt;--- HERE
    xy = torch.mul(_52, torch.select(CONSTANTS.c3, 0, 0))
    _53 = torch.mul(torch.slice(y, 4, 2, 4), CONSTANTS.c4)

Traceback of TorchScript, original code (most recent call last):
/home/kubler/data/det_track/yolov5_bis/models/yolo.py(66): forward
/home/kubler/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py(1039): _slow_forward
/home/kubler/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py(1051): _call_impl
/home/kubler/data/det_track/yolov5_bis/models/yolo.py(143): _forward_once
/home/kubler/data/det_track/yolov5_bis/models/yolo.py(121): forward
/home/kubler/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py(1039): _slow_forward
/home/kubler/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py(1051): _call_impl
/home/kubler/anaconda3/lib/python3.8/site-packages/torch/jit/_trace.py(952): trace_module
/home/kubler/anaconda3/lib/python3.8/site-packages/torch/jit/_trace.py(735): trace
export.py(54): export_torchscript
export.py(279): run
/home/kubler/anaconda3/lib/python3.8/site-packages/torch/autograd/grad_mode.py(28): decorate_context
export.py(328): main
export.py(333): &lt;module&gt;
RuntimeError: The size of tensor a (36) must match the size of tensor b (80) at non-singleton dimension 3
</code></pre>
<p>The input tensor size of the original repo is <code>img.size = torch.Size([1, 3, 384, 288])</code><br>
I format my tensor to be of the same type and size .</p>
<p>I already had encountered this error before with YOLO and this was because of input dimensions that didn’t match and I needed to make the input square. At the time, I solved it by padding the image to make it square but here it didn’t work.</p>
<p>Here is my code  :</p>
<pre><code class="lang-auto">//______________________RUN_________________________
void PoseDetector::run(bool display = false){
  if (camera_.isOpened())
    dataset_ = getCameraFrame();
  else if (! ( getDataPath().empty() ) )
    dataset_ = loadDataset();

  std::cout&lt;&lt; " dataset_.at(0).size() = " &lt;&lt; dataset_.at(0).size() &lt;&lt;std::endl;
  input_ = prepocData( dataset_ ,  inputSize_ );
  std::cout&lt;&lt; "input_.at(0).size() = " &lt;&lt; input_.at(0).toTensor().sizes() &lt;&lt;std::endl;
  detections_tens_ = inference(input_);
/*
  detections_ = postProcessing(detections_tens_);
  if ( display ){
    demo(  dataset_.at(0) , detections_ , labels_ );
  }
*/
}


//______________________PREPROC_DATA___________________

std::vector&lt;torch::jit::IValue&gt; PoseDetector::prepocData(std::vector&lt;cv::Mat&gt;&amp; matrix_data , cv::Size dst_sz){
  //std::cout &lt;&lt; "entering preproc data"&lt;&lt;std::endl;

  cv::Mat src , normalized , resized , padded  ;
  std::vector&lt;torch::jit::IValue&gt; res ;

  for( auto it = std::begin(matrix_data) ; it != std::end(matrix_data) ; it ++  ){

    //________detect_preprocess
    (*it).convertTo( normalized,  CV_32FC3, 1.0 / 255, 0); //normalization and type matching tensor input type as in  yolov5 ultralytics detect.py
    //std::cout &lt;&lt; "normalized shape =  " &lt;&lt; normalized.cols &lt;&lt; " , " &lt;&lt; normalized.rows &lt;&lt; " , " &lt;&lt; normalized.channels() &lt;&lt; std::endl ;

    //___________resizing&amp;padding to square the image
    //resize bigger dim to match yolo training input size.
    scale_ = resize( normalized , resized , dst_sz);
    //std::cout &lt;&lt; "resized shape =  " &lt;&lt; resized.cols &lt;&lt; " , " &lt;&lt; resized.rows &lt;&lt; " , " &lt;&lt; resized.channels() &lt;&lt; std::endl ;

    padToSize(resized , padded , dst_sz);
    //std::cout &lt;&lt; "after pad : padded shape =  " &lt;&lt; padded.cols &lt;&lt; " , " &lt;&lt; padded.rows &lt;&lt; " , " &lt;&lt; padded.channels() &lt;&lt; std::endl ;

    torch::Tensor tens ;
    toTensor(padded , tens);
    //std::cout &lt;&lt; "tens.sizes() = " &lt;&lt; tens.sizes() &lt;&lt; std::endl ;
    //std::cout &lt;&lt; "tens.device() = " &lt;&lt; tens.device() &lt;&lt; std::endl ;

    res.emplace_back(tens);
  }
  //std::cout &lt;&lt; "leaving preproc data"&lt;&lt;std::endl;

  return res;
}

//______________________INFERENCE_______________________
torch::Tensor PoseDetector::inference(std::vector&lt;torch::jit::IValue&gt;&amp; input) {
  //runs model inference on loaded model
  std::cout &lt;&lt; "entring inference" &lt;&lt; std::endl;
  //for ( auto it =  std::begin(input) ; it != std::end(input) ; it ++){
  //  std::cout &lt;&lt; "input it device = " &lt;&lt; ((torch::Tensor) (*it)).device() &lt;&lt; std::endl;
  //}
  torch::jit::IValue output_ = module_.forward(input);
  std::cout &lt;&lt; "forward" &lt;&lt; std::endl;
  torch::Tensor detections_tens_ = output_.toTuple()-&gt;elements()[0].toTensor();
  std::cout &lt;&lt; "extracting detections" &lt;&lt; std::endl;
  std::cout &lt;&lt; "leaving inference" &lt;&lt; std::endl;
  return detections_tens_;
}
</code></pre>
<p>Model loading :</p>
<pre><code class="lang-auto">torch::jit::script::Module Detector::loadDetector(const std::string&amp; path , torch::Device device ) {
  //load torch module from a given path
  //module is the name given to a pytorch machine learning model
  try {
    // Deserialize the script module.torchscript.pth from a file using torch::jit::load().
    torch::jit::script::Module module ;
    module_ = torch::jit::load(path , device);
  }
  catch (const c10::Error&amp; e) {
    std::cerr &lt;&lt; "error loading the model located in : " &lt;&lt; path &lt;&lt; std::endl ;
    std::exit(EXIT_FAILURE);
  }
  //setting up device
  module_.to(device_);
  if (device != torch::kCPU) {
    module_.to(torch::kHalf);
  }
  module_.eval();
  return module_ ;
}```

Thanks for your help !</code></pre>

ptrblck: <p>As you’ve described, the issue is caused by a shape mismatch in the spatial dimension.<br>
Did you trace or script the model before exporting it? The former case would not capture any data-dependent control flow and I would assume that YOLO would need it.</p>

birds_are_drones: <p>Well I have tried but the program won’t allow me to do so. I’m working with mmpose and the complexity of the repo doesn’t allow that.<br>
I either get a non traced .pt or a .onnx . Would you happend to know how do I open a .onnx on libtorch OR how i can convert this onnx to pytorch simply?</p>

ptrblck: <p>No, unfortunately I don’t know the current support between libtorch and ONNX (and potential tools to convert between them).</p>

birds_are_drones: <p>Thans for your help <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I have finally found a way to do so.<br>
As you said,  my model was indeed not traced and this is what led to the error.</p>
<p><a href="https://github.com/ToriML/onnx2pytorch" rel="noopener nofollow ugc">I used this repos</a> to transform my onnx module to a pytorch traced module with the following unfininshed-but-you-get-the-idea script that converts onnx to pth and trace the pth model.<br>
(note : this code requires that you have a “torch” folder in your current folder to work well.<br>
I have a current folder that looked like this :</p>
<ul>
<li>onnx2torch.py</li>
<li>torch (where torch output models are stored)</li>
<li>onnx  (where onnx input models are stored)<br>
)</li>
</ul>
<pre><code class="lang-auto">#onnx2torch.py
import onnx
from onnx2pytorch import ConvertModel
import sys , os
import torch
import numpy as np

def main(argv):
    path_to_onnx = sys.argv[1]
    size = [1,3,384,288] # sys.argv[2]
    #size = np.fromstring(size, dtype=int, sep=',')
    print(size)

    torch_file = path_to_onnx[ path_to_onnx.find('/') :path_to_onnx.find('.') ] + ".pth"
    path_to_torch = os.path.join("torch" , torch_file)
    path_to_torch = "torch" + path_to_torch
    print(f"load path = {path_to_onnx}")
    print(f"save_path = {path_to_torch}")

    onnx_model = onnx.load(path_to_onnx)
    pytorch_model = ConvertModel(onnx_model)

    # An example input you would normally provide to your model's forward() method.
    dummy_input = torch.ones(size)

    # Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
    traced_script_module = torch.jit.trace(pytorch_model, dummy_input)
    traced_script_module.save(path_to_torch)
    print("saved")
    return


if __name__ == "__main__" :
    main(sys.argv)
</code></pre>