Problems of image segmentation using LibTorch

hktxt: <p>Hey, guys</p>
<p>I trained a image segmentation model and want to do inference using libtorch. The model predict a mask with a input image.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc0ad41b9cfecefd40e04a24114a93b8322afb17.png" data-download-href="https://discuss.pytorch.org/uploads/default/bc0ad41b9cfecefd40e04a24114a93b8322afb17" title="捕获1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc0ad41b9cfecefd40e04a24114a93b8322afb17_2_690x230.png" alt="捕获1" data-base62-sha1="qPuXNwS3lyTIMt7PbZ7xyG0rUq3" width="690" height="230" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc0ad41b9cfecefd40e04a24114a93b8322afb17_2_690x230.png, https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc0ad41b9cfecefd40e04a24114a93b8322afb17.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc0ad41b9cfecefd40e04a24114a93b8322afb17.png 2x" data-dominant-color="A68CA4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">捕获1</span><span class="informations">794×265 45.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I traced the model for later C++ usage.</p>
<pre><code class="lang-auto">#define CV_8UC3 CV_MAKETYPE(CV_8U,3)
#include &lt;torch/script.h&gt; // One-stop header.
#include &lt;iostream&gt;
#include &lt;memory&gt;
#include&lt;opencv2/opencv.hpp&gt;
#include &lt;opencv2/core/core.hpp&gt;
#include &lt;opencv2/imgproc/imgproc.hpp&gt;
#include &lt;opencv2/highgui/highgui.hpp&gt;
#include &lt;opencv2/imgproc/types_c.h&gt;
#include &lt;cuda.h&gt;
#include &lt;cuda_runtime.h&gt;

using namespace cv;
using namespace std;

int main() {
    std::string model_path = "D:/project/WDD/model_cpu.pt";
    std::string image_path = "D:/data/glass_crack/converted/84/img.png";

    torch::jit::script::Module model = torch::jit::load(model_path);
    assert(module != nullptr);
    std::cout &lt;&lt; "load model sucessfully.\n";

    //load img and normalize
    Mat img = imread(image_path, 1); 
    cv::cvtColor(img, img, CV_BGR2RGB);
 
    if (img.empty())
    {
        printf("could not show image...");
        return -1;
    }
    
    cv::Mat img_float;
    img.convertTo(img_float, CV_32FC3, 1.0f / 255.0f); 

    auto tensor_image = torch::from_blob(img_float.data, { 1, img.cols, img.rows, 3 });
    tensor_image = tensor_image.permute({ 0, 3, 1, 2 });
    //normalize
    tensor_image[0][0] = tensor_image[0][0].sub(0.485).div(0.229); 
    tensor_image[0][1] = tensor_image[0][1].sub(0.456).div(0.224);
    tensor_image[0][2] = tensor_image[0][2].sub(0.406).div(0.225);

    std::vector&lt;torch::jit::IValue&gt; inputs;
    inputs.emplace_back(tensor_image);

    // Execute the model and turn its output into a tensor.
    at::Tensor out_tensor = model.forward(inputs).toTensor();

    // convert result to CV mat and save
    out_tensor = out_tensor.squeeze().detach(); 
    out_tensor = out_tensor.mul(255).clamp(0, 255).to(torch::kU8);
    std::cout &lt;&lt; out_tensor.sizes() &lt;&lt; '\n';
    cv::Mat resultImg(img.rows, img.cols, CV_8UC3);
    std::memcpy((void*)resultImg.data, out_tensor.data_ptr(), sizeof(torch::kU8) * out_tensor.numel());
    imwrite("landscape_output.jpg", resultImg);
    std::cout &lt;&lt; "Done!\n";
    while (1);
}
</code></pre>
<p>it loaded model and did forward sucessfully, however, the problem is the output. The output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f7d8dfa4772e855859e3ab6048b492850d523ce.png" alt="捕获" data-base62-sha1="fUhZV2B5QLon8lfCLwzwMnI9tYW" width="467" height="316"><br>
the output is not as expected, totally nonsense…</p>
<p>can someone help?</p>

ptrblck: <p>Could you try to permute the dimensions again to <code>NHWC</code> before converting the output to an OpenCV array?</p>

hktxt: <p>thanks for reply~<br>
the output mask has a shape of [N, C, H, W], here N =1, C=1.<br>
after</p>
<pre><code class="lang-auto">out_tensor = out_tensor.squeeze().detach(); 
</code></pre>
<p>out_tensor’s shape: [H, W].</p>
<p>As you recommend:</p>
<pre><code class="lang-auto">ut_tensor = out_tensor.squeeze(0).detach().permute({ 1,2,0 });
</code></pre>
<p>I use this line of code instead.  so the shape become: [H, W, 1]. the mask:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c3186f94c36aa81013bcfcfbfca341445953795.jpeg" alt="landscape_output" data-base62-sha1="6iX7swSkaHWSRfZZCVp0A17Zxrv" width="650" height="450"><br>
not working…</p>
<p>I find that C = 1, so I changed</p>
<pre><code class="lang-auto">cv::Mat resultImg(img.rows, img.cols, CV_8UC3);
</code></pre>
<p>to</p>
<pre><code class="lang-auto">cv::Mat resultImg(img.rows, img.cols, CV_8UC1);
</code></pre>
<p>then the predicted mask:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52bce28958981b2964e035b59b5e38da0463e38f.jpeg" alt="landscape_output" data-base62-sha1="bNVPRZtNfBXYuyNFidNLocQQv1R" width="650" height="450"></p>
<p>…</p>

ptrblck: <p>Thanks for the update!<br>
The current output still looks as if the copying of the data is running into a row/columns mismatch.<br>
You see these diagonal “edges”, which would correspond to the desired width, but are shifted in each row.<br>
I would recommend to check the shape of the output, the created OpenCV image and make sure that the output tensor is contiguous in memory.</p>

hktxt: <p>problem solved. thanks for help.</p>
<p>Yes, the row/col problem. the seg model should outputs a mask which has the same shape of input image. However, a input image with 450x650 has a predicted mask of shape 448x648. python predicts the same shape but libtorch does not.<br>
so I modified:</p>
<pre><code class="lang-auto">cv::Mat resultImg(out_tensor.sizes()[0], out_tensor.sizes()[1], CV_8UC1);
</code></pre>
<p>problem solved.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/353cd2a5798a6ad4ff9f75942564ba260f60082a.jpeg" alt="landscape_output" data-base62-sha1="7AXCkB9vd3soU2sii4VE07axgls" width="648" height="448"></p>

ptrblck: <p>Good to hear you solved the issue!<br>
However, why doesn’t <code>libtorch</code> output the same shape as the Python model?<br>
Did you narrowed down this issue, as it sounds like a bug.</p>

hktxt: <p>caffe has a differnent implementation of convolutional layers compared with pytorch.<br>
if the code of libtorch comes from caffe, then it explains why.</p>