How to convert an OpenCV image into libtorch Tensor?

username: Shisho_Sama
cooked: <p>How should I be converting between OpenCVs Mat into libtorch <code>Tensor</code>? so that I can feed it to my modules ?<br>
The modules are converted from Pytorch(python) so I’m not sure if they should be in <code>torch::Tensor</code> or <code>torch::jit::IValue</code> !<br>
So any help is greatly appreciated</p>

username: Shisho_Sama
cooked: <p>OK here is a simple sample that implements differents ops in libtorch, I wrote this as I was playing with different things , I guess  this might come in handy to lots of new commers :<br>
The following simple demo shows how to :</p>
<ol>
<li>Convert an opencv image into a Tensor</li>
<li>How to convert a Tensor into IValue which is used to feed the networks</li>
<li>How to Convert back from an existing tensor back to an OpenCV image</li>
<li>How to do simple or complex preprocessings on the input image/tensor</li>
<li>show basic stuff like how to transpose and or add batch dim to the tensor</li>
<li>How to see the shape of the tensor and slice of it<br>
7 and How to define a Module (load a pretrained model) and feed forward it and get its output!</li>
</ol>
<pre><code class="lang-cpp">
std::string get_image_type(const cv::Mat&amp; img, bool more_info=true) 
{
    std::string r;
    int type = img.type();
    uchar depth = type &amp; CV_MAT_DEPTH_MASK;
    uchar chans = 1 + (type &gt;&gt; CV_CN_SHIFT);

    switch (depth) {
    case CV_8U:  r = "8U"; break;
    case CV_8S:  r = "8S"; break;
    case CV_16U: r = "16U"; break;
    case CV_16S: r = "16S"; break;
    case CV_32S: r = "32S"; break;
    case CV_32F: r = "32F"; break;
    case CV_64F: r = "64F"; break;
    default:     r = "User"; break;
    }

    r += "C";
    r += (chans + '0');
   
    if (more_info)
        std::cout &lt;&lt; "depth: " &lt;&lt; img.depth() &lt;&lt; " channels: " &lt;&lt; img.channels() &lt;&lt; std::endl;

    return r;
}

void show_image(cv::Mat&amp; img, std::string title)
{
    std::string image_type = get_image_type(img);
    cv::namedWindow(title + " type:" + image_type, cv::WINDOW_NORMAL); // Create a window for display.
    cv::imshow(title, img);
    cv::waitKey(0);
}

auto transpose(at::Tensor tensor, c10::IntArrayRef dims = { 0, 3, 1, 2 })
{
    std::cout &lt;&lt; "############### transpose ############" &lt;&lt; std::endl;
    std::cout &lt;&lt; "shape before : " &lt;&lt; tensor.sizes() &lt;&lt; std::endl;
    tensor = tensor.permute(dims);
    std::cout &lt;&lt; "shape after : " &lt;&lt; tensor.sizes() &lt;&lt; std::endl;
    std::cout &lt;&lt; "######################################" &lt;&lt; std::endl;
    return tensor;
}

auto ToTensor(cv::Mat img, bool show_output = false, bool unsqueeze=false, int unsqueeze_dim = 0)
{
    std::cout &lt;&lt; "image shape: " &lt;&lt; img.size() &lt;&lt; std::endl;
    at::Tensor tensor_image = torch::from_blob(img.data, { img.rows, img.cols, 3 }, at::kByte);

    if (unsqueeze)
    {
        tensor_image.unsqueeze_(unsqueeze_dim);
        std::cout &lt;&lt; "tensors new shape: " &lt;&lt; tensor_image.sizes() &lt;&lt; std::endl;
    }
    
    if (show_output)
    {
        std::cout &lt;&lt; tensor_image.slice(2, 0, 1) &lt;&lt; std::endl;
    }
    std::cout &lt;&lt; "tenor shape: " &lt;&lt; tensor_image.sizes() &lt;&lt; std::endl;
    return tensor_image;
}

auto ToInput(at::Tensor tensor_image)
{
    // Create a vector of inputs.
    return std::vector&lt;torch::jit::IValue&gt;{tensor_image};
}

auto ToCvImage(at::Tensor tensor)
{
    int width = tensor.sizes()[0];
    int height = tensor.sizes()[1];
    try
    {
        cv::Mat output_mat(cv::Size{ height, width }, CV_8UC3, tensor.data_ptr&lt;uchar&gt;());
        
        show_image(output_mat, "converted image from tensor");
        return output_mat.clone();
    }
    catch (const c10::Error&amp; e)
    {
        std::cout &lt;&lt; "an error has occured : " &lt;&lt; e.msg() &lt;&lt; std::endl;
    }
    return cv::Mat(height, width, CV_8UC3);
}

int main(int argc, const char* argv[]) 
{
    std::string msg = "sample image";
    auto currentPath = "cpp\\port\\LibtorchPort\\imgs\\img1.jpg";
    auto img = cv::imread(currentPath);
    show_image(img, msg);

    // convert the cvimage into tensor
    auto tensor = ToTensor(img);

    // preprocess the image. meaning alter it in a way a bit!
    tensor = tensor.clamp_max(c10::Scalar(50));

    auto cv_img = ToCvImage(tensor);
    show_image(cv_img, "converted image from tensor");
    // convert the tensor into float and scale it 
    tensor = tensor.toType(c10::kFloat).div(255);
    // swap axis 
    tensor = transpose(tensor, { (2),(0),(1) });
    //add batch dim (an inplace operation just like in pytorch)
    tensor.unsqueeze_(0);

    auto input_to_net = ToInput(tensor);
   

    torch::jit::script::Module r18;

    try 
    {
        std::string r18_model_path = "D:\\Codes\\python\\Model_Zoo\\jitcache\\resnet18.pt";


        // Deserialize the ScriptModule from a file using torch::jit::load().
        r18 = torch::jit::load(r18_model_path);
    
        // Execute the model and turn its output into a tensor.
        at::Tensor output = r18.forward(input_to_net).toTensor();

        //sizes() gives shape. 
        std::cout &lt;&lt; output.sizes() &lt;&lt; std::endl;
        std::cout &lt;&lt; "output: " &lt;&lt; output[0] &lt;&lt; std::endl;
        //std::cout &lt;&lt; output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) &lt;&lt; '\n';
    
    }
    catch (const c10::Error&amp; e) 
    {
        std::cerr &lt;&lt; "error loading the model\n" &lt;&lt;e.msg();
        return -1;
    }

    std::cout &lt;&lt; "ok\n";
    std::system("pause");
    return 0;
}

</code></pre>
<p>All of the info here are nearly learnt from <a href="https://github.com/pytorch/pytorch/issues/12506" rel="nofollow noopener">this issue</a> and <a href="https://github.com/pytorch/pytorch/issues/14330" rel="nofollow noopener">this one</a><br>
Its a shame that a single issue has way much more information about libtorch than the entirity of the official documentation for libtorch/cpp!<br>
This is not fair to CPP!</p>

username: hedeya1980
cooked: <p>Hi <a class="mention" href="/u/shisho_sama">@Shisho_Sama</a>,<br>
Could you pls review my question at <a href="https://discuss.pytorch.org/t/libtorch-the-u-net-segmentation-output-is-correct-but-repeated-3-times-beside-each-other/158540">this link</a>, and help me out? Thanks a lot.</p>

username: JamesDickens
cooked: <p>Any thoughts on the at:: type for an unsigned 16 bit integer?</p>

username: lastlaugher
cooked: <p>Hi <a class="mention" href="/u/shisho_sama">@Shisho_Sama</a>,</p>
<p>Thanks for your sample code. It really helped me a lot. By the way, I have one thing to point out. The local variable <code>tensor_image</code> in <code>ToTensor()</code> function is destroyed when the function is returned, so <code>tensor</code> variable in <code>main()</code> function doesn’t have valid values. I hope this can help any other guys.</p>