Libtorch C++ convert a Tensor to cv:Mat (single channel

jinfagang: <p>Hi, big masters and huge Gods,</p>
<p>I wanna ask a question hope anyone open this link can give me a help which is convert a Tensor to cv::Mat.</p>
<p>I have a Tensor outputs from my model and it’s [1, 19, 32, 43] shape, which is, batch_size, 19 instances, and 32x43 is the featuremap size.</p>
<p>I have it’s type: CPUFloatType. And I want converts it into cv::Mat, here is what I do:</p>
<pre><code class="lang-auto"> cv::Mat one_mat(heatMapsTensor.size(2), heatMapsTensor.size(3), CV_32FC1);
std::memcpy((void *) one_mat.data, one_heat_map.data_ptr(), sizeof(torch::kFloat32) * one_heat_map.numel());
</code></pre>
<p>heatMapsTensor is the tensor described above. one_mat is what I need to got.</p>
<p>But I got the mat incorrect. what did I wrong?</p>
<p>I suspect 2 maybe mistake:</p>
<ul>
<li>the type maybe not float32, but I do not know which it is;</li>
<li>the mat shape is not wrong</li>
</ul>
<p>Any body could help me out?</p>

Haydnspass: <p>Since you are using the tensor.data_ptr() you would not use <code>...sizeof(torch::kFloat32) * numel...</code> but rather <code>...sizeof(float) * numel...</code></p>
<p>Try something like:</p>
<pre><code class="lang-auto">std::memcpy(cv_pointer, tensor.data_ptr(), tensor.numel() * sizeof(float));
</code></pre>
<p>Assuming you are using floats.<br>
You may also want to have a look (<a href="https://discuss.pytorch.org/t/convert-torch-tensor-to-cv-mat/42751" class="inline-onebox">Convert torch::tensor to cv::Mat</a> . from <a class="mention" href="/u/mhubii">@mhubii</a> )</p>

jinfagang: <p>thanks for you reply!!</p>
<p>I changed into this:</p>
<pre><code class="lang-auto">	  cv::Mat one_mat(heatMapsTensor.size(1), heatMapsTensor.size(2), CV_32FC1);
	  std::memcpy(one_mat.data, one_heat_map.data_ptr(), sizeof(float) * one_heat_map.numel());
</code></pre>
<p>This time, the cv::Mat values seems normal, but result still not right.</p>
<p>what I want , is convert a tensor in multi dimension: [19, 32, 46] into a vector, length is 19, and every member in vetor is cv::Mat with size of 32,46.</p>
<p>Now, dims is right, the issue is the value converted into cv::Mat is always not right, it is not the same value in tensor.</p>

Haydnspass: <p>Ahh,<br>
then those are two kinds of problems.<br>
What you can do is loop over the first dimension and slice the tensor and then do std::mmcpy</p>
<p>So something like this</p>
<pre><code class="lang-auto">std::vector&lt;cv::Mat&gt; list_of_mat;
for (int i=0; i &lt; tensor.size(0); i++) {
// tensor slicing
// std::memcpy to dummy
list_of_mat.pushback(dummy)
}
</code></pre>

jinfagang: <p>This is actually what I am doing for now:</p>
<pre><code class="lang-auto">std::vector&lt;cv::Mat&gt; heatMaps(heatMapsTensor.size(0));

	for (size_t i = 0; i &lt; heatMaps.size(); i++) {
	  torch::Tensor one_heat_map = heatMapsTensor[i];
	  cv::Mat one_mat(heatMapsTensor.size(1), heatMapsTensor.size(2), CV_32FC1);
	  std::memcpy(one_mat.data, one_heat_map.data&lt;float&gt;(), sizeof(float) * one_heat_map.numel());
	  heatMaps[i] = one_mat;
	}
</code></pre>
<p>I just saw this usage <code>tensor_a.data&lt;float&gt;()</code> which seems a clue so I changed to this but still not right.</p>
<p>I am trying another thread:</p>
<p>get the tensor pointer say float* p, then copy it’s data to cv::Mat. However, how should I get the point of a tensor ? using tensor_a.data_ptr()? But that is (void*) not what I need float*</p>

dambo: <p>From cv2 to TORCH:</p>
<pre><code class="lang-auto"> cv::cvtColor(frame, frame, CV_BGR2RGB);
 frame.convertTo(frame, CV_32FC3, 1.0f / 255.0f);
 auto input_tensor = torch::from_blob(frame.data, {1, frame_h, frame_w, kCHANNELS});
 input_tensor = input_tensor.permute({0, 3, 1, 2});
</code></pre>
<p>From TORCh to cv2:</p>
<pre><code class="lang-auto">input_tensor = input_tensor.to(at::kCUDA);
                torch::Tensor out_tensor = module-&gt;forward({input_tensor}).toTensor();

                out_tensor = out_tensor.squeeze().detach().permute({1, 2, 0});
                out_tensor = out_tensor.mul(255).clamp(0, 255).to(torch::kU8);
                out_tensor = out_tensor.to(torch::kCPU);
                cv::Mat resultImg(frame_h, frame_w, CV_8UC3);
                std::memcpy((void *) resultImg.data, out_tensor.data_ptr(), sizeof(torch::kU8) * out_tensor.numel());
</code></pre>

jinfagang: <p>this really works!!!</p>

maris_ancans: <p>A million thanks to you!<br>
Would it be possible to also post how to get CV_32C3 format instead of CV_8UC3?<br>
I cant get it working, im getting artifacts</p>

Erun: <p>It works! I think the process"  out_tensor = out_tensor.to(torch::kCPU) " is crucial , I am right??<br>
only we must put the output tensor from GPU to CPU, then we can convert the data to Mat?!</p>

dambo: <p>Yes, indeed. Since the OpenCV MAT does not reside on the GPU.</p>

aurelien-m: <p>Would it be possible to load the image directly in RGB? It slows down a bit my inferences… <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

edgarriba: <p>Try other libraries as Pangolin</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/stevenlovegrove/Pangolin">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">

      <a href="https://github.com/stevenlovegrove/Pangolin" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/e41b939ac6e086c007d73741fabcc5a998181476ad64d8abfd263a18054fe959/stevenlovegrove/Pangolin" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/stevenlovegrove/Pangolin" target="_blank" rel="noopener nofollow ugc">GitHub - stevenlovegrove/Pangolin: Pangolin is a lightweight portable rapid...</a></h3>

  <p>Pangolin is a lightweight portable rapid development library for managing OpenGL display / interaction and abstracting video input. - GitHub - stevenlovegrove/Pangolin: Pangolin is a lightweight po...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


firminshao25: <p>Hi Shlomo,may I ask a question, thank you in advance.<br>
What if my <code>out_tensor</code> is two segmentation maps（float） and I want to convert it to <code>CV_8UC1</code> mat?</p>
<pre><code class="lang-auto">    torch::Tensor output = model.forward({ tensor_image }).toTensor();
    output.print();  //  ==&gt;1x2x256x256
    //apply softmax layer
    torch::Tensor probs = torch::softmax(output, 1)[0];
    probs.print(); //  ==&gt; 2x256x256
</code></pre>
<p>Here,I got two probabilities map for a semantic segmentation task.One is the background class and the other is the object class.Now I want to convert from torch to cv2,simply I just use this one line code,</p>
<pre><code class="lang-auto">Mat seg_map(256, 256, CV_32FC1, probs[1].data_ptr());
</code></pre>
<p>However, I found there are some difference from model inference in Python.Is there a better way to copy tensor data to mat?<br>
Thank you for your attention.</p>

rbaleksandar: <p>I would offer a couple of notes here in order for people to perhaps get a bigger picture. This code is great and simply works (very difficult to find working code for libtorch in general PERIOD).</p>
<p>A usual workflow would include PyTorch (where most people would develop the model and train it). After the model has been trained, you need to dump it to the CPU (that is, the RAM, not VRAM) and export it using tracing, which is part of TorchScript. This process is basically serializing your model so that it can be loaded both in Python as well as in C++.</p>
<p>A very important aspect is how you train your model. I was just playing around with SRCNN (among the oldest super resolution models out there) and it dawned on me that I have trained it with single channel images and not 3. So accordingly when you are loading an image with OpenCV you will have to consider that too.</p>