How to use Pytorch Tensor object in Opencv without convert to numpy array?

soaxeus: <p>I’m trying to develop a text detection application with Pytorch and Opencv in Python.<br>
I can use Pytorch tensor with Opencv like below.</p>
<p><code>val = y[0,:,:,0].data.cpu().numpy()</code><br>
<code>cv2.threshold(val , 0.4, 1, 0)</code></p>
<p>But it takes a lot of time. I need to do this operation by using tensor object. How can I do that?</p>

albanD: <p>Hi,</p>
<p>Give that you use <code>.cpu()</code>, I guess you have a cuda Tensor?<br>
Unfortunately, I don’t think opencv support gpu? So you will have to move the Tensor back to CPU to use it with opencv.<br>
Note that the conversion to numpy itself is almost free as we share memory with the numpy array.</p>
<p>If you use operations that are available on pytorch, I would advise using pytorch’s gpu version of these ops to keep best performances !</p>

soaxeus: <p>I already used <code>.cpu()</code> but I got <code>Expected Ptr&lt;cv::UMat&gt; for argument '%s'</code> error. I know I should GPU version but OpenCV does not support gpu and convert from <code>torch.tensor() numpy array</code> is slowing the process.</p>
<p>So I want to give image to OpenCV as <code>torch.tensor()</code>(CPU version) but I got this error <code>Expected Ptr&lt;cv::UMat&gt; for argument '%s'</code></p>

albanD: <p>Hi,</p>
<p>The conversion from the torch Tensor to the numpy array cannot be the slow part as it is doing almost nothing.<br>
It might be the conversion done by openCV from a numpy array to an open cv matrix?</p>

Isaac_Kargar: <p>I have the same question. In my case, I have a tensor which is the output of my model with <code>requires_grad = True</code> and I want to do some process on it using OpenCV in my custom loss. I’m trying to do it on GPU while keeping the <code>requires_grad = True</code>. Do you have any suggestions? I found kornia and it seems they are adding some OpenCV functions, but not the ones I need.</p>

albanD: <p>Hi,</p>
<p>If you want to use OpenCV, you won’t be able to use autograd (just like if you want to use numpy).<br>
You will need to write a custom autograd Function to tell the autograd what the backward should be: <a href="https://pytorch.org/docs/stable/notes/extending.html">https://pytorch.org/docs/stable/notes/extending.html</a></p>

Isaac_Kargar: <p>Thanks. The link was very helpful. I can avoid using opencv at the first step. but there is another problem.<br>
Let me explain my problem a little bit more.  The outputs of my network are <code>n</code> (x,y) positions, for a car, and they are in the world coordinate frame. I do some transformation on the output to bring them in the image coordinate frame by using pytorch and kornia and keep the gradient. I wanted to draw some rectangles or ellipses for these points on a mask but for now I just want to consider them as a single point in the mask tensor.<br>
Now I have a target mask (an rgb image with zero and one elements) and want to calculate a metric like <code>IoU</code> between this target mask and my <code>n</code> points which are transformed into image coordinate frame. I wanted to create a mask for these points and easily calculate the <code>IoU</code> with the target mask, but I don’t know how to do it to be able to use backprop and keep gradient. Can I attach the created mask tensor from my points to the graph and do backpropagation?</p>

albanD: <p>Hi,</p>
<p>Few things that jump to my mind:</p>
<ul>
<li>IoU is not differentiable AFAIK</li>
<li>If you create a mask containing binary values, you won’t be able to backprop as gradients only exist for continuous values.</li>
</ul>

Isaac_Kargar: <p>You are right. I think I need to have a head to output a mask directly. Thanks for the help.</p>