Cv2.rectangle is not working on images which are converted from tensor to numpy array?

username: ssshukla26
cooked: <p>Hi,</p>
<p>The problem is that when I retrieve the stored images from the custom data loader along with annotations, I am unable to draw a bounding box around the images which are stored as tensor and then retrieved as a NumPy array. Am using simple custom transforms. Below is the code of how the image is read and then what custom transforms are applied to it in my custom dataset class.</p>
<pre><code class="lang-auto">...
       # Read image from the file
        image =  cv2.cvtColor(cv2.imread(os.path.join(self.__root_dir, image_desc["file_name"])),
                              cv2.COLOR_BGR2RGB)
...
</code></pre>
<pre><code class="lang-auto">CUSTOM_TRANSFORMS = transforms.Compose([transforms.ToTensor()])
</code></pre>
<p>My custom data loader is working fine. And my custom dataset is also working fine. Below is the test code of the custom data loader and its output. (Batch size is equal to 1 for simplicity)</p>
<p>Code</p>
<pre><code class="lang-auto">for i, (images, annotations) in enumerate(train_dataloader):
    print(type(image[0]), annotations[0])
</code></pre>
<p>Output</p>
<pre><code class="lang-auto">&lt;class 'numpy.ndarray'&gt; {'image_id': tensor([0]), 'boxes': tensor([[140., 188., 306., 293.]]), 'area': tensor([17430.]), 'iscrowd': tensor([0]), 'labels': tensor([1])}
&lt;class 'numpy.ndarray'&gt; {'image_id': tensor([1]), 'boxes': tensor([[216., 188., 381., 358.]]), 'area': tensor([28050.]), 'iscrowd': tensor([0]), 'labels': tensor([1])}
&lt;class 'numpy.ndarray'&gt; {'image_id': tensor([2]), 'boxes': tensor([[ 78., 257., 330., 433.]]), 'area': tensor([44352.]), 'iscrowd': tensor([0]), 'labels': tensor([1])}
&lt;class 'numpy.ndarray'&gt; {'image_id': tensor([3]), 'boxes': tensor([[ 59., 113., 325., 542.]]), 'area': tensor([114114.]), 'iscrowd': tensor([0]), 'labels': tensor([1])}
&lt;class 'numpy.ndarray'&gt; {'image_id': tensor([4]), 'boxes': tensor([[113., 153., 439., 382.]]), 'area': tensor([74654.]), 'iscrowd': tensor([0]), 'labels': tensor([1])}
&lt;class 'numpy.ndarray'&gt; {'image_id': tensor([5]), 'boxes': tensor([[144.,  78., 487., 342.]]), 'area': tensor([90552.]), 'iscrowd': tensor([0]), 'labels': tensor([1])}
</code></pre>
<p><strong>Below code is an explanation of what is happening in my case.</strong></p>
<p>Code</p>
<pre><code class="lang-auto"># Read image
image = cv2.cvtColor(cv2.imread("/content/dataset/train/img_0.jpg"), cv2.COLOR_BGR2RGB)
print(type(image))

# Make Bounding Box
bbox = np.array([140., 188., 306., 293.], dtype=np.int32)

# This Works
image_orig = np.copy(image)
print(type(image_orig), image_orig.shape)
cv2.rectangle(image_orig, (bbox[0], bbox[1]), (bbox[2], bbox[3]), color = (255, 0, 0), thickness = 3)

# This doesn't work
tensor_image = CUSTOM_TRANSFORMS(image) # The way image stored
tensor_image = tensor_image.permute(1, 2, 0) # Reshape
image_clone = tensor_image.numpy() # Convert To numpy array
print(type(image_clone), image_clone.shape)
cv2.rectangle(image_clone, (bbox[0], bbox[1]), (bbox[2], bbox[3]), color = (255, 0, 0), thickness = 3)

# This works for both image_orig and image_clone
for i in range(bbox[1], bbox[3]):
    for j in range(bbox[0], bbox[2]):
        image_orig[i, j, 1] = 255
        image_clone[i, j, 1] = 255

# Plot
fig = plt.figure(figsize=(18, 6), facecolor="white")

plt.subplot(131)
plt.title("Original image")
plt.xticks([])
plt.yticks([])
plt.imshow(image)

plt.subplot(132)
plt.title("original image with bounding box and overlay")
plt.xticks([])
plt.yticks([])
plt.imshow(image_orig)

plt.subplot(133)
plt.title("Converted image from tensor with overlay only (no bbox)")
plt.xticks([])
plt.yticks([])
plt.imshow(image_clone)

plt.suptitle("cv2.rectangle not working on data which is converted from tensor to image")
plt.subplots_adjust(wspace=0)
plt.show()
</code></pre>
<p>Output</p>
<pre><code class="lang-auto">Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
&lt;class 'numpy.ndarray'&gt;
&lt;class 'numpy.ndarray'&gt; (600, 450, 3)
&lt;class 'numpy.ndarray'&gt; (600, 450, 3)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74e6c3563b989761ffe48546cb50d4b5d68f49ed.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/74e6c3563b989761ffe48546cb50d4b5d68f49ed" title="Compare"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74e6c3563b989761ffe48546cb50d4b5d68f49ed_2_690x271.jpeg" alt="Compare" data-base62-sha1="gG9NeuHBau9GIRvHa6HdSMDGSwR" width="690" height="271" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74e6c3563b989761ffe48546cb50d4b5d68f49ed_2_690x271.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/7/4/74e6c3563b989761ffe48546cb50d4b5d68f49ed.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/4/74e6c3563b989761ffe48546cb50d4b5d68f49ed.jpeg 2x" data-dominant-color="96A08C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Compare</span><span class="informations">983×387 127 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please help me to understand what am doing wrong here.</p>
<p>Regards,<br>
Sunny.</p>

username: Epoching
cooked: <p><code>image_clone = tensor_image.numpy()</code> will cause <code>image_clone</code> to have values 0-1 instead of 0-255.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82fad39cf7f1981464ed6163213b3727937f0c9b.png" data-download-href="https://discuss.pytorch.org/uploads/default/82fad39cf7f1981464ed6163213b3727937f0c9b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82fad39cf7f1981464ed6163213b3727937f0c9b_2_565x500.png" alt="image" data-base62-sha1="iGHsOj4EK1KsFQ8CGFNE3u1QoEr" width="565" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82fad39cf7f1981464ed6163213b3727937f0c9b_2_565x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/82fad39cf7f1981464ed6163213b3727937f0c9b_2_847x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/2/82fad39cf7f1981464ed6163213b3727937f0c9b.png 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1082×956 37.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7e4cd3370f949036e500b1745f014cec39d154c9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7e4cd3370f949036e500b1745f014cec39d154c9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7e4cd3370f949036e500b1745f014cec39d154c9_2_452x500.jpeg" alt="image" data-base62-sha1="i1iL0xLncsy0DP49ZEVvsR3JJCV" width="452" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7e4cd3370f949036e500b1745f014cec39d154c9_2_452x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7e4cd3370f949036e500b1745f014cec39d154c9_2_678x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7e4cd3370f949036e500b1745f014cec39d154c9_2_904x1000.jpeg 2x" data-dominant-color="E9EAE8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1079×1193 255 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/2/620e88ea9cda5745ae4762b6bd1147b3799f1200.png" data-download-href="https://discuss.pytorch.org/uploads/default/620e88ea9cda5745ae4762b6bd1147b3799f1200" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/2/620e88ea9cda5745ae4762b6bd1147b3799f1200.png" alt="image" data-base62-sha1="dZrVPOkEhkZsia5OabqyMxA7nIk" width="690" height="429" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1063×661 39 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Epoching
cooked: <p>Following the above post, starting from where you make <code>tensor_image</code>. To cast <code>tensor_image</code> safely back into <code>numpy</code> with values being within 0-255, I’d use <code>torchvision</code>’s <code>ToPILImage</code> and from there go into <code>numpy,</code> as so:</p>
<p>(note that <code>tensor_to_image</code> was set equal to <code>ToPILImage()</code> in the second cell in the above post.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54ee57b6cdbee786109d150828d3e064c56d2f71.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/54ee57b6cdbee786109d150828d3e064c56d2f71" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54ee57b6cdbee786109d150828d3e064c56d2f71_2_509x500.jpeg" alt="image" data-base62-sha1="c7kKLlRPjjlNga4S4vu2mI1x0qt" width="509" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54ee57b6cdbee786109d150828d3e064c56d2f71_2_509x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54ee57b6cdbee786109d150828d3e064c56d2f71_2_763x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54ee57b6cdbee786109d150828d3e064c56d2f71_2_1018x1000.jpeg 2x" data-dominant-color="E8E9E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1128×1107 205 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f09ebfe14e9306563841022bc21644074b2bcef1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f09ebfe14e9306563841022bc21644074b2bcef1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f09ebfe14e9306563841022bc21644074b2bcef1_2_563x500.jpeg" alt="image" data-base62-sha1="ykCI9JLeP1DdqEEPhsSFC6bYz1n" width="563" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f09ebfe14e9306563841022bc21644074b2bcef1_2_563x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f09ebfe14e9306563841022bc21644074b2bcef1_2_844x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/0/f09ebfe14e9306563841022bc21644074b2bcef1.jpeg 2x" data-dominant-color="EAEBE8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1095×971 163 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ssshukla26
cooked: <p>Thanks for helping, the below code works.</p>
<pre><code class="lang-auto"># This Works
tensor_image = CUSTOM_TRANSFORMS(image) # The way image stored
image_clone = np.array(transforms.ToPILImage()(tensor_image))
print(type(image_clone), image_clone.shape)
cv2.rectangle(image_clone, (bbox[0], bbox[1]), (bbox[2], bbox[3]), color = (255, 0, 0), thickness = 3)
</code></pre>

username: Epoching
cooked: <p>Great to hear <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> Don’t forget to mark a reply as a correction answer in case other people are searching for a solution too!</p>
<p>Also: If you are wanting to go from tensor to numpy directly (and skip the PIL Image conversion part), that may be a good idea in terms of runtime. I’d imagine doing this for a bunch of images may be quite slow! I tried looking around for a bit and not sure what the exact cause of this error was</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/5/95d016786f122bbc8c457d69756d839b228b5b0b.png" data-download-href="https://discuss.pytorch.org/uploads/default/95d016786f122bbc8c457d69756d839b228b5b0b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95d016786f122bbc8c457d69756d839b228b5b0b_2_690x80.png" alt="image" data-base62-sha1="lniY5drPpsqlL9TpJrKudrPrUoz" width="690" height="80" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95d016786f122bbc8c457d69756d839b228b5b0b_2_690x80.png, https://discuss.pytorch.org/uploads/default/original/3X/9/5/95d016786f122bbc8c457d69756d839b228b5b0b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/5/95d016786f122bbc8c457d69756d839b228b5b0b.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">790×92 46.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It may have to do with something with the datatype (i.e. <code>numpy</code>’s <code>uint8</code> from <code>image_orig</code> versus <code>image_clone</code>’s datatype that comes from <code>tensor_image.numpy()</code>. I tried converting <code>image_clone</code> to be of <code>numpy</code>’s <code>uint8</code> via <code>(image_clone*255).astype(np.uint8)</code>, but got the same error when trying to draw rectangles, very weird. Additionally tried doing <code>(tensor_image*255).long().numpy()</code> so it’s already in a non-float format (and between 0-255) but no luck. There must be some other states of these numpy variables behind-the-scenes perhaps.</p>

username: hhxx
cooked: <p>The reason is that after “tensor_image.permute(1, 2, 0)”, the tensor_image is not contiguous in memory, which makes image_clone also not contiguous.</p>
<p>You can change the code in below two ways:</p>
<ol>
<li>tensor_image = tensor_image.permute(1, 2, 0); tensor_image.contiguous()<br>
or</li>
<li>image_clone = np.ascontiguousarray(image_clone)</li>
</ol>
<p>Then cv.rectangle can draw on the contiguous numpy object directly.</p>