How to combine an image tensor (4D) and a depth tensor (4D) to create a 5D tensor [batch size, channels, depth, height, width]?

lef6: <p>I am using the <a href="https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html" rel="noopener nofollow ugc">FlyingThings3D and Monkaa datasets</a>.<br>
During training, I load the images and disparity data. The image tensor is of shape: [2, 3, 256, 256], and disparity/depth tensor is of shape: [2, 1, 256, 256] (batch size, channels, height, width). I would like to use Conv3D, so I need to combine these two tensors and create a new tensor of shape: [2, 3, 256, 256, 256] (batch size, channels, depth, height, width).<br>
The depth values range from 0-400, and a possibility is to divide that into intervals, e.g., 4 intervals of 100. I would like the resulting tensor to look like a voxel, similarly to the technique used in this <a href="https://openaccess.thecvf.com/content_ICCV_2017_workshops/papers/w17/Zia_RGB-D_Object_Recognition_ICCV_2017_paper.pdf" rel="noopener nofollow ugc">paper</a>.<br>
The training loop that iterates over the data is below:</p>
<pre><code class="lang-auto">for batch_id, sample in enumerate(train_loader):
        
        sample = {name: tensor.cuda() for name, tensor in sample.items()}

        # image tensor [2, 3, 256, 256]
        rgb_image = transforms.Lambda(lambda x: x.mul(255))(sample["frame"]) 
        
        # translate disparity to depth
        depth_from_disparity_frame = 132.28 / sample["disparity_frame"]
        # depth tensor [2, 1, 256, 256]
        depth_image = depth_from_disparity_frame.unsqueeze(1) 
</code></pre>

ptrblck: <p>I’m not sure how the transformation or “combination” should work, as you have:</p>
<pre><code class="lang-python">2 * 1 * 256 * 256 + 2 * 3 * 256 * 256 = 524288
</code></pre>
<p>values initially in both tensors and want to create a single tensor with:</p>
<pre><code class="lang-python">2 * 3 * 256 * 256 * 256 = 100663296
</code></pre>
<p>values, which are 192x more values.<br>
Could you  explain a bit more how you would like to “create” these new values?</p>

lef6: <p>Thank you for your response.</p>
<p>This is better illustrated in the figure below – this is only an illustration for the R channel:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f9566e4a07a847a45c62ba02592ef7fed59a851.png" data-download-href="https://discuss.pytorch.org/uploads/default/5f9566e4a07a847a45c62ba02592ef7fed59a851" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f9566e4a07a847a45c62ba02592ef7fed59a851_2_517x283.png" alt="image" data-base62-sha1="dDzs0SQc6rz4zgpc2eHwajQjzNv" width="517" height="283" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f9566e4a07a847a45c62ba02592ef7fed59a851_2_517x283.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f9566e4a07a847a45c62ba02592ef7fed59a851_2_775x424.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f9566e4a07a847a45c62ba02592ef7fed59a851.png 2x" data-dominant-color="F1ECEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">930×510 96 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The depth values are quantized and then are used to convert the RGB images to RGB-D voxel representations. Currently, the depth tensor contains a single depth value per pixel. The values range from 0 to 400, so they can be quantized in 4 intervals. Do you have any idea or advice on how I can utilize the depth tensor to produce this kind of input representation? Thanks again.</p>

Matias_Vasquez: <p>I think the best way to do this is by forming a [2, 4, 256, 256] tensor.</p>
<p>As you showed in your picture, the depth (D) represents the distance.</p>
<p>So for each pixel in the 256x256 image, you have one R, one G, one B and now one D additional value.<br>
But the batch (B) and the height (H) and width (W) stay the same.</p>
<p>So for this example you gave, I think this would be enough.</p>
<p>However, if what you actually want are all of the Voxels (having zero wherever the actual pixel is not present), then what you should do is change the depth D into a one hot vector encoding of the dimension you want.</p>
<p>So for example you mention that the range is 0-400</p>
<p>Then you could have a tensor [2, 3, 400, 256, 256], where 400 is the one hot vector encoding for the depth, having only a 1 in the depth where the pixel is.</p>
<p>I think both approaches should work, but it is up to you how you implement it.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Matias_Vasquez: <p>I am sure there is a better solution instead of the for loops, but this should work</p>
<pre><code class="lang-python">d = torch.randint(0, 400, (2, 1, 256, 256))   # Your depth tensor
x = torch.randint(0, 256, (2, 3, 256, 256))   # Your image tensor

# Your desired dimension depth
depth = 256

z = torch.zeros((2, 3, depth, 256, 256))        # Tensor with zeros

# Scaling from 400 to the desired depth
d = d * depth / 400 

for b in range(2):
    for h in range(256):
        for w in range(256):
            # Select only the desired depth for each pixel in each batch and put the actual values only there
            z[b, :, int(d[b, 0, h, w]), h, w] = x[b, :, h, w]

# The tensor consists of mainly zeros, only where the depth indicates are actual RGB values
print(z)
</code></pre>

lef6: <aside class="quote no-group quote-modified" data-username="Matias_Vasquez" data-post="5" data-topic="147246">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matias_vasquez/48/47847_2.png" class="avatar"> Matias_Vasquez:</div>
<blockquote></blockquote>
</aside>
<p>Thank you so much for your help.</p>
<p>I think the below works – transforming the tensor to one-hot encoding:</p>
<pre><code class="lang-auto">depth = depth * 256.0 / 400.0 
 
depth_ohe = torch.nn.functional.one_hot(depth.to(torch.int64), num_classes=256)       # if i have 256 classes, I will also try 400
bchwd_tensor = rgb_image.unsqueeze(-1) * depth_ohe       # of shape (batch, channel, height, width, depth)
bcdhw_tensor = bchwd_tensor.permute(0, 1, 4, 2, 3) 
</code></pre>