Intel

/

ldm3d-4c

like 20

Hi @terryryu !!

In ldm3d-4c/vae/config.json it is written that the VAE has four input channels, which is different from the expected six channels written in the paper (RGB + RGB-like depth map).

Indeed! For this version of ldm3d, that we called "ldm3d-4c", we mapped the depth to a 1D vector, making the input a 4-channel vector (3 for RGB and 1 for Depth).

Also, did you use bicubic upsampling to scale the depthmap resolution from 384 x 384 to 512 x 512?

We are using dpt-512 on a 512 input size. The output is automatically of 512 resolution

As for the error, I am not sure. A bit hard to answer. Maybe you have a short snippet so I can try to reproduce?

Best
Estelle

Thank you for your help @estellea !

I've come up with a minimal example that encodes and decodes the RGBD output of the lemon example.

import torch
import cv2
import numpy as np
from einops import rearrange
from diffusers import AutoencoderKL
from diffusers.image_processor import VaeImageProcessorLDM3D

def load_images(rgb_path, depth_path):
    rgb_img = cv2.imread(rgb_path) / 255.
    depth_img = cv2.imread(depth_path, cv2.IMREAD_UNCHANGED) # ensures 16-bit is preserved

    if depth_img.dtype != np.uint16:
        raise ValueError("Depth image is not 16-bit!")

    depth_img = depth_img / 65536.


    depth_img_expanded = np.expand_dims(depth_img, axis=-1)
    merged_img = np.concatenate([rgb_img, depth_img_expanded], axis=-1)

    return merged_img

with torch.no_grad():

    test_rgbd = load_images("/home/terryryu/Experiments/LDM3D/lemons_ldm3d_rgb.jpg", "/home/terryryu/Experiments/LDM3D/lemons_ldm3d_depth.png")

    vae = AutoencoderKL.from_pretrained("/home/terryryu/Weights/LDM3D/vae/", local_files_only=True, torch_dtype=torch.float16).cuda()
    vae_scale_factor = 2 ** (len(vae.config.block_out_channels) - 1)
    processor = VaeImageProcessorLDM3D(vae_scale_factor=vae_scale_factor)
    
    test_rgbd = rearrange(test_rgbd, "h w c -> 1 c h w")
    test_rgbd = torch.cuda.HalfTensor(test_rgbd)
    latents = vae.encode(test_rgbd).latent_dist.mode()
    image = vae.decode(latents / vae.config.scaling_factor, return_dict=False)[0]
    output_type = "pil"
    do_denormalize = [True] * image.shape[0]
    rgb, depth = processor.postprocess(image, output_type=output_type, do_denormalize=do_denormalize)
    
    rgb[0].save("./minimal_test_rgb.png")
    depth[0].save("./minimal_test_depth.png")

Strangely, the color of the decoded result is broken. Maybe I've made a minor mistake somewhere...?

Input RGB	Input Depth	Output RGB	Output Depth

Best,
Ryu

I have the same problem how to use the ldm3d-4c vae to reconstruct the RGB and depth image?

Before encoding, make sure to normalize both the image and the depth to the range of [-1, 1], by adding this to 'load_images':
rgb_img = 2. * rgb_img - 1.
depth_img = 2. * depth_img - 1.
also, there is no need to divide the latents by any scaling factor during the reconstruction since the latent space does not undergo scaling at this stage. However, when utilizing the diffusion aspect of the process, ensure that you scale the latent space both prior to and after the diffusion to achieve the desired results.
@terryryu , attaching updated example:

import torch
import cv2
import numpy as np
from einops import rearrange
from diffusers import AutoencoderKL
from diffusers.image_processor import VaeImageProcessorLDM3D

def load_images(rgb_path, depth_path):
    rgb_img = cv2.imread(rgb_path) / 255.
    rgb_img = 2.*rgb_img - 1.
    depth_img = cv2.imread(depth_path, cv2.IMREAD_UNCHANGED) # ensures 16-bit is preserved

    if depth_img.dtype != np.uint16:
        raise ValueError("Depth image is not 16-bit!")

    depth_img = depth_img / 65536.
    depth_img = 2.*depth_img - 1.


    depth_img_expanded = np.expand_dims(depth_img, axis=-1)
    merged_img = np.concatenate([rgb_img, depth_img_expanded], axis=-1)

    return merged_img

with torch.no_grad():

    test_rgbd = load_images("/home/terryryu/Experiments/LDM3D/lemons_ldm3d_rgb.jpg", "/home/terryryu/Experiments/LDM3D/lemons_ldm3d_depth.png")

    vae = AutoencoderKL.from_pretrained("/home/terryryu/Weights/LDM3D/vae/", local_files_only=True, torch_dtype=torch.float16).cuda()
    vae_scale_factor = 2 ** (len(vae.config.block_out_channels) - 1)
    processor = VaeImageProcessorLDM3D(vae_scale_factor=vae_scale_factor)
    
    test_rgbd = rearrange(test_rgbd, "h w c -> 1 c h w")
    test_rgbd = torch.cuda.HalfTensor(test_rgbd)
    latents = vae.encode(test_rgbd).latent_dist.mode()
    image = vae.decode(latents, return_dict=False)[0]
    output_type = "pil"
    do_denormalize = [True] * image.shape[0]
    rgb, depth = processor.postprocess(image, output_type=output_type, do_denormalize=do_denormalize)
    
    rgb[0].save("./minimal_test_rgb.png")
    depth[0].save("./minimal_test_depth.png")

Best,
Gabi