Repository: kornia/kornia

Issue:
warp_frame_depth fails with different aspect ratios between src and dst
## 🐛 Bug

As in the title: I am trying to get `kornia.geometry.warp_frame_depth` to work for my project and a seemingly intermittent bug appears to boil down to that - `warp_frame_depth` works when the aspect ratios of `src` and `dst` match but fails otherwise.
<!-- A clear and concise description of what the bug is. -->

## To Reproduce
I made a small repro repository with sample images, depths and camera parameters [here](https://github.com/jatentaki/warp-bug). Just clone it and execute `warp.py` to see the behavior (it uses kornia, numpy, matplotlib, imageio and h5py modules). You can comment out lines 6-7 in favor of 10-11 to see a working example with images of equal aspect ratio.

## Expected behavior
The warping behavior should not depend on aspect ratios. It may seem that a workaround is to pad the images until they are of equal shape, but this doesn't seem to be the case.

## Environment
kornia 0.4.0

PyTorch version: 1.6.0
Is debug build: No
CUDA used to build PyTorch: None

OS: Ubuntu 18.04.5 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.18.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] torch==1.6.0
[pip3] torch-cluster==1.5.4
[pip3] torch-dimcheck==0.0.1
[pip3] torch-geometric==1.4.3
[pip3] torch-localize==0.1.0
[pip3] torch-scatter==2.0.4
[pip3] torch-sparse==0.6.2
[pip3] torchvision==0.7.0
[conda] blas                      1.0                         mkl  
[conda] cpuonly                   1.0                           0    pytorch
[conda] mkl                       2020.1                      217  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.1.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.1           py38hbc911f0_0  
[conda] numpy-base                1.19.1           py38hfa32c7d_0  
[conda] pytorch                   1.6.0               py3.8_cpu_0  [cpuonly]  pytorch
[conda] torch-cluster             1.5.4                    pypi_0    pypi
[conda] torch-dimcheck            0.0.1                    pypi_0    pypi
[conda] torch-geometric           1.4.3                    pypi_0    pypi
[conda] torch-localize            0.1.0                    pypi_0    pypi
[conda] torch-scatter             2.0.4                    pypi_0    pypi
[conda] torch-sparse              0.6.2                    pypi_0    pypi
[conda] torchvision               0.7.0                  py38_cpu  [cpuonly]  pytorch



Discussion:
@jatentaki not suprised since we didn't design the functionality to support this use case. Just to verify, does it work if you resize the depth maps to the image resolution ? by doing that you might have to scale the camera intrinsics too https://kornia.readthedocs.io/en/latest/geometry.epipolar.html#kornia.geometry.epipolar.scale_intrinsics
@edgarriba I'm not sure if I follow: if I rescale an image to match the size of another with a different aspect ratio, I will apply different scaling to x and y axes. This means that `kornia.geometry.epipolar.scale_intrinsics` will not work since it applies uniform scaling. Am I misunderstanding something?

My actual use case is to warp between images of equal *tensor* shape (so that they can be batched), but they represent different camera aspect ratios. I scale (along with camera intrinsics) and zero-pad them to arrive at the same tensor size, but logically they represent different photo shapes. It still doesn't work in that case. In fact, I first thought there was something wrong with my scale/pad/adjust intrinsics code, and that's why I am posting this code above, as a minimal reproducing example.
I made a [branch](https://github.com/jatentaki/warp-bug/blob/equal_shape/warp.py) where I implement my actual use case: I first scale the images while preserving aspect ratio and then pad the shorter side to arrive at uniform (1024, 1024) shape. I adjust K with `scale_intrinsics`. As far as I understand, that's what you had in mind when asking me to resize depth maps to image resolution. The issue remains as can be seen when running the code.
I'll investigate further. If you find the actual issue, please propose a fix for this
@edgarriba thanks. I made yet another [branch](https://github.com/jatentaki/warp-bug/tree/centered) where I symmetrically pad the images on both sides (so it's like `0 0 [..image..] 0 0` instead of `[..image..] 0 0 0 0`) and set `cx, cy` to the center of the padded image (since in my data the focal point is always the center anyway). This doesn't work either, but to my surprise it gives different results (distortions) than the previous mode of padding.

I am far from comfortable with epipolar geometry so it can well be a grave misunderstanding of mine rather than a bug, but in any case we would ideally arrive at a working example to include in the repo.
Just had a quick glance on the problem and while the [depth warper](https://github.com/kornia/kornia/blob/master/kornia/geometry/warp/depth_warper.py) can deal with different cameras from source do destination, it's not the case for the function [warp_frame_depth](https://github.com/kornia/kornia/blob/8991e4d5f7a0fa1f04029311821a50d9fcf5e93f/kornia/geometry/depth.py#L107)

You should try to use the `kornia.geometry.warp.depth_warp` function instead. It will need you to create two pinhole cameras though. (`kornia.geometry.camera.PinholeCamera`, see [here](https://github.com/kornia/kornia/blob/8991e4d5f7a0fa1f04029311821a50d9fcf5e93f/kornia/geometry/camera/pinhole.py#L11) )

We might need to add an optional K2 parameter to the `warp_depth_frame` function to take this into account.


@ClementPinard thanks for your suggestion - you are definitely right. In case @jatentaki's issue is solved using `depth_warper`, we must definitely merge both functionalities.
@ClementPinard thanks a lot for investigating, I'll use `depth_warp` soon and let you know if my problem is solved.
@ClementPinard could you share your code using `depth_warp`? I [tried](https://github.com/jatentaki/warp-bug/tree/pinhole_equal) but I can't get it right. Interestingly, now the previously problematic case (unequal aspect ratio) is close to being correct while the equal aspect ratio pair is completely distorted. I am a bit confused about the API, in particular

1. What is the use of `height, width` parameters in `depth_warp`, given that we provide both intrinsics (as part of `PinholeCamera`) and image tensor shapes (since we provide both `depth_src` and `image_dst`)? It mirrors the API of `DepthWarper`, but in this case couldn't this be inferred from the remaining parameters?
2. What should be the layout of `intrinsics` matrix? The introduction to [`kornia.geometry.camera`](https://kornia.readthedocs.io/en/latest/geometry.camera.pinhole.html) introduces a 3x4 Rt and a 3x3 K but then the docs for [`PinholeCamera`](https://kornia.readthedocs.io/en/latest/geometry.camera.pinhole.html#kornia.geometry.camera.pinhole.PinholeCamera) itself speak of both being 4x4. I inferred from source code that both should be "extended" by an extra row/column with zeros and one on the diagonal, but I am not sure if I am reading this correctly.
I completely agree with you that the depth_warp API is far from flexible enough. Height and width should be deducted from depth shape, and Intrinsics the here needs to be 4x4 should only need to be 3x3 and the conversion should be done within the class.

Other than that I couldn't get right results for the moment. I can share you the code, but it does not work and I don't know exactly why. I'll get back to you with the solution when I find it :)

Anyway, here's what I wrote (on top of your own script):
https://gist.github.com/ClementPinard/e7353dee56faf3f4c62f0753a2703568

It's possible I don't use the API right myself, I'll have to investigate.
After some more investigation, there IS a bug in kornia. in `test/geometry/warp/test_depth_warper.py` , the case with different intrinsics from src to dst is never tested, and when we change that, the test fails.

This does not say that my implementation is correct, but at least we know that something in Kornia either regarding depth warper or depth warper's tests needs to be changed.

We might need to change the test script so that it now addresses the different intrinsics case and solve the now failing test and then we will be able to look at this problem. And obviously, there is also the problem of the warp_depth_frame function API that needs to change as well to accept the second intrinsics parameter.
I think I found it. [This](https://github.com/kornia/kornia/blob/master/kornia/geometry/warp/depth_warper.py#L83) line effectively computes

```python
trans = pinhole_src.extrinsics.inverse() @ self._pinhole_dst.extrinsics
```

whereas it should be 


```python
trans = self._pinhole_dst.extrinsics @ pinhole_src.extrinsics.inverse()
```

instead. I'll make a PR.

Resolving patch:
diff --git a/kornia/geometry/warp/depth_warper.py b/kornia/geometry/warp/depth_warper.py
--- a/kornia/geometry/warp/depth_warper.py
+++ b/kornia/geometry/warp/depth_warper.py
@@ -3,8 +3,7 @@
 import torch.nn as nn
 import torch.nn.functional as F
 
-from kornia.geometry.linalg import transform_points
-from kornia.geometry.linalg import relative_transformation
+from kornia.geometry.linalg import transform_points, inverse_transformation, compose_transformations
 from kornia.geometry.conversions import convert_points_to_homogeneous
 from kornia.geometry.conversions import normalize_pixel_coordinates
 from kornia.geometry.camera import PinholeCamera
@@ -80,8 +79,10 @@ def compute_projection_matrix(
                             "PinholeCamera. Got {}".format(type(pinhole_src)))
         # compute the relative pose between the non reference and the reference
         # camera frames.
-        dst_trans_src: torch.Tensor = relative_transformation(
-            pinhole_src.extrinsics, self._pinhole_dst.extrinsics)
+        dst_trans_src: torch.Tensor = compose_transformations(
+            self._pinhole_dst.extrinsics,
+            inverse_transformation(pinhole_src.extrinsics),
+        )
 
         # compute the projection matrix between the non reference cameras and
         # the reference.


Test patch: