Supplementary Material for
Taking a Deeper Look at the Inverse Compositional Algorithm
Zhaoyang Lv1,2 Frank Dellaert1 James M. Rehg1 Andreas Geiger2
1
Georgia Institute of Technology, Atlanta, United States
2
Autonomous Vision Group, MPI-IS and University of Tübingen, Germany{zhaoyang.lv, rehg}@gatech.edu

frank.dellaert@cc.gatech.edu

andreas.geiger@tue.mpg.de

In this supplementary document, we first provide details regarding the network architectures used in our model. Next,
we show additional experiments on the 2D affine transformation estimation task. Then we provide details on the Jacobian implementation for the rigid motion estimation. We
also present more details on our TUM RGB-D experiments
as well as the generation of the MovingObjects3D dataset.
Finally, we provide additional qualitative experiments forour method on the MovingObjects3D dataset.

duce computation. The network is composed of four convolutional layers, with dilation [1,2,4,1], followed by a sigmoid layer which normalizes the output to the range [0,1].
Note that despite the small size of our network, the dilation
layers and the coarse-to-fine process ensure a sufficiently
large receptive field.

1. Network Architectures

the right-hand-side (RHS) vector JT Wrk ∈ R1×6 corre(i)sponding to each residual map rk . Next, we flatten the
N RHS vectors jointly with the approximate Hessian matrix JT WJ ∈ R6×6 into a single vector, which is the input
to our Trust Region Network. This network is composed ofthree fully connected layers and outputs the damping vector.
At the last layer, a ReLU ensures non-negative elements.

(C) Trust Region Network: Fig. 3 shows the operations and parameters of our Trust Region Network. Given
the N residual maps rk(i) , i ∈ {1...N }, we first calculate(i)

In the following, we describe the network architectures
used in our model.
(A) Two-view Feature Encoder: Fig. 1 shows the architecture of our two-view feature encoder for estimating bothTθ and Iθ . The network takes two concatenated RGB-D
views as input. For the depth channel, we use the inverse
depth d clamped to [0, 10]. For the 2D affine experiments
we use only the RGB channels.
The feature pyramid comprises one (A) Two-view Feature Encoder at each of the four pyramid levels. Each feature encoder uses three dilated convolutional layers. We usea Spatial Average Pooling layer to downsample the output
features from the fine scale as input to the next coarser scale.

2. 2D Affine Motion Estimation
The proposed framework is general and can be applied
to a wide range of motion models apart from the 3D rigid
motion estimation tasks presented in the main paper, see,e.g., [5]. To demonstrate its generality, this section provides
results on 2D affine motion estimation. While 2D affine
motion estimation is in general easier than 3D rigid motion
estimation, occlusions cannot be treated explicitly as depth
is unknown. Thus, any successful method must implicitly
identify occlusions as outliers during estimation.

(B) Convolutional M-Estimator: Fig. 2 shows the operations and parameters of the Convolutional M-Estimator weuse in this paper. In the coarse-to-fine inverse compositional
refinement, we add one more input to the network which isthe predicted weight from the coarser level pyramid. At
each image pyramid, we bilinear upsample the weight matrix predicted from the coarse scale Win , and concatenate
it with Tθ , Iθ and r0 , which we use as input to the Convolutional M-estimator. The network predicts W at the current scale. Different from traditional M-estimators which
evaluate W at every step when rk is updated, we only compute W once for all following K iterations. This way, we
approximate the classical M-estimator and significantly re-

Implementation: Given pixel x = (x, y)T ∈ R2 , we define the warping function Wξ using the following parameterization (see also [1])
1 + ξ1ξ3
ξ5(1)
ξ2
1 + ξ4 ξ6
where ξ = (ξ1 , ξ2 , ξ3 , ξ4 , ξ5 , ξ6 )T . The analytic form of theJacobian in (1) is
x 0 y 0 1 0(2)
0 x 0 y 0 1
1Spatial-Average-Pooling

Spatial-Average-Pooling

Spatial-Average-Pooling[K:3, D:1, In: 32, Out: 32][K:3, D:1, In: 64, Out: 64][K:3, D:1, In: 96, Out: 96][K:3, D:2, In: 16, Out: 32][K:3, D:2, In: 32, Out: 64][K:3, D:2, In: 64, Out: 96][K:3, D:2, In: 128, Out: 128]

[K:3, D:2, In: 32, Out: 32][K:3, D:2, In: 64, Out: 64][K:3, D:2, In: 96, Out: 96][K:3, D:2, In: 128, Out: 128]

Channel-wise Sum

Channel-wise Sum

Channel-wise Sum

Channel-wise Sum[K:3, D:1, In: 8, Out: 16]

Input Dim:
Bx8xHxW
Pyramid 1
Output Dim:
Bx1xHxW

Pyramid 2
Output Dim:
Bx1xH/2xW/2Pyramid 3
Output Dim:
Bx1xH/4xW/4Pyramid 4
Output Dim:
Bx1xH/8xW/8Figure 1: (A) Two-view Feature Encoder. We use [K, D, In, Out] as abbreviation for [Kernel size, Dilation, Input channel
size, Output channel size]. All convolutional layers are followed by a BatchNorm layer and a ReLU layer. We use [B,H,W]
as abbreviation for the feature size [Batch size, Height of feature, Width of feature]. We use spatial average pooling of size 2
to downsample features between two feature pyramids. We channel-wise sum the output features of the encoder at each scale
to obtain the resulting feature maps.

𝐓"

𝐈"

𝐫(

[K:3, D:1, In: 4, Out: 16]

Bilinear Up-sample

𝐖[K:3, D:2, In: 16, Out: 32][K:3, D:4, In: 32, Out: 64][K:3, D:1, In: 64, Out: 1]

𝐖%&

Sigmoid

Figure 2: (B) Convolutional M-Estimator. We use [K, D, In, Out] as abbreviation for [Kernel size, Dilation, Input channel
size, Output channel size]. All convolutional layers are followed by a BatchNorm layer and a ReLU layer. In our default
weight-sharing setting, all weights are shared across networks in different pyramids. At the coarsest image level which does
not require the up-sampled W as input, we set Win to 1.
Dim: Bx(6x6+ 6xN)⋯

Flattened andConcatenate

Linear + ReLU [Out: 128]

Linear + ReLU [Out: 256]

Linear + ReLU [Out: 6]

Dim: Bx6x6

𝜆

Dim: Bx6

Dim: Bx6xN

Figure 3: (C) Trust Region Network. We use B as abbreviation for Batch size. N indicates the number of damping
proposals. In the weight-sharing setting, all weights are shared across networks at different pyramid levels. We use ’Linear’
to represent a fully connected layer. The last ReLU layer ensures that the output λ is non-negative.

2

The composition of ξ ◦ ∆ξ is given by
ξ1 + ∆ξ1 + ξ1 ∆ξ1 + ξ3 ∆ξ2ξ2 + ∆ξ2 + ξ2 ∆ξ1 + ξ4 ∆ξ2 ξ3 + ∆ξ3 + ξ1 ∆ξ3 + ξ3 ∆ξ4 ξ4 + ∆ξ4 + ξ2 ∆ξ3 + ξ4 ∆ξ4 ξ5 + ∆ξ5 + ξ1 ∆ξ5 + ξ3 ∆ξ6 
ξ6 + ∆ξ6 + ξ2 ∆ξ5 + ξ4 ∆ξ6The parameters of the inverse affine ξ −1 in (1) are−ξ1 − ξ1 ξ4 + ξ2 ξ3


−ξ2




1
−ξ
3



−ξ
−
ξ
ξ
+
ξ
ξ
(1 + ξ1 )(1 + ξ4 ) − ξ2 ξ3 
1 4
2 3
 4
−ξ5 − ξ4 ξ5 + ξ3 ξ6 
−ξ6 − ξ1 ξ6 + ξ2 ξ5

L1 error
No Learning
DeepLK, adapted from [6]Ours: (A)Ours: (A)+(B)
Ours: (A)+(B)+(C)

(3)

0.219
0.158
0.151
0.132
0.071

Table 1: Quantitative evaluation on the test set using L1
error wrt. the estimated 2D affine transform.

(4)

Sec.1. We set W to the identity matrix when the Convolutional M-Estimator (B) is not used and use Gauss-Newton
optimization in the absence of the Trust Region Network(C). We consider the following configurations:

Datasets: We download 1800 natural images from flickr
and use them to generate a 2D dataset for training affine 2D
transformation estimation. We use the downloaded flickr
images as T. Given a random affine transform which induces a 2D warping field, we synthesize the template I by
applying the warping field to T using bilinear interpolation.
To remove the boundary effects caused by zero padding in
the warping process, we crop a region of size 240x320 fromthe central region of both I and T. According to the actual
raw image size and the coordinates of the cropped region,
we adjust the warping field to ensure the cropped region has
the correct affine transform. Fig. 4 shows examples of generated pairs T and I.
For each template image, we randomly generate six different affine transforms, and synthesize six different images
I using the warping field induced from each affine transform. The generated affine transforms are used as groundtruth for training and evaluation. We use five out of the six
generated affine transforms and their corresponding image
pairs as the training set, and use the rest for testing.

• Ours (A)+(B)+(C): Our proposed method with sharedweights. We perform coarse-to-fine iterations on threepyramid levels with three IC iterations at each level. We
use shared weights for all iterations in (B) and (C).
• DeepLK-6DoF We implemented a variant of DeepLK[6] which predicts the affine transform instead of translation and scale as in the original 2D task. We use GaussNewton as the default optimization for this approach andno Convolutional M-Estimator. Comparing this approachwith our method when using only the two-view feature
network (A) demonstrates the utility of our two-view feature encoder.
Training: During training, we minimizes the L1 norm of
the distance from our estimated affine transform ξ ? to theground truth affine transform ξ GT . We train our method andall baselines using a learning rate of 0.005.
Results: Table 1 shows a quantitative evaluation of our
2D affine motion estimation experiments. We evaluate theerror in L1 norm by comparing the estimated results to theground truth. Compared to all baseline methods, our model((A)+(B)+(C)) yields the most accurate solution.

Baselines: We follow the experiment settings in the main
paper. We use (A), (B), (C) to refer to our contributions in

Note that different from the 3D motion estimation experiments in the main paper, there exists no motion ambiguity in the datasets used for 2D affine transform, rendering this setting simpler. We observe small improvements
when using our two-view feature encoder (A) compared tousing only a single view [6]. Using (B) Convolutional Mestimator gives better performance which may potentiallyaddress outliers induced by occlusions. Using (C) Trust
Region Network further helps the network to boost performance. Similarly to our results in the main paper, we observe that we obtain the most accurate results when combining all components ((A)+(B)+(C)) of our model.

T

I

Figure 4: Examples of T and I used for our 2D affine transformation estimation experiments.
33. Implementation Details of Jacobian
Given pixel x ∈ R2 , camera intrinsic K and depth D(x),we define the corresponding 3D point p = (px , py , pz ) as
p = D(x)K−1 x. At each iteration, we apply an exponential map to the output of the network ξ ∈ se(3) to obtainthe transformation matrix Tξ = exp([ξ]× ). Suppose K
is the intrinsic matrix for a pin-hole camera without distortion, which can be parameterized as [fx , fy , cx , cy ] with
fx , fy as its focal length and cx , cy as its offset along the twoaxes. The Jacobian J of template image T(0) with respect
to warp parameter ξ is given by∂W
J = ∇T
∂ξ
h
i
∂T(0)
∂T(0)
∇T = ∂u
∂v

 
px py
∂W
fx  − p2z
=
·
p2
fy
∂ξ
−1 − y
p2z

(5)
(6)
p2

1 + px2

− pyz

p

1
pz

0

− ppx2



p p

px
pz

0

1
pz

− py2

p



y

− xp2 yz(a) aeroplane(b) boat(c) bus(d) car(e) motorbike(f) bicycle

Figure 5: Example images from the MovingObjects3D
dataset. We use ’boat’ and ’motorbike’ categories as test
set and the others as train/validation set.

zz

’fr2/pioneer 360’. From the ablation, we also observe that
adding the trust-region module (C) helps to stabilize training and yields the most significant improvement in test accuracy. One possible reason for this is that the trust-region
module can adjust the damping parameters and adapt to awide range of motion magnitudes in the data. At training time, adaptive damping helps to stabilize the training
loss and potentially contributes to learning better featuresin modules (A) and (B). At inference time, the method canadjust its trust-region step to adapt to different motion magnitudes with a fixed number of iterations.

(7)
where · is the element-wise product along each row. ∇T
is the image gradient of template T(0). In the proposedmodel, we compute the Jacobian of Tθ from the output of
network (A) by substituting T with Tθ in (5).
We further simplify (7) by exploiting the inverse
depth parameterization p = (pu /pd , pv /pd , 1/pd ) where(pu , pv ) ∈ R2 is the pixel coordinate and pd ∈ R1 is theinverse depth. We obtain:
 

∂W
f
−pu pv 1 + p2u −pv pd 0 −pd pu= x ·pu
0 pd −pd pv
fy −1 − p2v pu pv∂ξ(8)
For more details we refer the reader to [2].

5. Details of MovingObjects3D
Background Set-up: We use a cubic box of size 6 × 6 × 6
as the background layout. We download 150 backgroundtextures from TextureNinja1 and use them as the textures forthe inner side of the cube. We randomly locate the camera
inside the cube with zero elevation. We randomly place four
point light sources at four different directions.

4. Details of TUM RGB-D Experiments
Our evaluation split of the TUM RGB-D SLAM dataset[4] consists of four trajectories of different conditions, trajectory length and motion magnitudes. After synchronization of the color image, depth and the ground truth trajectory, we obtain 750 frames in ’fr1/360’, 584 frames in’fr1/desk’, 2203 frames in ’fr2/desk’ and 830 frames in’fr2/pioneer 360’. To better understand our method operating under different conditions, we present a detailed quantitative evaluation in Table 2 using the Relative Pose Error(RPE) and the 3D End Point Error (3D EPE) metrics foreach trajectory.

Object Trajectories: For each trajectory, we randomly
generate keyframe poses (we use one keyframe every ten
frames). We calculate poses between keyframes using thedefault trajectory interpolation in Blender. We randomly locate the objects within a 1.5m radius ball around the centerof the cube which ensures that the objects always move inside of the box. There is a significant number of views forwhich only parts of the object is visible. We exclude allframes where the entire object is outside the field of view of
the camera.

Discussion: Our method with all modules (A)+(B)+(C)outperforms the baseline RGBD visual odometry [3] acrossall subsampled trajectories, except in ’fr1/desk’ of keyframe[1,2] and ’fr2/desk’ of keyframe 2 in which the performanceis close to each other. Our method shows clear advantages when the motion magnitude is large, e.g. ’fr1/360’,

Rendered Images and Ground Truth: We render the images, depth maps and instance segmentation masks using
1 https://texture.ninja/

4

KF 1

mRPE: θ (Deg) ↓ / t (cm) ↓
KF 2
KF 4

KF 8

KF 1

3D EPE (cm)
KF 2 KF 4

KF 8

16.08/17.32
14.33/16.26
14.23/17.52
7.24/6.64

1.33
1.24
1.18
1.05

5.98
2.39
1.92
1.21

21.34
13.37
10.67
2.64

52.50
49.60
49.13
22.40

10.71/19.83
7.14/11.27
6.59/10.87
4.30/7.11

0.59
0.74
0.74
0.68

0.91
1.31
1.28
0.96

5.46
3.89
3.42
1.74

18.84
12.89
11.21
6.11

1.44/3.80
1.46/4.30
1.43/4.14
1.11/2.79

0.96
0.80
0.80
0.73

0.93
1.04
1.04
0.94

2.20
1.60
1.62
1.29

4.75
2.93
2.83
2.09

8.54/14.36
7.60/14.83
6.91/13.31
2.38/6.85

6.39
3.55
3.57
2.76

9.22
5.67
5.42
3.15

21.16
13.78
12.77
4.46

48.46
39.41
36.80
13.52

2.31
1.58
1.56
1.31

4.38
2.60
2.41
1.57

12.67
8.15
7.10
2.53

31.13
26.20
24.69
11.03

1.80
1.39
1.38
1.16

3.53
2.18
2.05
1.41

9.58
6.22
5.52
2.18

23.14
19.16
18.31
8.28

fr1/360
RGBD VO [3]Ours: (A)Ours: (A)+(B)
Ours: (A)+(B)+(C)0.46/1.03
0.50/1.33
0.45/1.18
0.33/0.61

2.45/5.26
1.32/3.84
1.00/3.18
0.49/1.20

7.47/10.31
5.68/11.79
4.96/11.62
2.64/2.63
fr1/desk

RGBD VO [3]Ours: (A)Ours: (A)+(B)
Ours: (A)+(B)+(C)0.43/0.69
0.58/1.04
0.57/1.02
0.55/0.90

0.76/1.04
1.12/2.05
1.08/2.03
0.89/1.55

3.52/5.15
2.75/4.87
2.54/4.63
1.58/2.59
fr2/desk

RGBD VO [3]Ours: (A)Ours: (A)+(B)
Ours: (A)+(B)+(C)0.30/0.56
0.30/0.54
0.31/0.56
0.27/0.42

0.35/0.72
0.44/0.98
0.45/1.03
0.39/0.73

0.79/1.78
0.72/1.94
0.77/2.04
0.61/1.37
fr2/pioneer 360

RGBD VO [3]Ours: (A)Ours: (A)+(B)
Ours: (A)+(B)+(C)1.02/1.82
0.76/1.77
0.72/1.80
0.65/0.85

2.00/4.19
1.04/3.66
0.95/3.52
0.74/1.07

4.20/6.51
2.34/8.16
2.15/6.96
0.98/1.79

Average over trajectories mRPE and EPE
RGBD VO [3]Ours: (A)Ours: (A)+(B)
Ours: (A)+(B)+(C)0.55/1.03
0.53/1.17
0.51/1.14
0.45/0.69

1.39/2.81
0.97/2.63
0.87/2.44
0.63/1.14

3.99/5.95
2.87/6.89
2.60/6.56
1.10/2.09

9.20/13.83
7.63/12.16
7.30/11.21
3.76/5.88

Average over frames mRPE and EPE
RGBD VO [3]Ours: (A)Ours: (A)+(B)
Ours: (A)+(B)+(C)0.48/0.90
0.46/0.98
0.45/0.96
0.39/0.59

1.08/2.20
0.79/2.12
0.73/2.00
0.54/0.98

2.95/4.60
2.16/5.35
1.99/5.15
0.91/1.83

6.54/10.26
5.58/9.65
5.35/9.41
2.82/4.80

Table 2: Detailed Results on TUM RGB-D Dataset [45]. This table shows the mean relative pose error (mRPE) on our test
split of the TUM RGB-D Dataset [45]. KF denotes the size of the key frame intervals.
Blender. We show examples of objects for all six categories
in Fig. 5. We use images rendered using the categories’boat’ and ’motorbike’ as test set and images from the categories ’aeroplane’, ’bicycle’, ’bus’, ’car’ as the training set.
We will make the dataset creation tool and the dataset itself
public upon publication.

show a qualitative visualization of our method at different
iterations. We warp the image I using the iterative estimatedposes at the four coarse-to-fine pyramid scales (I(ξ ?1 ) →I(ξ ?2 ) → I(ξ ?3 ) → I(ξ ?final ) ). I(ξ ?final ) is the final output
of our method. Our results demonstrate that the proposed
method is able to iteratively refine the estimation towards
the global optimal solution despite the challenging scenario.

6. Qualitative Visualizations

Comparison to Baselines: Fig. 8 shows a comparisonof our full method to DeepLK 6DoF [6] and ablated models using only parts of the proposed modules. Our results demonstrate that the combination of all modules yields
high-quality registrations.

We now demonstrate additional qualitative results on 3D
rigid transformation estimation on MovingObjects3D. Notethe difficulty of the task (truncation, independent background object) and the high quality of our alignments.
Visualizations of Iterative Estimation: Fig. 6 and Fig. 7
5

T

I

I(ξ ?1 )I(ξ ?2 )I(ξ ?3 )I(ξ ?final )I(ξ GT )Figure 6: Qualitative Results on MovingObjects3D. Visualization of the warped image I(ξ) using the ground truth object
motion ξ GT (last row) and the object motion ξ ? estimated using our method at each pyramid (ξ ?1 , ξ ?2 , ξ ?3 , ξ ?final ) on theMovingObjects3D validation and test set. In I, we plot the instance boundary in red and that of T in green as comparison.
Note the difficulty of the task (truncation, independent background object) and the high quality of our alignments. Blackregions in the warped image are due to truncation or occlusion.
6

T

I

I(ξ ?1 )I(ξ ?2 )I(ξ ?3 )I(ξ ?final )I(ξ GT )Figure 7: Qualitative Results on MovingObjects3D. Visualization of the warped image I(ξ) using the ground truth object
motion ξ GT (last row) and the object motion ξ ? estimated using our method at each pyramid (ξ ?1 , ξ ?2 , ξ ?3 , ξ ?final ) on theMovingObjects3D validation and test set. In I, we plot the instance boundary in red and that of T in green as comparison.
Note the difficulty of the task (truncation, independent background object) and the high quality of our alignments. Blackregions in the warped image are due to truncation or occlusion.
7

T

I

I(ξ ? )DeepLK
6DoF [6]I(ξ ? )
Ours: (A)I(ξ ? )Ours:
(A)+(B)I(ξ ? )
Ours: (A)+
(B)+(C)I(ξ GT )Figure 8: Qualitative Comparisons of Our Method to Baselines on MovingObjects3D. We compared the object motion
ξ ? estimated using our proposed modules (A)+(B)+(C) ξ ? (row 6) to the optimal poses output from DeepLK 6DoF (row 3),
ours (A) (row 4) and ours (A)+(B) (row 5) on the MovingObjects3D validation and test set. We visualize the warped image
I(ξ) using the ground truth object motion ξ GT (last row). In I, we plot the instance boundary in red and that of T in green
for qualitative comparison of the two shapes in 2D.

8

References[1] S. Baker and I. Matthews. Lucas-kanade 20 years on: A unifying framework. Intl. J. of Computer Vision, 56(3):221–255,
Feb. 2004. 1[2] J.-L. Blanco. A tutorial on se(3) transformation parameterizations and on-manifold optimization. Technical report, University of Malaga, Sept. 2010. 4[3] F. Steinbrücker, J. Sturm, and D. Cremers. Real-time visual
odometry from dense rgb-d images. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, pages 719–722. IEEE, 2011. 4, 5
[4] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers. A benchmark for the evaluation of rgb-d slam systems.
In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems(IROS), Oct. 2012. 4[5] R. Szeliski. Image alignment and stitching: A tutorial. Foundations and Trends in Computer Graphics and Vision, 2(1):1–
104, 2007. 1[6] C. Wang, H. K. Galoogahi, C.-H. Lin, and S. Lucey. Deep-lk
for efficient adaptive object tracking. In IEEE Intl. Conf. on
Robotics and Automation (ICRA), 2018. 3, 5, 8

9