Taking a Deeper Look at the Inverse Compositional Algorithm
Zhaoyang Lv1,2 Frank Dellaert1 James M. Rehg1 Andreas Geiger2
1
Georgia Institute of Technology, Atlanta, United States
2
Autonomous Vision Group, MPI-IS and University of Tübingen, Germany{zhaoyang.lv, rehg}@gatech.edu

frank.dellaert@cc.gatech.edu

andreas.geiger@tue.mpg.de

denote the warped template and image, respectively1 , theLucas-Kanade objective can be stated as follows

Abstract
In this paper, we provide a modern synthesis of the classic inverse compositional algorithm for dense image alignment. We first discuss the assumptions made by this wellestablished technique, and subsequently propose to relax
these assumptions by incorporating data-driven priors into
this model. More specifically, we unroll a robust version of
the inverse compositional algorithm and replace multiple
components of this algorithm using more expressive models
whose parameters we train in an end-to-end fashion from
data. Our experiments on several challenging 3D rigid motion estimation tasks demonstrate the advantages of combining optimization with learning-based techniques, outperforming the classic inverse compositional algorithm aswell as data-driven image-to-pose regression approaches.

2

min kI(ξ) − T(0)k2(1)

ξ

where I(ξ) denotes image I transformed using warp parameters ξ and T(0) = T denotes the original template.
Minimizing (1) is a non-linear optimization task as theimage I depends non-linearly on the warp parameters ξ.
The Lucas-Kanade algorithm therefore iteratively solves forthe warp parameters ξ k+1 = ξ k ◦ ∆ξ. At every iteration k,
the warp increment ∆ξ is obtained by linearizing2

min kI(ξ k + ∆ξ) − T(0)k2

(2)

∆ξ

using first-order Taylor expansion
min I(ξ k ) +∆ξ

1. Introduction∂I(ξ k )∆ξ − T(0)
∂ξ

2

(3)
2

Note that the “steepest descent image” ∂I(ξ k )/∂ξ needs tobe recomputed at every iteration as it depends on ξ k .

Since the seminal work by Lucas and Kanade [32], dense
image alignment has become an ubiquitous tool in computer
vision with many applications including stereo reconstruction [41], tracking [5, 42, 50], image registration [8, 29, 43],
super-resolution [22] and SLAM [13, 16, 17]. In this paper, we provide a learning-based perspective on the Inverse
Compositional algorithm, an efficient variant of the original Lucas-Kanade image registration technique. In particular, we lift some of the restrictive assumptions by parameterizing several components of the algorithm using neuralnetworks and training the entire optimization process endto-end. In order to put contributions into context, we will
now briefly review the Lucas-Kanade algorithm, the Inverse
Compositional algorithm, as well as the robust M-Estimatorwhich form the basis for our model. More details can befound in the comprehensive reviews of Baker et al. [2, 3].

Inverse Compositional Algorithm: The inverse compositional (IC) algorithm [3] avoids this by applying the warp
increments ∆ξ to the template instead of the image
2

min kI(ξ k ) − T(∆ξ)k2

(4)

∆ξ

using the warp parameter update ξ k+1 = ξ k ◦ (∆ξ)−1 . In
the corresponding linearized equation∂T(0)min I(ξ k ) − T(0) −
∆ξ
∆ξ
∂ξ

2

(5)
2

∂T(0)/∂ξ does not depend on ξ k and can thus be precomputed, resulting in a more efficient algorithm.

Lucas-Kanade Algorithm: The Lucas-Kanade algorithm
minimizes the photometric error between a template and an
image. Letting T : Ξ → RW ×H and I : Ξ → RW ×H

1 The warping function W : R2 → R2 might represent translation,ξaffine 2D motion or (if depth is available) rigid or non-rigid 3D motion. To
avoid clutter in the notation, we do not make Wξ explicit in our equations.

1

Robust M-Estimation: To handle outliers or ambiguities(e.g., multiple motions), robust estimation [19, 54] can beused. The robust version of the IC algorithm [2] has thefollowing objective function
min rk (∆ξ)T W rk (∆ξ)∆ξ(A) We propose a Two-View Feature Encoder which replaces I, T with feature representations Iθ , Tθ thatjointly encode information about the input image I andthe template T. This allows our model to exploit spatial as well as temporal correlations in the data.

(6)(B) We propose a Convolutional M-Estimator that replaces W in (6) with a learned weight matrix Wθwhich encodes information about I, T and rk in a way
such that the unrolled optimization algorithm ignores
irrelevant or ambiguous information as well as outliers.

where rk (∆ξ) = I(ξ k ) − T(∆ξ) is the residual between
image I and template T at the k’th iteration, and W is adiagonal weight matrix that depends on the residual2 and ischosen based on the desired robust loss function ρ [54].
Optimization: The minimizer of (6) after linearization is
obtained as the Gauss-Newton update step [6](JT WJ)∆ξ = JT W rk (0)(C) We propose a Trust Region Network which replaces
the damping matrix λ diag(JT WJ) in (8) with alearned damping matrix diag(λθ ) whose diagonal entries λθ are estimated from “residual volumes” which
comprise residuals of a Levenberg-Marquardt update
when applying a range of hypothetical λ values.

(7)

where J = ∂T(0)/∂ξ is the Jacobian of the template T(0)with respect to the warp parameters ξ. As the approximate
Hessian JT WJ easily becomes ill-conditioned, a damping
term is added in practice. This results in the popular Levenberg–Marquardt (trust-region) update equation [35]:∆ξ = (JT WJ + λ diag(JT WJ))

−1 T

J W rk (0)We demonstrate the advantages of combining the classicalIC method with deep learning on the task of 3D rigid motion estimation using several challenging RGB-D datasets.
We also provide an extensive ablation study about the relevance of each model component that we propose. Resultson traditional affine 2D image alignment tasks are provided
in the supplementary material. Our implementation is publicly accessible.3(8)

For different values of λ, the parameter update ∆ξ varies
between the Gauss-Newton direction and gradient descent.
In practice, λ is chosen based on simple heuristics.
Limitations: Despite its widespread utility, the IC method
suffers from a number of important limitations. First, it assumes that the linearized residual leads to an update which
iteratively reaches a good local optimum. However, this assumption is invalid in the presence of high-frequency textural information or noise in I or T. Second, choosing a goodrobust loss function ρ is difficult as the true data/residual
distribution is often unknown. Moreover, Equation (6) does
not capture correlations or higher-order statistics in the inputs I and T as the residuals operate directly on the pixel
values and the weight matrix W is diagonal. Finally, damping heuristics do not fully exploit the information available
during optimization and thus lead to suboptimal solutions.

2. Related WorkWe are not the first to inject deep learning into an optimization pipeline. In this section, we first review classicalmethods, followed by direct pose regression techniques andrelated work on learning-based optimization.
Classical Methods: Direct methods [18, 21] that align images using the sum-of-square error objective (1) are prone tooutliers and varying illuminations. Classical approaches address this problem by exploiting more robust objective functions [32, 37], heuristically chosen patch- [47] or gradientbased [29] features, and photometric calibration as a preprocessing step [16]. The most common approach is to use
robust estimation (6) as in [7]. However, the selection of agood robust function ρ is challenging and traditional formulations assume that the same function applies to all pixels,
ignoring correlations in the inputs. Moreover, the inversionof the linearized system (7) may still be ill-conditioned [1].
To overcome this problem, soft constraints in the form of
damping terms (8) can be added to the objective [2,3]. However, this may bias the system to sub-optimal solutions.
This paper addresses these problems by relaxing themain assumptions of the Inverse Compositional (IC) algorithm [2, 3] using data-driven learning. More specifically,
we propose to learn the feature representation (A), robustContributions: In this paper, we propose to combine thebest of both (optimization and learning-based) worlds byunrolling the robust IC algorithm into a more general parameterized feed-forward model which is trained end-toend from data. In contrast to generic neural network estimators, this allows our algorithm to incorporate knowledge
about the structure of the problem (e.g., family of warping functions, 3D geometry) as well as the advantages ofa robust iterative estimation framework. At the same time,
our approach relaxes the restrictive assumptions made in theoriginal IC formulation [3] by incorporating trainable modules and learning the entire model end-to-end.
More specifically, we make the following contributions:
2 We omit this dependency to avoid clutter in the notation.

3 https://github.com/lvzhaoyang/DeeperInverseCompositionalAlgorithm

2estimator (B) and damping (C) jointly to replace the traditional heuristic rules of classical algorithms.

Besides, they neither learn input features (A) nor leveragetrust-region optimization (C). Instead, they solve their optimization problem using singular value decomposition.
Concurrent to our work, Tang and Tan [48] proposea photometric Bundle Adjustment network by learning toalign feature spaces for monocular reconstruction of a static
scene. Different from us, they did not exploit temporal correlation in the inputs (A) and do not employ a robust formulation (B). While they propose to learn the damping parameters (C), in contrast to our trust-region volume formulation, they regress the damping parameters from the global
average pooled residuals.

Direct Pose Regression: A notably different approach toclassical optimization techniques is to directly learn the entire mapping from the input to the warping parameters ξfrom large amounts of data, spanning from early work using
linear hyperplane approximation [23] to recent work using
deep neural networks [9, 49, 51, 53, 58]. Prominent examples include single image to camera pose regression [25,26],
image-based 3D object pose estimation [34,46] and relativepose prediction from two views [9, 49]. However, learninga direct mapping requires high-capacity models and large
amounts of training data. Furthermore, obtaining pixelaccurate registrations remains difficult and the learned representations do not generalize well to new domains.
To improve accuracy, recent methods adopt cascaded
networks [20,52] and iterative feedback [24]. Lin et al. [31]combines the multi-step iterative spatial transformer network (STN) with the classical IC algorithm [2, 3] for aligning 2D images. Variants of this approach have recently been
applied to various 3D tasks: Li et al. [30] proposes to iteratively align a 3D CAD model to an image. Zhou et al. [56]jointly train for depth, pose and optical flow.
Different from [31] and its variants which approximate
the pseudo-inverse of the Jacobian implicitly using stacked
convolutional layers, we exploit the structure of the optimization problem and explicitly solve the original robust
objective (6) with learned modules using few parameters.

3. Method
This section describes our model. A high-level overview
over the proposed unrolled inverse-compositional algorithm
is given in Fig. 1. Using the same notation as in Section 1,
our goal is to minimize the error by warping the image towards the template, similar to (6):min r(ξ)T Wθ r(ξ)(9)ξ

The difference to (6) is that in our formulation, the weightmatrix Wθ as well as the template Tθ (ξ) and the imageIθ (ξ) (and thus also the residual r(ξ) = Iθ (ξ) − Tθ (0))
depend on the parameters of a learned model θ.
We exploit the inverse compositional algorithm to solve
the non-linear optimization problem in (9), i.e., we linearize
the objective and iteratively update the warp parameters ξ:

Learning-based Optimization: Recently, several methods have exploited the differentiable nature of iterative optimization algorithms by unrolling for a fixed number of iterations. Each iteration is treated as a layer in a neural network [12, 28, 36, 39, 40, 55]. In this section, we focus on
the most related work which also tackles the least-squares
optimization problem [13, 38, 48, 51]. We remark that mostof these techniques can be considered special cases of ourmore general deep IC framework.
Wang et al. [51] address the 2D image tracking problem by learning an input representation using a two-stream
Siamese network for the IC setting. In contrast to us, they
exploit only spatial but not temporal correlations in the inputs (A), leverage a formulation which is not robust (B) anddo not exploit trust-region optimization (C).
Clark et al. [13] propose to jointly learn depth and pose
estimation by minimizing photometric error using the formulation in (4). In contrast to us, they do not learn feature
representations (A) and neither employ a robust formulation(B) nor trust-region optimization (C).
Ranftl et al. [38] propose to learn robust weights forsparse feature correspondences and apply their model tofundamental matrix estimation. As they do not target direct
image alignment, their problem setup is different from ours.

ξ k+1

=

ξ k ◦ (∆ξ)−1

(10)
−1 T

∆ξ

=

(JT Wθ J + diag(λθ ))rk=Iθ (ξ k ) − Tθ (0)(12)∂Tθ (0)/∂ξ

(13)

J =

J Wθ rk(11)

starting from ξ 0 = 0.
The most notable change from (8) is that the image features (Iθ , Tθ ), the weight matrix (Wθ ) and the damping
factors λθ are predicted by learned functions which havebeen collectively parameterized by θ = {θI , θW , θλ }. Notethat also the residual rk as well as the Jacobian J implicitly
depend on the parameters θ, though we have omitted thisdependence here for notational clarity.
We will now provide details about these mappings.
(A) Two-View Feature Encoder: We use a fully convolutional neural network φθ to extract feature maps from theimage I and the template T:Iθ=φθ ([I, T])

(14)

Tθ

=

φθ ([T, I])(15)

Here, the operator [·, ·] indicates concatenation along thefeature dimension. Instead of I and T we then feed Iθ
3

[", $]
Feature Pyramid(A) Two-View
Feature Encoder(A) Two-View
Feature Encoder[$, "]"'K InverseComposition∑(A) Two-View
Feature Encoder"(

Pre-computation

"(

∑

$'

$'

Inverse-composition
at ;th iteration∑

K InverseComposition

$'

'

"'

∑

K InverseComposition

K InverseComposition$'(A) Two-View
Feature Encoder

Residual maps from damping proposals:("# (*)(,(.)-(.) → Δ1(.) → %2+1 , . ∈ {1 … N}

"#
−

%0

(B) Convolutional
M-estimator

−

%2

$#

(C) Trust Region
Network,:,2+1 = ,2 ∘ Δ, −>Δ,-?

Figure 1: High-level Overview of our Deep Inverse Compositional (IC) Algorithm. We stack [I, T] and [T, I] as inputs
to our (A) Two-View Feature Encoder pyramid which extracts 1-channel feature maps Tθ and Iθ at multiple scales using
channel-wise summation. We then perform K IC steps at each scale using Tθ and Iθ as input. At the beginning of each scale,
we pre-compute W using our (B) Convolutional M-estimator. For each of the K IC iterations, we compute the warped(i)
image I(ξ k ) and rk . Subsequently, we sample N damping proposals λ(i) and compute the proposed residual maps rk+1 . Our(C) Trust Region Network takes these residual maps as input and predicts λθ for the trust region update step.
and Tθ to the residuals in (12) and to the Jacobian in (13).
Note that we use the notation Iθ (ξ k ) in (12) to denote thatthe feature map Iθ is warped by a warping function that isparameterized via ξ. More formally, this can be stated asIθ (ξ) = Iθ (Wξ (x)), where x ∈ R2 denote pixel locationsand Wξ : R2 → R2 is a warping function that maps a pixel
location to another pixel location. For instance, Wξ may
represent the space of 2D translations or affine transformations. In our experiments, we will focus on challenging 3D
rigid body motions, using RGB-D inputs and representing
ξ as an element of the Euclidean group ξ ∈ se(3).
Note that compared to directly using the image I and T
as input, our features capture high-order spatial correlationsin the data, depending on the receptive field size of the convolutional network. Moreover, they also capture temporal
information as they operate on both I and T as input.

determined by a fully convolutional network ψθ that operates on the feature maps and the residual:Wθ = diag(ψθ (Iθ (ξ k ), Tθ (0), rk ))(16)

as is illustrated in Fig. 1. Note that this enables our algorithm to reason about relevant image information while
capturing spatial-temporal correlations in the inputs which
is not possible with classical M-Estimators. Furthermore,
we do not restrict the implicit robust function ρ to a particular error model, but instead condition ρ itself on the input.
This allows for learning more expressive noise models.
(C) Trust Region Network: For estimating the damping
λθ we use a fully-connected network as illustrated in Fig. 1.
We first sample a set of scalar damping proposals λi on
a logarithmic scale and compute the resulting LevenbergMarquardt update step as(B) Convolutional M-Estimator: We parameterize theweight matrix Wθ as a diagonal matrix whose elements are−1 T∆ξ i = (JT WJ + λi diag(JT WJ))4

J W rk (0) (17)We stack the resulting N residual maps(i)

rk+1 = Iθ (ξ k ◦ (∆ξ i )−1 ) − Tθ (0)Training Objective: To balance the influences of translation and rotation we follow [30] and exploit the 3D EndPoint-Error (EPE) as loss function. Let p = D(x)K−1 x
denote the 3D point corresponding to pixel x in image I
and let P denote the set of all such 3D points. We minimize
the following loss function
1 XX
2
kTgt p − T(ξ l ) pk2
(21)
L=
|P|

(18)

into a single feature map, flatten it, and pass it to a fully
connected neural network νθ that outputs the damping parameters λθ :

h
i
(1)
(N )
λθ = νθ JT WJ, JT Wrk+1 , . . . , JT Wrk+1(19)l∈L p∈P

where L denotes the set of coarse-to-fine pyramid levels (weapply our loss at the final iteration of every pyramid level)
and Tgt is the ground truth transformation.

The intuition behind our trust region networks is that theresiduals predicted using the Levenberg-Marquardt update
comprise valuable information about the damping parameter itself. This is empirically confirmed by our experiments.

Implementation: We use the Sobel operator to compute
the gradients in T and analytically derive J. We calculate
the matrix inverse on the CPU since we observed that inverting a small dense matrices H ∈ R6×6 is significantly fasteron the CPU than on the GPU. In all our experiments we
use N = 10 damping proposals for our Trust Region Network, sampled uniformly in logscale between [10−5 , 105 ].
We use four coarse-to-fine pyramid levels with three iterations each. We implemented our model and the baselines in
PyTorch. All experiments start with a fixed learning rate of
0.0005 using ADAM [27]. We train a total of 30 epochs,
reducing the learning rate at epoch [5,10,15].

Coarse-to-Fine Estimation: To handle large motions, it iscommon practice to apply direct methods in a coarse-to-fine
fashion. We apply our algorithm at four pyramid levels withthree iterations each. Our entire model including coarse-tofine estimation is illustrated in Fig. 1. We extract features at
all four scales using a single convolutional neural networkwith spatial average pooling between pyramid levels. We
start with ξ = 0 at the coarsest pyramid level, perform 3
iterations of our deep IC algorithm, and proceed with thenext level until we reach the original image resolution.

4.1. Datasets

Training and Inference: For training and inference, we
unroll the iterative algorithm in equations (10)-(13). We obtain the gradients of the resulting computation graph using
auto-differentiation. Details about the network architectures
which we use can be found in the supplementary material.

We systematically train and evaluate our method on four
datasets which we now briefly describe.

Using the warped coordinates, compute Iθ (ξ) via bilinear
sampling from Iθ and set the warped feature value to zerofor all occluded areas (estimated via z-buffering).

MovingObjects3D: For the purpose of systematically evaluating highly varying object motions, we downloaded six
categories of 3D models from ShapeNet [10]. For each object category, we rendered 200 video sequences with 100
frames in each sequence using Blender. We use data rendered from the categories ’boat’ and ’motorbike’ as test set
and data from categories ’aeroplane’, ’bicycle’, ’bus’, ’car’
as training set. From the training set we use the first 95%of the videos for training and the remaining 5% for validation. In total, we obtain 75K images for training, 5K images
for validation, and 25K for testing. We further subsample
the sequences using sampling intervals {1, 2, 4} in order toobtain small, medium and large motion subsets.
For each rendered sequence, we randomly select one 3D
object model within the chosen category and stage it in astatic 3D cuboid room with random wall textures and four
point light sources. We randomly choose the camera viewpoint, point light source position and object trajectory, see
supplementary material for details. This ensures diversity inobject motions, textures and illumination. The videos also
contain frames where the object is only partially visible. We
exclude all frames where the entire object is not visible.

4 Additional experiments on classical affine 2D motion estimation tasks
can be found in the supplementary material.

BundleFusion: To evaluate camera motion estimation ina static environment, we use the eight publicly released

4. Experiments
We perform our experiments on the challenging taskof 3D rigid body motion estimation from RGB-D inputs4 .
Apart from the simple scenario of purely static scenes where
only the camera is moving, we also consider scenes where
both the camera as well as objects are in motion, hence resulting in strong ambiguities. We cast this as a supervised
learning problem: using the ground truth motion, we train
the models to resolve these ambiguities by learning to focus
either on the foreground or the background.
Warping Function: Given pixel x ∈ R2 , camera intrinsics K and depth D(x), we define the warping Wξ (x) induced by rigid body transform Tξ with ξ ∈ se(3) asWξ (x) = K Tξ D(x)K−1 x

(20)

5

scenes from BundleFusion5 [14] which provide fully synchronized RGB-D sequences. We hold out ’copyroom’ and’office2’ scenes for test and split the remaining scenes intotraining (first 95% of each trajectory) and validation (last5%). We use the released camera trajectories as groundtruth. We subsampled frames at intervals {2, 4, 8} to increase motion magnitudes and hence the level of difficulty.

when increasing the number of iterations. For all our experiments, we thus used three iterations.
• Cascaded-PoseCNN: A cascaded network with three iterations, similar to IC-PoseCNN but with independent
weights for each iteration.
Learning-based Optimization: We implemented the following related algorithms within our deep IC framework.
For all methods, we use the same number of iterations,training loss and learning rate as used for our method.
• DeepLK-6DoF: We implemented a variant of DeepLK[50] which predicts the 3D transformation ξ ∈ se(3) instead of translation and scale prediction in their original
2D task. We use Gauss-Newton as the default optimization for this approach and no Convolutional M-Estimator.
A comparison of this approach with our method whenusing only the two-view feature network (A) shows thebenefits of our two-view feature encoder.
• IC-FC-LS-Net: We also implemented LS-Net [13]within our IC framework with the following differences
to the original paper. First, we do not estimate or refine depth. Second, we do not use a separate network
to provide an initial pose estimation. Third, we replace
their LSTM layers with three fully connected (FC) layers
which take the flattened JT WJ and JT Wrk as input.

DynamicBundleFusion: To further evaluate camera motion estimation under heavy occlusion and motion ambiguity, we use the DynamicBundleFusion dataset [33] whichaugments the scenes from BundleFusion with non-rigidly
moving human subjects as distractors. We use the same
training, validation and test split as above. We train andevaluate frames subsampled at intervals {1, 2, 5} due to theincreased difficulty of this task.
TUM RGB-D SLAM: We evaluate our camera motion estimates on the TUM RGB-D SLAM dataset [45]. We holdout ’fr1/360’, ’fr1/desk’, ’fr2/desk’ and ’fr2/pioneer 360’
for testing and split the remaining trajectories into training(first 95% of each trajectory) and validation (last 5%). We
randomly sample frame intervals from {1,2,4,8}. All images are resized to 160×120 pixels and depth values outside
the range [0.5m, 5.0m] are considered invalid.

4.2. Baselines

Ablation Study: We use (A), (B), (C) to refer to our contributions in Sec.1. We set W to the identity matrix when the
Convolutional M-Estimator (B) is not used and use GaussNewton optimization in the absence of the Trust Region
Network (C). We consider the following configurations:• Ours (A)+(B)+(C): Our proposed method with sharedweights. We perform coarse-to-fine iterations on four
pyramid levels with three IC iterations at each level. We
use shared weights for all iterations in (B) and (C).
• Ours (A)+(B)+(C) (No WS): A version of our method
without shared weights. All settings are the same asabove except that the network for (B) and (C) have independent weight parameters at each coarse-to-fine scale.
• Ours (A)+(B)+(C) (K iterations/scale): The same network as the default setting with shared weights, exceptthat we change the inner iteration number K.
• No Learning: Vanilla coarse-to-fine IC alignment minimizing photometric error (4) without learned modules.

We implemented the following baselines.
ICP: We use classical Point-to-Plane ICP [11] and Pointto-Point-ICP [4] implemented in Open3D [57]. To examine
the effect of ambiguity between the foreground and background in the object motion estimation task, we also evaluate a version for which we provide the ground truth instancesegmentation to both methods. Note that this is an upperbound to the performance achievable by ICP methods. We
thus call these the Oracle ICP methods.
RGB-D Visual Odometry: We compare to the RGB-D
visual odometry method [44] implemented in Open3D [57]on TUM RGBD SLAM datasets for visual odometry.
Direct Pose Regression: We compare three different variants that directly predict the mapping f : I, T → ξ. All
three networks use Conv1-6 encoder layers from FlowNetSimple [15] as two-view regression backbones. We use spatial average pooling after the last feature layer followed by afully-connected layer to regress ξ. All three CNN baselines
are trained using the loss in (21).
• PoseCNN: A feed-forward CNN that directly predicts ξ.
• IC-PoseCNN: A PoseCNN with iterative refinement using the IC algorithm, similar to [31] and [30]. We noticed
that training becomes unstable and performance saturates

4.3. Results and Discussion
Table 1 and Table 2 summarize our main results. For
each dataset, we evaluate the method separately for threedifferent motion magnitudes {Small, Medium, Large}. In
Table 1, {Small, Medium, Large} correspond to frames
sampled from the original videos at intervals {1, 2, 4}. In
Table 2, [Small, Medium, Large] correspond to frame intervals {2, 4, 8} on BundleFusion and {1, 2, 5} on DynamicBundleFusion. We show the following metrics/statistics:

5 http://graphics.stanford.edu/projects/bundlefusion/6Θ (Deg) ↓ / t (cm) ↓ / (t < 5 (cm) & Θ < 5◦ ) ↑ on Test

3D EPE (cm) ↓ on Validation/Test
Medium

Large

Small

MediumLarge

ICP

4.88/4.28
5.02/4.38
3.91/3.31
4.34/3.99

10.13/8.74
10.33/9.06
10.68/9.63
11.83/10.29

20.24/17.43
20.43/17.68
22.53/19.98
21.27/26.13

4.32/10.54/66.23%
4.04/10.51/70.0%
2.74/9.75/78.6%
3.81/10.11/75.0%

8.29/20.90/33.4%
7.89/20.15/33.8%
8.31/19.72/40.3%
9.30/26.41/39.1%

15.40/40.45/11.2%
15.33/40.52/11.1%
16.64/41.40/16.4%
19.53/62.30/14.1%

PoseCNN
IC-PoseCNN
Cascaded-PoseCNN

5.18/4.60
5.14/4.56
5.24/4.68

10.43/9.20
10.40/9.13
10.43/9.21

20.08/17.74
19.80/17.31
20.32/17.30

3.91/10.51/69.8%
3.93/10.49/70.1%
3.90/10.50/70.0%

7.89/21.10/34.7%
7.90/21.01/34.8%
7.90/21.11/34.6%

15.34/40.54/11.0%
15.31/40.51/11.1%
15.32/40.60/10.8%

No learning
IC-FC-LS-Net, adapted from [13]DeepLK-6DoF, adapted from [50]
Ours: (A)Ours: (A)+(B)
Ours: (A)+(B)+(C)Ours: (A)+(B)+(C) (No WS)
Ours: (A)+(B)+(C) (K = 1)
Ours: (A)+(B)+(C) (K = 5)

11.66/11.26
4.96/4.62
4.41/3.75
4.35/3.66
4.33/3.26
3.58/2.91
3.62/2.89
4.12/3.37
3.60/2.92

21.85/22.95
10.49/9.21
9.05/7.54
8.80/7.23
8.84/7.30
7.30/5.94
7.54/6.08
8.64/7.08
7.49/6.09

37.01/38.88
20.31/17.34
18.46/15.33
18.28/15.06
18.14/15.04
15.48/12.96
16.00/12.98
17.67/14.92
16.06/13.01

4.29/15.90/66.8%
3.94/10.49/70.0%
4.03/10.25/68.1%
4.09/10.19/68.6%
4.02/10.11/69.2%
3.74/9.73/74.5%
3.69/9.73/74.9%
3.85/9.95/71.2%
3.68/9.77/74.4%

8.33/31.80/32.2%
7.90/21.20/34.5%
7.96/20.34/33.6%
8.00/20.28/32.9%
7.96/20.26/33.4%
7.41/19.60/38.2%
7.37/19.74/38.4%
7.80/20.13/35.5%
7.46/19.69/37.8%

15.78/55.70/10.44%
15.33/40.63/11.2%
15.43/39.56/10.6%
15.37/39.49/10.9%
15.35/39.56/10.9%
14.71/38.39/12.9%
14.65/38.69/12.8%
15.21/39.30/11.6%
14.76/38.65/12.4%

Learning Optim

Small

Point-Plane ICP [11]Point-Point ICP [4]Oracle Point-Plane ICP [11]Oracle Point-Point ICP [4]DPR

Model Descriptions

Table 1: Quantitative Evaluation on MovingObjects3D. We evaluate the average 3D EPE, angular error in Θ (Euler angles),translation error t and success ratios t < 5 & Θ < 5◦ for three different motion magnitudes {Small, Medium, Large} whichcorrespond to frames sampled from the original videos using frame intervals {1, 2, 4}.
3D EPE (cm) ↓ Validation/Test
on BundleFusion [14]Inference
Time (ms)Small

Medium

Large

Small

Medium

Large

ICP

Model
Size (K)

Point-Plane ICP [11]Point-Point ICP [4]2.81/2.01
3.62/2.48

6.85/4.52
8.17/5.72

16.20/11.11
17.23/12.58

1.26/0.97
1.42/1.08

2.64/2.09
3.40/2.42

10.38/4.89
13.09/7.43

-

310
402

DPR

3D EPE (cm) ↓ Validation/Test
on DynamicBundleFusion [33]PoseCNN
IC-PoseCNN
Cascaded-PoseCNN

4.76/3.41
4.32/3.26
4.46/3.41

9.78/6.85
8.98/6.52
9.38/6.81

17.90/13.31
16.30/12.81
16.50/13.02

2.20/1.65
2.21/1.66
2.20/1.60

4.22/3.19
4.24/3.18
4.13/3.15

13.90/8.24
12.99/8.05
12.97/8.15

19544
19544
58632

5.7
14.1
14.1

Learning Optim

Model Descriptions

No learning
IC-FC-LS-Net, adapted from [13]DeepLK-6DoF, adapted from [50]
Ours: (A)Ours: (A)+(B)
Ours: (A)+(B)+(C)Ours: (A)+(B)+(C) (No WS)

4.52/3.35
4.04/3.03
4.09/2.99
3.59/2.65
2.42/1.75
2.27/1.48
2.14/1.52

8.64/6.30
9.06/6.85
8.14/5.84
7.68/5.46
5.39/3.47
5.11/3.09
5.15/3.10

17.06/12.51
17.15/13.32
16.81/12.27
16.56/11.92
12.59/8.40
12.16/7.84
12.26/7.81

4.89/3.39
2.33/1.80
2.15/1.72
2.01/1.65
2.22/1.70
1.09/0.74
0.93/0.61

5.64/4.69
4.43/3.45
3.78/3.12
3.68/2.96
3.61/2.97
2.15/1.54
1.86/1.32

13.88/8.58
13.84/8.35
12.73/7.22
12.68/7.11
12.57/6.88
9.78/4.64
8.88/3.82

674
596
597
622
662
883

1.5
1.7
2.2
2.2
2.4
7.6
7.6

Table 2: Quantitative Evaluation on BundleFusion and DynamicBundleFusion. In BundleFusion [14], the motion magnitudes {Small, Medium, Large} correspond to frame intervals {2, 4, 8}. In DynamicBundleFusion [33], the motion magnitudes {Small, Medium, Large} correspond to frame intervals {1, 2, 5} (we reduce the intervals due to the increased difficulty).

KF 1
RGBD VO [44]Ours: (A)Ours: (A)+(B)
Ours: (A)+(B)+(C)0.55/1.03
0.53/1.17
0.51/1.14
0.45/0.69

tion using the norm of Euler angles Θ, translation t in cm
and the success ratio (t < 5 (cm) & Θ < 5◦ ), in Table 1.
• Relative Pose Error: We follow the TUM RGBD metric[45] using relative axis angle θ and translation t in cm.
• Model Weight: The number of learnable parameters.
• Inference speed: The forward execution time for animage-pair of size 160 × 120, using GTX 1080 Ti.

mRPE: θ (Deg) ↓ / t (cm) ↓
KF 2
KF 4
KF 8
1.39/2.81
0.97/2.63
0.87/2.44
0.63/1.14

3.99/5.95
2.87/6.89
2.60/6.56
1.10/2.09

9.20/13.83
7.63/12.16
7.30/11.21
3.76/5.88

Table 3: Results on TUM RGB-D Dataset [45]. This tableshows the mean relative pose error (mRPE) on our test splitof the TUM RGB-D Dataset [45]. KF denotes the size ofthe key frame intervals. Please refer to the supplementarymaterials for a detailed evaluation of individual trajectories.

Comparison to baseline methods: Compared to all baseline methods (Table 1 row 1-10 and Table 2 row 1-8), ourfull model ((A)+(B)+(C)) achieves the overall best performance across different motion magnitudes and datasets
while maintaining fast inference speed. Compared to ICP
methods (ICP rows in Table 1 and Table 2) and classicalmethod (No Learning in Table 1 and Table 2), our method
achieves better performance without instance information
at runtime. Compared to RGB-D visual odometry [44],our method works particularly well in unseen scenarios and• 3D End-Point-Error (3D EPE): This metric is definedin (21). We only evaluate errors on the rigidly moving
objects, i.e., the moving objects in MovingObjects3D andthe rigid background mask in DynamicBundleFusion.
• Object rotation and translation: We evaluate 3D rota7

T

I

I(ξ GT )I(ξ ? )

Figure 2: Qualitative Results on MovingObjects3D. Visualization of the warped image I(ξ) using the ground truth object
motion ξ GT (third row) and the object motion ξ ? estimated using our method (last row) on the MovingObjects3D validation(left two) and test sets (right three). In I, we plot the instance boundary in red and the instance boundary of T in green
as comparison. Note the difficulty of the task (truncation, independent background object) and the high quality of ouralignments. Black regions in the warped image are due to truncation or occlusion.
in the presence of large motions (Table 3). Besides, notethat our model can achieve better performance with a significantly smaller number of weight parameters compared
to direct image-to-pose regression (DRP rows in Table 1
and Table 2). Fig. 2 shows a qualitative comparison of
our method on the MovingObject3D dataset by visualizing
I(ξ). Note that our method achieves excellent two-view image alignments, despite large motion, heavy occlusion andvarying illumination of the scene.

era motion in dynamic scenes (Table 2 row 11 in DynamicBundleFusion). We find our trust region network (C) to behighly effective for learning accurate relative motion in thepresence of large variations in motion magnitude(Table 3).

5. Conclusion
We have taken a deeper look at the inverse compositionalalgorithm by rephrasing it as a neural network with three
trainable submodules which allows for relaxing some of themost significant restrictions of the original formulation. Experiments on the challenging task of relative rigid motion
estimation from two RGB-D frames demonstrate that our
method achieves more accurate results compared to bothclassical IC algorithms as well as data hungry direct imageto-pose regression techniques. Although our data-driven IC
method can better handle challenges in large object motions, heavy occlusions and varying illuminations, solving
those challenges in the wild remains a subject for future research. To extend the current method to real-world environments with complex entangled motions, possible future
directions include exploring multiple motion hypotheses,
multi-view constraints and various motion models which
will capture a broader family of computer vision tasks.

Ablation Discussion: Across all ablation variants anddatasets, our model achieves the best performance by combining all three proposed modules ((A)+(B)+(C)). This
demonstrates that all components are relevant for robust
learning-based optimization. Note that the influence of the
proposed modules may vary according to the properties ofthe specific task and dataset. For example, in the presenceof noisy depth estimates from real data, learning a robust Mestimator (B) (Table 2 row 10 in BundleFusion) provide significant improvements. In the presence of heavy occlusions
and motion ambiguities, learning the trust region step (C)helps to find better local optima which results in large improvements in our experiments, observed when estimating
object motion (Table 1 row 13) and when estimating cam8

References[16] J. Engel, V. Koltun, and D. Cremers. Direct sparse odometry. IEEE Trans. Pattern Anal. Machine Intell., 40:611–625,
2018. 1, 2
[17] J. Engel, T. Schöps, and D. Cremers. LSD-SLAM: Largescale direct monocular SLAM. In European Conf. on Computer Vision (ECCV), September 2014. 1[18] B. K. Horn and E. Weldon. Direct methods for recovering
motion. Intl. J. of Computer Vision, 2(1):51–76, 1988. 2[19] P. J. Huber. Robust estimation of a location parameter. The
Annals of Mathematical Statistics, 35(1):73–101, 03 1964. 2[20] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and
T. Brox. FlowNet 2.0: evolution of optical flow estimation
with deep networks. In IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2017. 3[21] M. Irani and P. Anandan. About direct methods. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV)Workshops, 2000. 2[22] M. Irani and S. Peleg. Improving resolution by image registration. Graphical Model and Image Processing (CVGIP),
53(3):231–239, 1991. 1[23] F. Jurie and M. Dhome. Hyperplane approximation for template matching. IEEE Transactions on Pattern Analysis andMachine Intelligence, 24(7):996–1000, 2002. 3[24] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik. Endto-end recovery of human shape and pose. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2018. 3[25] A. Kendall, M. Grimes, and R. Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization. In
Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2015. 3[26] A. Kendall, H. Martirosyan, S. Dasgupta, and P. Henry. Endto-end learning of geometry and context for deep stereo regression. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017. 3[27] D. P. Kingma and J. Ba. Adam: A method for stochastic
optimization. In Proc. of the International Conf. on Learning
Representations (ICLR), 2015. 5[28] E. Kobler, T. Klatzer, K. Hammernik, and T. Pock. Variational networks: Connecting variational methods and deeplearning. In Proc. of the German Conference on Pattern
Recognition (GCPR), 2017. 3[29] A. Levin, A. Zomet, S. Peleg, and Y. Weiss. Seamless image
stitching in the gradient domain. In Proc. of the European
Conf. on Computer Vision (ECCV), 2004. 1, 2[30] Y. Li, G. Wang, X. Ji, Y. Xiang, and D. Fox. Deepim: Deep
iterative matching for 6d pose estimation. In Proc. of theEuropean Conf. on Computer Vision (ECCV), 2018. 3, 5, 6
[31] C.-H. Lin and S. Lucey. Inverse compositional spatial transformer networks. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2017. 3, 6[32] B. D. Lucas and T. Kanade. An iterative image registration
technique with an application to stereo vision. In Proc. of theInternational Joint Conf. on Artificial Intelligence (IJCAI),
1981. 1, 2
[33] Z. Lv, K. Kim, A. Troccoli, D. Sun, J. Rehg, and J. Kautz.
Learning rigidity in dynamic scenes with a moving camera[1] P. Anandan. A computational framework and an algorithm
for the measurement of visual motion. International Journalof Computer Vision (IJCV), 2(3):283–310, 1989. 2[2] S. Baker, R. Gross, I. Matthews, and T. Ishikawa. Lucaskanade 20 years on: A unifying framework: Part 2. Technicalreport, Carnegie Mellon University, 2003. 1, 2, 3
[3] S. Baker and I. Matthews. Lucas-kanade 20 years on: A unifying framework: Part 1. Technical report, Carnegie Mellon
University, Pittsburgh, PA, July 2002. 1, 2, 3
[4] P. J. Besl and N. D. McKay. A method for registrationof 3-d shapes. IEEE Trans. Pattern Anal. Machine Intell.,14(2):239–256, Feb. 1992. 6, 7
[5] A. Bibi, M. Mueller, and B. Ghanem. Target response adaptation for correlation filter tracking. In Proc. of the European
Conf. on Computer Vision (ECCV), 2016. 1[6] A. Björck. Numerical Methods for Least Squares Problems.
Siam Philadelphia, 1996. 2[7] M. J. Black and P. Anandan. The robust estimation ofmultiple motions: Parametric and piecewise-smooth flow
fields. Computer Vision and Image Understanding (CVIU),
63(1):75–104, 1996. 2[8] Brown, Matthew, Lowe, and David. Automatic panoramic
image stitching using invariant features. International Journal of Computer Vision (IJCV), 74(1):59–73, August 2007.
1[9] A. Byravan and D. Fox. SE3-Nets: Learning rigid body motion using deep neural networks. In Proc. IEEE International
Conf. on Robotics and Automation (ICRA), pages 173–180.
IEEE, 2017. 3[10] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan,
Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su,
J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rich
3D Model Repository. Technical Report arXiv:1512.03012[cs.GR], Stanford University — Princeton University —Toyota Technological Institute at Chicago, 2015. 5[11] Y. Chen and G. Medioni. Object modelling by registrationof multiple range images. Image and Vision Computing,10(3):145–155, 1992. 6, 7
[12] I. Cherabier, J. Schönberger, M. Oswald, M. Pollefeys, and
A. Geiger. Learning priors for semantic 3d reconstruction.
In Proc. of the European Conf. on Computer Vision (ECCV),
2018. 3[13] R. Clark, M. Bloesch, J. Czarnowski, S. Leutenegger, andA. J. Davison. Learning to solve nonlinear least squares formonocular stereo. In European Conf. on Computer Vision(ECCV), Sept. 2018. 1, 3, 6, 7
[14] A. Dai, M. Nießner, M. Zollöfer, S. Izadi, and C. Theobalt.
BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface re-integration. ACM Transactions on Graphics 2017 (TOG), 2017. 6, 7
[15] A. Dosovitskiy, P. Fischer, E. Ilg, P. Haeusser, C. Hazirbas,
V. Golkov, P. v.d. Smagt, D. Cremers, and T. Brox. Flownet:
Learning optical flow with convolutional networks. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV),
2015. 6

9

network for learning monocular stereo. In Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2017.
3[50] C. Wang, H. K. Galoogahi, C.-H. Lin, and S. Lucey. Deep-lk
for efficient adaptive object tracking. In IEEE Intl. Conf. on
Robotics and Automation (ICRA), 2018. 1, 6, 7
[51] C. Wang, J. Miguel Buenaposada, R. Zhu, and S. Lucey.
Learning depth from monocular videos using direct methods. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), June 2018. 3[52] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh. Convolutional pose machines. In IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), pages 4724–4732, 2016. 3[53] Z. Yin and J. Shi. GeoNet: Unsupervised Learning of Dense
Depth, Optical Flow and Camera Pose. In Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2018.
3[54] Z. Zhang, P. Robotique, and P. Robotvis. Parameter estimation techniques: a tutorial with application to conic fitting.
Image and Vision Computing (IVC), 15:59–76, 1997. 2[55] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet,
Z. Su, D. Du, C. Huang, and P. H. S. Torr. Conditional random fields as recurrent neural networks. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2015. 3[56] H. Zhou, B. Ummenhofer, and T. Brox. Deeptam: Deep
tracking and mapping. In Proc. of the European Conf. on
Computer Vision (ECCV), 2018. 3[57] Q.-Y. Zhou, J. Park, and V. Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847, 2018. 6[58] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised learning of depth and ego-motion from video. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 3

for 3d motion field estimation. In European Conf. on Computer Vision (ECCV), 2018. 6, 7
[34] F. Manhardt, W. Kehl, N. Navab, and F. Tombari. Deepmodel-based 6d pose refinement in rgb. In Proc. of the European Conf. on Computer Vision (ECCV), September 2018.
3[35] D. W. Marquardt. An algorithm for least-squares estimationof nonlinear parameters. Journal of the Society for Industrial
and Applied Mathematics (SIAM), 11(2):431–441, 1963. 2[36] T. Meinhardt, M. Möller, C. Hazirbas, and D. Cremers.
Learning proximal operators: Using denoising networks forregularizing inverse imaging problems. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2017. 3[37] R. A. Newcombe, S. Lovegrove, and A. J. Davison. DTAM:
dense tracking and mapping in real-time. In Proc. of theIEEE International Conf. on Computer Vision (ICCV), 2011.
2[38] R. Ranftl and V. Koltun. Deep fundamental matrix estimation. In Proc. of the European Conf. on Computer Vision(ECCV), 2018. 3[39] R. Ranftl and T. Pock. A deep variational model for image
segmentation. In Proc. of the German Conference on Pattern
Recognition (GCPR), 2014. 3[40] G. Riegler, M. Rüther, and H. Bischof. Atgv-net: Accurate
depth super-resolution. In Proc. of the European Conf. on
Computer Vision (ECCV), 2016. 3[41] D. Scharstein and R. Szeliski. A taxonomy and evaluationof dense two-frame stereo correspondence algorithms. International Journal of Computer Vision (IJCV), 47:7–42, 2002.
1[42] J. Shi and C. Tomasi. Good features to track. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
1994. 1[43] H.-Y. Shum and R. Szeliski. Systems and experiment paper:
Construction of panoramic image mosaics with global andlocal alignment. International Journal of Computer Vision(IJCV), 36(2):101–130, February 2000. 1[44] F. Steinbrücker, J. Sturm, and D. Cremers. Real-time visual odometry from dense rgb-d images. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, pages 719–722. IEEE, 2011. 6, 7
[45] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers. A benchmark for the evaluation of rgb-d slam systems.
In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems(IROS), Oct. 2012. 6, 7
[46] M. Sundermeyer, Z.-C. Marton, M. Durner, M. Brucker, and
R. Triebel. Implicit 3d orientation learning for 6d object detection from rgb images. In Proc. of the European Conf. on
Computer Vision (ECCV), September 2018. 3[47] R. Szeliski. Image alignment and stitching: A tutorial.
Foundations and Trends in Computer Graphics and Vision,
2(1):1–104, 2007. 2[48] C. Tang and P. Tan. Ba-net: Dense bundle adjustment network. In Intl. Conf. on Learning Representations (ICLR),
2019. 3[49] B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg,
A. Dosovitskiy, and T. Brox. Demon: Depth and motion

10