Aligning Images in the Wild
Yasuyuki Matsushita2
Kok-Lim Low3
Siying Liu1
Linlin Liu∗3
Institute of Infocomm Research1
Microsoft Research Asia2
National University of Singapore3{wdlin,sliu}@i2r.a-star.edu.sg1 , {liulin,lowkl}@comp.nus.edu.sg3 , yasumat@microsoft.com2(* denotes joint first author.)
Wen-Yan Lin∗1

Abstract

under uncontrolled conditions is even harder.
Input Images

Aligning image pairs with significant appearancechange is a long standing computer vision challenge. Muchof this problem stems from the local patch descriptors’ instability to appearance variation. In this paper we suggest this instability is due less to descriptor corruption andmore the difficulty in utilizing local information to canonically define the orientation (scale and rotation) at which apatch’s descriptor should be computed. We address this issue by jointly estimating correspondence and relative patchorientation, within a hierarchical algorithm that utilizes asmoothly varying parameterization of geometric transformations. By collectively estimating the correspondence andorientation of all the features, we can align and orient features that cannot be stably matched with only local information. At the price of smoothing over motion discontinuities(due to independent motion or parallax), this approach canalign image pairs that display significant inter-image appearance variations.

NRDC

Ours

OverlayInterpolated

Our Warp

NRDC Zoom

1. Introduction
Obtaining point-to-point correspondence across image
pairs is a fundamental problem for various vision tasks,
such as structure-from-motion, image super-resolution andhigh-dynamic-range imaging. However, even for imagesof the same scene, correspondence computation becomeschallenging when they exhibit large inter-image appearance
variations. Unfortunately, a myriad of factors affect the final appearance of an image. These include (1) illumination,that varies shading and atmospheric absorption caused byrain/ haze, (2) view point change, (3) camera settings, suchas camera response function and aperture as well as intrinsics, (4) occlusion and background changes in the temporal interval between photographs, and (5) post processing,when images are enhanced using photo editing software.
These factors make the problem of computing correspondence very difficult. In particular, aligning images taken

Our Zoom

Figure 1. Two images of Machu Picchu taken under different
imaging conditions. It illustrates the difficulty in of obtaining full
frame warping by interpolating even fairly dense correspondenceof NRDC [9]. Alignment quality is displayed by overlaying the
warp’s green channel with the underlying image.

Most previous attempts at aligning images taken under
different conditions focus on modeling appearance variation, with Andrews et al. [3], Weijer et al. [23] and Ha1

Cohen et al. [9] utilizing color transforms or photometricinvariant image representations. However, modeling every
eventuality is difficult. When image sections violate themodel, correspondences become undefined, or large errors
are incurred if a matching result is forced. Hence, problems
like good quality, full frame Internet image alignment, are
yet unsolved.
This paper discusses how varying imaging conditions affect descriptor-based matching and proposes a purely geometric solution. By not relying on appearance models tovalidate matches or remove incorrect correspondence, we
generate dense, full frame warps for image pairs with large
appearance variations. This is illustrated in Figure 1.
Algorithms like SIFT [13], SURF [5] and A-SIFT [16]agglomerate a patch’s gradient information into a local descriptor. As noted by Lowe [13], descriptors combine invariance to affine lighting changes with robustness to nonlinear lighting variation. This can be considered a weakform of photometric invariance. Descriptor-based nearestneighbor matching is remarkably resistant to minor variations in imaging conditions. However, the number andaccuracy of matches decrease sharply with increased appearance variation. Even for more sophisticated group-wise
descriptor alignment techniques such as those by Shum etal. [20], Liu et al. [12], Lin et al. [11], attaining good quality alignment under varying imaging conditions remains achallenge. We suggest that much of this performance decline is not due to irretrievable descriptor corruption and canbe alleviated by a carefully designed matching algorithm.
The key of our solution lies in the descriptor orientation. As a region quantity, a descriptor value depends on
the orientation at which it encodes a patch. Hence, descriptor comparisons are only meaningful if the descriptors fromboth images are computed at the correct relative orientation.
Usually, local image information is used to canonically orient descriptors to ensure comparability [13]. However, atincreased appearance variation, local information may beinsufficient to canonically orient image descriptors.
This problem’s effects are subtle, as the result of descriptor mis-orientation resembles that of descriptor corruption. However, for images taken under similar conditions,works like Fan et al. [8] have reported significant improvements in matching performance through better orientation
handling. Further more, by assuming image pairs are preoriented (feature orientation is not allowed to change), SIFT
flow [12] allows meaningful alignment of different objects.
These results lead us to posit that as imaging conditions
change, descriptors remain discriminative, long after welose the ability to canonically orient them with local information. If true, descriptors can still be utilized for matchingby coupling correspondence and relative feature orientation
estimation within a group-wise matching framework.
To achieve this coupling, we formulate the non-linear re-lationship between feature orientation and descriptors intoa more manipulatable linear orientation choice. We jointly
compute feature correspondence and orientation using a hierarchical series of models that progressively relax groupwise constraints. At the price of smoothing over motion discontinuities, we can use sparsely distributed feature descriptors to interpolate dense warps across image pairs taken atvery different imaging conditions. This is surprising given
our purely geometric approach and indicates the potential
for better results by fusing both lighting and geometric cues.
Our contributions are summarized as follows• We posit that as imaging conditions change, descriptors remain discriminative, despite loss of ability to
canonically orient them with local information;• We demonstrate this using a smoothly varying orientation and correspondence estimation algorithm to estimate full frame warps between two images of a scenetaken under different imaging conditions;• We demonstrate camera pose recovery and highdynamic-range imaging as applications of our method.

1.1. Related Works
There is a considerable body of prior work on simultaneous region growing and orientation including works by
HaCohen et al. [9], Barnes et al. [4] Vedaldi et al. [24] andCheng et al. [7]. However, many require specific color modeling to handle illumination changes well. While more flexible, growing algorithms do not work well when local information is unstable and under appearance variation, theirgrown correspondence can be erroneous. Although correspondence errors can be rejected with a photometric-based
thresholding such as that employed by HaCohen et al. [9],this results in sparse correspondences. Interpolation on suchcorrespondence is very vulnerable to correspondence errors.
In contrast, by evolving all the features together, we can interpolate across weak textures and our dense warping results
can be used “as is” without further thresholding.
Our algorithm is more similar in spirit to descriptor
based SIFT flow [12]. While we cannot achieve warping
across images of different objects, our orientation varying
descriptors permit finer alignment and better handling oforientation changes.
Our work is also an attempt at establishing relational
structure between images taken under uncontrolled conditions. This motivation is shared by many works utilizing
Internet images, such as “Building Rome in a Day” [2]that creates 3-D city models from community photo collections, Shrivastava et al.’s [19] cross-domain matching and
Kemelmacher-Shlizerman et al.’s [10] face reconstruction
from Internet images. However, these techniques are notdirectly applicable to our problem as they achieve stability

by leveraging on dataset size, while our work takes a single
image pair as input.
Compared to sophisticated color modeling techniquessuch as Color Eigenflows [15] and CSIFT [1], we utilize
the weaker invariance of generic SIFT features [13]. Instead, we handle differing imaging conditions by focusing
on geometric rather than photometric constraints.
Our formulation builds on point set registration works
such as thin plate spline [6], motion coherence [25], coherent point drift [17] and smoothly varying affine [11]. Of
these, we are most closely related to Lin et al.’s [11] work.
We also utilize motion coherence [25] to parameterize ourvariables as a smooth variation from approximate global
models. However, unlike [11] that depends on locally oriented SIFT descriptors, our joint estimation of feature orientation and correspondence handles appearance variations
much better as demonstrated in Figure 2.

orientations. Apart from feature orientation, we employ
the same “choosing” mechanism to enable finer registration
through feature re-localization.

2.1. Formulation
We formulate the correspondence problem between baseand target images as a registration of two evolving featuresets. The base feature set is B = {bi } and the target feature set T = {T j |T j = {tjθ }}. Here, the nested target
feature set is used to accommodate orientation choice.The
initial feature values are computed directly from the image.
These are denoted as B 0 = {b0i } and T 0 = {T 0j }, with
subscript 0 representing initial values.
2.1.1

Feature Definitions

A base feature vector bi ∈ B takes the formbi = [ bci2. Our Approachbribdi ],

with sub-vectors bci , bri , and bdi denoting the feature’s spatial coordinate, orientation, and descriptor, respectively.
Target feature definition is more complex. T j = {tjθ }represents a set of target features, sharing the same image
coordinates but differing orientations, with θ ∈ {1, . . . , Θ}
indexing feature orientations. This gives each bi feature achoice of orientation. tjθ ’s definition is similar to bi
Input Images

Smoothly Varying Affine
and orientation

Refined Matching

Smoothly Varying Affine [11]Similarity Transform( single orientation)

Affine Transform(single orientation)(using prev. affine asinitialization)

Figure 2. Illustration of our hierarchical approach, results using anaive smoothly varying affine [11] are shown at bottom right.

We use a smoothly varying field to warp and orient features from one image to the other. Scaling or rotating an image patch causes a permutation (with some re-sampling andanti-aliasing) of image pixels. This results in a complex,
non-linear relationship between feature orientation and descriptors. To side-step the problem of optimizing that relationship, we implement a linear orientation choice mechanism. On one image, multiple differently oriented featuredescriptors are computed at each spatial location. Featuresfrom the other image then choose the orientation best corresponding to their own. This “choosing” mechanism does
not increase the number of variables but comes at the costof a one-off descriptor comparison across many differenttjθ = [ tcjθ

trjθ

tdjθ ],

with sub-vectors tcjθ , trjθ , tdjθ denoting the feature’s spatial
coordinate, orientation and descriptor.
Practically, our initial feature sets {b0i } and {T 0j } arise
from regions of interest detected using Lowe’s [13] SIFT
algorithm, with i ∈ {1, . . . , M } and j ∈ {1, . . . , N } indexing regions of interest in the base and target image, respectively. bc0i and tc0jθ are the image coordinates of the regions
Tof interest, given in the homogeneous form [x y 1] . Unlike in a traditional feature detection, all feature descriptors
are computed at fixed orientations and scales. For the baseimage, all features are computed at patch radius p0 and rotation direction r0 . For the target image, feature t0jθ is assigned a patch of radius pθ and rotation angle rθ relative
to r0 . The initial target feature orientation is parameterizedwith respect to the base orientation. Thus,tr0jθ = p0 /pθ

cos(rθ )Tsin(rθ ) .

As only relative feature orientation is important, p0 and r0
are not actually assigned to br0i which is a dummy variable.
The SIFT features are computed at the appropriate image
coordinates and orientations to obtain the descriptors bd0i
and td0jθ .
We evolve the spatial and orientation components of feature sets B and T using a hierarchy of transformations from

initial B 0 and T 0 values. These transformations are progressively relaxed to permit point-wise evolution and finer
image alignment. Feature descriptors serve as guides to theevolution and remain invariant throughout, i.e., bdi = bd0i ,and tdi = td0i .
For notational simplicity, we also define Gaussian andcompound Gaussian functions:
(

2.2.2

Affine Transformation

For the second level, spatial coordinates are parameterizedwith a global affine transformation:
bci = Abc0i ,tjθ = t0jθ ,bri = [s u v]T .

(4)

We solve for parameters α = {A, s, u, v} by minimizing
Eqn(2). While orientation and spatial parameters are no2
2g(z, σ) = exp−(kzk /2σ )longer directly coupled, there remains an indirect couplingφijθ (bi , tjθ ) = g(tcjθ − bci , σc )g(trjθ − bri , σr )g(td0jθ − bd0i , σd ) effect due to their concatenation into one feature vector.

Here φijθ (·) is defined with respect to td0jθ and bd0i rather
than tdjθ and bdi because of the descriptor invariance bdi =bd0i and tdjθ = td0jθ .
2.1.2

Cost Function

2.2.3

Next we employ a smoothly varying orientation and affine
parameterization that allows bi coordinates to vary individually while robustly handling noise and missing entries:tjθ = t0jθ ,To quantify alignment accuracy, we treat the base feature
vectors bi as generative Gaussian centroids. Thus,
P (tjθ |B) =MX(φijθ (bi , tjθ ) + κ) ,i=1

where κ is a positive number included to handle featuresappearing in only one image. For interest region T j , itsΘ orientation candidates are treated as mutually exclusive
choices. Hence,
P (T j |B) =

Θ X
M
X

(φijθ (bi , tjθ ) + κ) .

Q(α) = −(1)

log (P (T j |B)) ,

(2)

j=1

where α represents the parameters of the transformations
applied to the initial base and target features b0i and t0jθ .

2.2. Hierarchy of Models
For robustness to local minima and efficiency, we do notminimize alignment cost directly. Rather, we parameterize
the evolution of sets B and T using a hierarchical seriesof models. The simpler, more robust models serve as approximate initializations for subsequent more refined models. The process is illustrated in Figure 2.
2.2.1

(5)

N
X

log (P (T j |B)) + λΨ(B),

(6)

j=1

We seek to minimize the overall negative log likelihood
Q(α) = −bci = (A + ∆Ai ) bc0i ,Tbri = ∆si ∆ui ∆vi

with the smoothness being applied to the point-wise varying ∆i coordinates. The smoothness is computed with respect to the global variables rather than to the original or
initialized positions. This allows for larger motion and lessvulnerability to initialization errors.
To enforce smoothness, we use the energy function

θ=1 i=1

N
X

Smoothly Varying Orientation and Affine

where α = {A, ∆Ai , ∆si , ∆ui , ∆vi } are the variable parameters. Note that while A and ∆Ai are 3 × 3 matrices,
only the entries of the first two rows are variables.
Smoothness function Ψ(·) takes the motion coherence
form employed in [11, 17]. This method ensures the warping field carrying the discrete points is smooth, permitting
dense warping of high dimensional sparse points.
Let v(z2×1 ) represent a 2-D smoothly varying field ofsome parameter. The coherence term penalizes discontinuities usingZ|v 0 (ω)|2dω,
0
R2 g (ω)(7)

where v 0 (ω) is the Fourier transform of the velocity field,
while g 0 (ω) is the Fourier transform of a Gaussian with spatial standard deviation γ.

Similarity Transformation

The first level of our hierarchy is the similarity transformation:cccctjθ = St0jθ , bi = Rb0i ,(3)trjθ = tr0jθ ,bri = [s u v]T ,hs 0 0ih u −v t1 i
where S = 0 s 0 and R = v u t2 . We solve for scale,
0 0 1

0 0

1

rotation and translation parameters, α = {s, u, v, t1 , t2 }, byminimizing Eqn (2). The similarity transformation tightly
couples feature orientation and spatial coordinates, providing a good initialization for the next level.

2.2.4

Feature Re-localization

The above steps provide only approximate alignment as thefeatures may not be consistently localized across images.
For finer registration, we replace the orientation choice witha localization choice. Target set T is replaced with a newe = {Te j |Te j = {t̃jk }, j ∈ {1, . . . , M }}. Each Tej
set T
consists of Ki localization candidates. These are high gradient regions (“Harris corners” without non-maxima suppression or thresholding) within a 30-pixel neighborhood

of each evolved bci . Similar to Eqn (6), we allow base features to choose more accurate localization. Feature vectors
e share the same format as T , except that all orientationin Tvalues are set to zero since we no longer choose orientation.
The transformation relating B and B 0 is the same as thatused in Eqn (5). The spatial parameters α = {A, ∆Ai } aresolved for by minimizing
Q(α) = −

M
X
j=1

log

Ki M
X
X


φijk (bi , t̃jk ) + κ + λΨ(B), (8)k=1 i=1

whose derivation is similar to Eqn (2). Note that by zeroing
all orientation values, orientation variables do not affect theminimization and can be ignored.

for the parameters {s, u, v, t1 , t2 }{m+1} .
Affine Transformation. Eqn (4) is updated by solving the
9 linear equations
Pi,j,θ Dijθ (cijθ ) = 06×1 ,i,j,θ

Smoothly Varying Orientation and Affine. Parameterization of Eqn (5) is updated by solving 6 + 9 × M linear
equationsP

i,j,θ Dijθ (cijθ ) = 06×1 ,GV + 2λσc2 ∆A = 0M ×6 ,We minimize the cost defined in Eqns (2), (6), and(8) using Expectation Maximization’s iterative refinement
procedure. At the m-th iteration, the alignment B {m}and T {m} can be used to define a set of equations which
are linear in terms of the α{m+1} parameters of the newalignment B {m+1} , T {m+1} . Solving these equations
provides an improved alignment which is in turn used tocompute a new set of equations. The process is repeateduntil convergence. Here, we simply state the linear update
equations, with detailed derivation in the appendix. In eachcase, when solving for the {m + 1} parameters, the m-th
registration values are treated as constants.
Notations. To simplify the description of the minimization,
we define the followings:
φ(bi ,tjθ )φijθ (B, T j ) = P P ijθ,(bh ,tjl )+κ
lh φhjl



c


0
03×1
b
3×1
0i

 Dijθ = φijθ (B m , T m
,
c
j )
03×1
03×1
b0i

T

−t0jθ(1) b0i(1) −b0i(2),Fijθ = φijθ (B m , T mj )b0i(1)

 −t0jθ(2) b0i(2)ccG(i,j) = g b0i − b0j , γ ,where Dijθ and Fijθ are 6 × 3 and 3 × 2 data matrices, G
is M × M affinity matrix and γ of Eqn (7) controls degreeof smoothness.
The feature coordinate vectors arecijθ = (bci ){m+1} − (tcjθ ){m+1} ,rijθ = (bri ){m+1} − (trjθ ){m+1} .

i,j,θ

P

i,j,θ represents a triple sum of indexes i, j and θ.

σr2 Fijθ (cijθ ) + σc2 φijθ (B, T j )rijθ = 03×1 ,P
i,j,θ φijθ (B, T j )cijθ(1:2) = 02×1 ,GU + 2λσr2 ∆R = 0M ×3 ,(11)

for the parameters {A, ∆Ai , ∆si , ∆ui , ∆vi }{m+1} . V and
U are matrices whose respective i-th rows areV(i,:) =Pjθ (Dijθ cijθ )T,

U(i,:) =Pjθ φijθ (B, T j ) (rijθ )(9)Tand the i-th row of matrices ∆A and ∆R are
h
i
{m+1}
{m+1}
∆A(i,:) = ∆Ai(1,1:3) , ∆Ai(2,1:3) ,T∆R(i,:) = (bri ){m+1} .

Feature Re-localization. Eqn (8) uses the same smoothly
varying affine spatial parameterization as Eqn (11) but replaces orientation choice with a spatial correspondence
choice. Its 6 + 6 × M linear update equations areXe ijk (c̃ijk ) = 06×1 ,
D

i,j,k

e + 2λσc2 ∆A = 0M ×6 ,
GV

(12)

with ˜ denoting matrix variants where orientation choiceterms tjθ , θ, Θ and N are replaced with the spatial choiceterms t̃jk , k, Ki and M defined in Eqn (8).
Spatial warping of a 2 × 1 base image coordinate z is defined by the sum of the global affine A and its smoothly
varying affine offset a(z). a(z) is a continuous function defined as
WM ×6 = [w1 , ..., wM ]T = G+ ∆A,PMa(z) = i=1 wi g(z − b0i(1:2) , γ),(13)where G+ is G’s pseudo-inverse, and wi is a 6 × 1 vector.

Similarity Transformation. Eqn (3) is updated by solving
the 5 linear equations
P(10)

for the parameters {A, s, u, v}{m+1} .

2.3. Minimization Technique

Operatorφijθ (B, T j )rijθ = 03×1 ,P

2.4. Implementation
System implementation details are as follows. SIFT descriptors are computed using VL-SIFT that allows specification of the desired feature orientation. After feature creation, image coordinates of features are normalized to zeromean, unit variance. Image alignment is computed using
Algorithm 1. For the similarity transformation, its initialscale value s is set to 1, while the remaining {u, v, t1 , t2 }parameters are initialized to the orientation neutral 0. The
remaining transformations are initialized from the previous
estimate. Throughout the algorithm, σd of the SIFT features
is held constant, while σc and σr of the evolving coordinates
are annealed smaller to force alignment. Parameters for thevarious algorithm stages are listed below. Unless otherwise
stated, these parameters are used throughout the paper andthe algorithm is relatively insensitive to minor variations inparameter choice.
Similarity
AffineSmooth. Var.
Re-loc.

initial σc2
0.1
0.01
0.01
0.01

initial σr2
0.01
0.00040.0004
−

σd2
0.04
0.04
0.04
0.04

λ
−
−
20
20

γ
−
−
4
4

Input: Base image, Target image
for each parameterization dowhile σc above threshold do
while no convergence doswitch{parameterization}case Similarity:
update alignment with Eqn (9)case Affine:
update alignment using Eqn (10)case Smoothly Varying Orientation and
Affine:update alignment using Eqn (11)case Re-localization:
update alignment using Eqn (12)endAnneal σc = σc , σr = σr ,where  = 0.97.
end
end
Output: Aligned images
Algorithm 1: Overall algorithm

3. Results
We align image pairs with relative scale and rotation
ranges of [0.5, 2] and [−45◦ , 45◦ ]. While the algorithm canhandle larger orientation changes, there is a trade off between stability to appearance variations and the permittedorientation range. Over these ranges, a mixed Matlab and C
implementation of our algorithm can handle most 360×480
images within15 minutes on an Intel Core i7 computer.
In Sections 3.1 and 3.2, we evaluate our alignment results, while Sections 3.3 and 3.4 demonstrate Camera Pose
Recovery and High Dynamic Range imaging as two of theapplications of our alignment method.

3.1. Evaluation
To evaluate warping quality over illumination change,
we capture a set of images at different locations. At eachlocation, images are taken at a number of fixed illumination
conditions. Warping is computed between images taken at
different illumination conditions. The warp is transfered
to image pairs taken under similar illumination conditions,which serves as evaluation ground-truth.
We compute the root-mean-square error (rmse) between
images. We also compute the percentage of warped pixels
whose target surroundings do not contain any similar pixels (% outliers). Surroundings are defined as a 4-pixel radius while the similar pixel threshold is set at 10 gray levels.
The latter metric is similar in spirit to the “earth movers distance” and seeks to filter away illumination variation noise
while avoiding over penalization of small discrepancies.
We evaluate two scenes, with the second containing significant depth discontinuity. While our algorithm’s error isclearly higher on the second scene that violates its smoothness assumption, it remains stable. For benchmarking purposes, we show the results for SIFT flow [12] and Large
Displacement optical flow [21].
SIFT flow

Large disp. optical flow

Ours

(14.6%, 37.2)(13.2%, 41.3)(6.1%, 33.8)(23.4%, 47.6)(29.2%, 49.8)(11.5%, 33.1)

B

T

GT

B

T

GT

Figure 3. Left column: Base image, target image and reference
ground truth. Right columns: Warps and overlays for SIFT
flow [12], Large Displacement optical flow [21] and our algorithm.
Errors are given below the images in the form (% outliers, rmse).

3.2. Alignment
We evaluate our alignments using standard image dataset
from Mikolajczyk et al. [14] and Internet images. Alignments are visualized by replacing the warp’s green channel
with that of the target image. We also show the results of
NRDC [9] and SIFT flow [12]. Our alignments are finer
than SIFT flow’s and we handle orientation change better.
Our warps are denser than NRDC’s correspondence, especially for less textured scenes. Dense warping is much moredifficult than sparse matching, as it requires maintenance ofa reasonable error rate while trying to maintain a perfect recall. Some examples of Internet image alignment are shownin Figure 4, with results for Mikolajczyk et al.’s [14] dataset
in the supplementary material.1
For Internet images, our algorithm works well between
scale ranges of ×3 and out of plane rotation of ±20◦ degrees. Most color variations can be handled as long as images are taken during the day, however, the illumination
variations caused by night photography are beyond our algorithms capability.
Note that these results are obtained without explicit color
modeling, thus validating our hypothesis that proper geometric handling of SIFT descriptors is sufficient to compensate for large variations in image appearance.

3.3. Camera Pose
Our dense alignment also facilitates “tracking” a point
across all images of a set. This is advantageous to camera pose recovery. By propagating correspondence acrossall images and using Hartley normalized image coordinates
to reduce size variation caused by differing focal lengths
or image resolutions, we can utilize the very stable factorization [22] algorithm for pose recovery from orthographic
image streams. This recovers of camera pose up to a forward translational ambiguity. Many Internet images can bemodeled as orthographic and reconstruction of a user chosen image set is shown in Figure 5. Works such as “Building
Rome in a Day” [2] cannot reconstruct the scene from suchsmall image sets because of the paucity of SIFT correspondences (approximately zero for this set).

3.4. High Dynamic Range Imaging
Another application is High-Dynamic-Range (HDR)
Imaging, a set of techniques to increase the dynamic
range between the lightest and darkest areas of an image.
Typically, this is achieved by merging multiple standarddynamic-range (SDR) images. Tone-mapping is used to display the HDR image on lower dynamic range devices [18].
One difficulty of HDR imaging is the alignment of multiple SDR images of different exposures. However, evenunder-exposed images contain SIFT feature our algorithm
can utilize to align images. This enables HDR capture witha hand-held camera.2 While better results can be obtainedby first applying exposure correction techniques, our algorithm directly aligns the un-processed images. Results areshown in Figure 6.

View 1

View 2

Figure 6. Alignment of SDR images using ours, Photomatix and
Photoshop. Alignment quality is visible in direct fusion withoutde-ghosting. Other alignment software incur significant ghosting.
Observe the surprising number of SIFT features on the underexposed image. This allows us to align it with the other images
despite significant appearance changes. Final result, an artisticallychosen tone-mapped image after using our algorithm’s alignment.

4. Discussions
Figure 5. Left: Two different 3-D views of an orthographic reconstruction. Images represent camera positions arranged sphericallyaround a reconstructed point cloud. Right: Input images.
1 Our algorithm’s emphasis on descriptor distances does not always cor-Our descriptor-based algorithm can align images overlarge appearance variations with only geometric constraints.
While there are limits to the tolerable variations (we are
restricted to day images), our approach reduces the relianceon photometric models and lays the foundation for better

respond to human perception. In particular, when matching faces, our algorithm’s low emphasis on the un-textured eyes can result in an overallcorrect warping outline but mis-aligned facial fiducial points.

2 This is different from matching day and night images which our algorithm does not perform well on.

Figure 4. Left to right: Input images, our warped
results and overlays for
NRDC [9], SIFT flow [12]and our algorithm. Observethat we handle orientation
change better than SIFT
flow, while NRDC’s correspondence for less textured
scenes is quite sparse.

Image 1

Image 2

Our warp

Overlay NRDC

results by employing both techniques. More practically,
this opens opportunities for harnessing rich, Internet image
content. An example is computation of relative Internet
image poses demonstrated in this paper. Our approaches
drawback is smoothing over discontinuous motion. This
causes errors when there is significant parallax.
Acknowledgment: This work is partly supported by Singapore MOE Academic Research Fund (Project Number: T1
251RES1104).

References[1] A. E. Abdel-Hakim and A. A. Farag. Csift: A sift descriptor
with color invariant characteristics. CVPR, 2006. 3[2] S. Agarwal, Y. Furukawa, N. Snavely, B. Curless, S. M.
Seitz, and R. Szeliski. Building rome in a day. ICCV, 2009.
2, 7[3] R. J. Andrews and B. C. Lovell. Color optical flow. Workshop on Digital Image Computing, 2003. 1[4] C. Barnes, E. Shechtman, and A. F. Dan B Goldman. The
generalized patchmatch correspondence algorithm. ECCV,2010. 2
[5] H. Bay, T. Tuytelaars, and L. V. Gool. Surf: Speeded up
robust features. ECCV, 2006. 2[6] F. L. Bookstein. Principal warps: Thin-plate splines and thedecomposition of deformations. PAMI, 1989. 3[7] H. Cheng, Z. Liu, N. Zheng, and J. Yang. A deformable local
image descriptor. CVPR, 2008. 2[8] B. Fan, F. Wu, and Z. Hu. Aggregating gradient distributions
into intensity orders. CVPR, 2011. 2[9] Y. HaCohen, E. Shechtman, D. B. Goldman, and D. Lischinski. Nrdc: Non-rigid dense correspondence with applications
for image enhancement. SIGGRAPH, 2011. 1, 2, 6, 8, 12[10] I. Kemelmacher-Shlizerman and S. M. Seitz. Face reconstruction in the wild. ICCV, 2011. 2[11] W.-Y. Lin, S. Liu, Y. Matsushita, T.-T. Ng, and L. F. Cheong.
Smoothly varying affine stitching. CVPR, 2011. 2, 3, 4, 9Overlay SIFT flow

Overlay ours

[12] C. Liu, J. Yuen, A. Torralba, J. Sivic, and W. T. Freeman. Siftflow: Dense correspondence across different scenes. ECCV,2008. 2, 6, 8, 12[13] D. G. Lowe. Distinctive image features from scale-invariant
keypoints. IJCV, 2004. 2, 3
[14] K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman,
J. Matas, F. Schaffalitzky, T. Kadir, and L. V. Gool. A comparison of affine region detectors. IJCV, 2006. 6, 7, 12[15] E. G. Miller and K. Tieu. Color eigenflows: Statistical modeling of joint color changes. ICCV, 2001. 3[16] J. Morel and G. Yu. Asift: A new framework for fully affine
invariant image comparison. SIAM Journal on Imaging Sciences, 2009. 2[17] A. Myronenko, X. Song, and M. Carreira-Perpinan. Nonrigid point set registration: Coherent point drift. NIPS, 2007.
3, 4, 9[18] E. Reinhard, M. Stark, P. Shirley, and J. Ferwerda. Photographic tone reproduction for digital images. Siggraph,2002. 7[19] A. Shrivastava, T. Malisiewicz, A. Gupta, and A. A. Efros.
Data-driven visual similarity for cross-domain image matching. Siggraph Asia, 2011. 2[20] H.-Y. Shum and R. Szeliski. Construction of panoramic image mosaics with global and local alignment. IJCV, 1999.
2[21] T.Brox and J.Malik. Large displacement optical flow: Descriptor matching in variational motion estimation. PAMI,
2010. 6[22] C. Tomasi and T. Kanade. Shape and motion from image
streams under orthography a factorization method. IJCV,1992. 7[23] J. van de Weijer and T. Gevers. Robust optical flow fromphotometric invariants. ICIP, 2004. 1[24] A. Vedaldi and S. Soatto. Local features, all grown up.
CVPR, 2006. 2[25] A. L. Yuille and N. M. Grywacz. The motion coherence theory. ICCV, 1988. 3