Camera Calibration with Lens Distortion from Low-rank Textures
Zhengdong Zhang∗
Tsinghua University, China

Yasuyuki Matsushita
Microsoft Research Asia

Yi Ma
Microsoft Research Asiazhangzdfaint@gmail.com

yasu@microsoft.com

mayi@microsoft.com

Abstract
We present a simple, accurate, and flexible method tocalibrate intrinsic parameters of a camera together with(possibly significant) lens distortion. This new method canwork under a wide range of practical scenarios: using multiple images of a known pattern, multiple images of an unknown pattern, single or multiple image(s) of multiple patterns, etc. Moreover, this new method does not rely on extracting any low-level features such as corners or edges. It
can tolerate considerably large lens distortion, noise, error,
illumination and viewpoint change, and still obtain accurate estimation of the camera parameters. The new method
leverages on the recent breakthroughs in powerful highdimensional convex optimization tools, especially those formatrix rank minimization and sparse signal recovery. We
will show how the camera calibration problem can be formulated as an important extension to principal component
pursuit, and solved by similar techniques. We characterize
to exactly what extent the parameters can be recovered incase of ambiguity. We verify the efficacy and accuracy ofthe proposed algorithm with extensive experiments on real
images.

1. Introduction
Camera calibration is arguably one of the most classic and fundamental problems in computer vision (andphotogrammetry), which has been studied extensively for
decades. It is fundamental because not only every newly
produced camera must run calibration to correct its radial
distortion and intrinsic parameters, but also it is the first step
towards many important applications in vision, such as reconstructing 3D structures from multiple images (structurefrom motion, photometric stereo, structured lights, etc.).
Existing methods have provided us with many choices
to solve this problem in different settings. To the best of
our knowledge, almost all of calibration methods rely on
extraction of certain local features first, such as corners,∗ This work was done while the first author was visiting Microsoft Research Asia.

(a) Image from a fisheye camera(b) Distortion automatically corrected

Figure 1. Distortion in an image of a building taken by a fisheye
camera automatically corrected by our method.

edges, and SIFT features, and then assemble them to establish correspondences, calculate vanishing points, inferlines or conic curves for calibration. It is well-known thatin practice it is difficult to accurately and reliably extract all
wanted features in all images in the presence of noise, occlusion, image blur, and change of illumination and viewpoint. Large noise, outliers, missing features, and mismatches all could render the calibration result inaccurate
and even invalid. Today arguably the only reliable way toobtain accurate calibration and distortion still relies on manually labeling the precise location of points in multiple images of a pre-designed pattern, as required by most standard
calibration toolboxes (e.g., [1]). Not only does the use of apre-designed pattern limit the use of such methods to restricted (laboratory) conditions, but also the careful manual
input makes camera calibration a time-consuming task.
Recently, breakthroughs in high-dimensional convex optimization have enabled people to correct global geometricdistortion of images directly using image intensity values.
In particular, the recent work [17] has shown that for an image of a plane whose texture, as a matrix, is very low-rank,
one can efficiently and accurately recover the low-rank texture from its perspectively distorted version via convex rank
minimization techniques. Inspired by that work, in this paper, we show how such optimization techniques can help
solve the camera calibration problem in a more convenientand flexible way. A representative result of our method isgiven in Fig. 1, in which the lens distortion of a fisheye camera is corrected based on an image itself.

Contributions. In this paper, we will show that this new
approach leads to a simple and accurate solution to camera
calibration or self-calibration without requiring extracting,labeling, or matching any low-level features such as points
and edges. The new algorithm directly works with raw image intensity values and can accurately estimate the camera
intrinsic parameters and lens distortion under a broad practical conditions: from a single or multiple images, from aknown or unknown pattern, even with possible noise, saturations, occlusion, and under different illuminating conditions. It can be used either for pre-calibrating the camera from a known pattern or for performing automatic selfcalibration from images of structured scenes. It requires
few, inaccurate initialization, and thus is very convenient
to use. Also, as it relies on scalable optimization techniques, with proper implementation, the speed can be very
fast. As we will verify with extensive experiments, the algorithm achieves comparable performance to the standard
toolbox, but with more flexible initialization and working
under broader realistic conditions.

1.1. Prior work
During the past several decades, researchers have studied
many different approaches for developing more convenient,practical, and accurate algorithms for camera calibration.
One important class of these solutions require a speciallydesigned calibration object, with 3-D geometric information known explicitly [2, 6, 7, 14, 15]. The calibration objects include 3-D [14], 2-D plane [15], and 1-D [16] line targets. By observing these targets from different viewpoints,
these techniques recover the camera intrinsic parameters.
The 3-D calibration object usually consists of two or threeplanes orthogonal to each other, and it gives the most accurate calibration with a simple algorithm; however, the setup
is more complicated and expensive. The 2-D plane-based
calibration requires observing a planar pattern from different viewpoints. The technique is implemented in Camera
Calibration Toolbox [1], and it gives accurate results withless complicated settings. The 1-D line-based calibration
uses a set of collinear points with known distances. Because
it can better avoid occlusion problems, it is often used for
multi-camera calibration.
Unlike above methods, camera self-calibration [11, 8]
avoids the use of known calibration pattern and aims at calibrating a camera by finding intrinsic parameters that areconsistent with the geometry of a given set of images. It
is understood that sufficient point correspondences among
three images are sufficient to recover both intrinsic and extrinsic parameters. Because self-calibration relies on point
correspondences across images, it is important for these approaches to extract accurate feature point locations and it
normally does not handle lens distortion.
Calibration based on vanishing points are also investi-

gated by researchers [3, 10, 13, 4, 9]. These approaches
utilize parallelism and orthogonality among lines in the 3D space. For example, certain camera intrinsics with therotation matrix can be estimated from three mutually orthogonal vanishing points. While useful, these approaches
strongly rely on a process of edge detection and line fitting
for accurately determining vanishing points. Methods thatuse line features like done by Devernay and Faugeras [5]share similar processes, and the accuracy and robustness are
too susceptible to noisy and faulty low-level feature extraction.
All in all, almost all calibration methods share one thingin common, i.e., almost exclusively relying on whether
points or lines can be reliably obtained from local corneror edge features. Feature extraction or labeling often becomes a bottleneck of the process, affecting robustness, accuracy, and convenience. The proposed new method naturally avoids this problem by a new formulation that does not
require any low-level feature extraction.

2. Camera Model with Lens Distortion
We first briefly describe the common mathematicalmodel used for camera calibration and introduce notation
used in this paper. We use a vector M = (X0 , Y0 , Z0 )T ∈R3 to denote the 3D coordinates of a point in the world
coordinate frame, use mn = (xn , yn )T ∈ R2 to denote
its projection on the canonical image plane in the camera
coordinate frame. For convenience, we always denote thehomogeneous coordinate of a point m as m̃ = [ m1 ].
Lens distortion model. If the lens of the camera is distorted, on the image plane, the coordinates of a point mn
may be transformed to a different one, denoted as md =(xd , yd )T ∈ R2 . A very commonly used general mathematical model for this distortion D : mn 7→ md is given by apolynomial distortion model [2] by neglecting any higherorder terms as below:. p 2
xn + yn2 ,r =.
f (r) = 1 + kc (1)r2 + kc (2)r4 + kc (5)r6 ,
(1)


2
2
f (r)xn + 2kc (3)xn yn + kc (4)(r + 2xn )md =.
f (r)xn + 2kc (4)xn yn + kc (3)(r2 + 2yn2 )Notice that this model has a total of five unknowns
kc (1), . . . , kc (5) ∈ R. If there is no distortion, simply set
all kc (i) to be zero, and then it becomes md = mn .
Intrinsic parameters. To transform a point into the pixel
coordinates, we use the usual pin-hole model parametrized
by an intrinsic matrix K ∈ R3×3 , which also have five unknowns; the focal length along x and y-axes fx and fy ,skew parameter θ, and coordinates of the principle point(ox , oy ). In the matrix form, it is described as


f
θ ox
.  x
K = 0 fy oy  ∈ R3×3 .
0 0 1

(2)

Extrinsic parameters. Finally, we use R = [r1 , r2 , r3 ] ∈SO(3) and T ∈ R3 to denote the Euclidean transformation from the world coordinate frame to the camera
frame – so-called extrinsic parameters. The rotation R
can be parameterized by a vector ω = (ω1 , ω2 , ω3 )T ∈R3 using the Rodrigues formula [7]: R(ω) = I +ω̂ 2ω̂+ (1 − cos kωk) kωk
sin kωk kωk
2 , where ω̂ denotes the3 × 3 matrix form of the rotation vector ω, defined asω̂ = [0, −ω3 , ω2 ; ω3 , 0, −ω1 ; −ω2 , ω1 , 0] ∈ R3×3 .
With all the notation, the overall imaging process of apoint M in the world to the camera pixel coordinates m bya pinhole camera can be describe as:m̃ = K m̃d = KD(m̃n );λm̃n = [R T ]M̃ ,(3)

where λ is the depth of the point. If there is no lens distortion (md = mn ), the above model reduces the typical
pin-hole projection with an uncalibrated camera: λm̃ =K[R T ]M̃ .
For compact presentation, later in this paper, we will let
τ0 denote the intrinsic parameters and lens distortion parameters all together. We use τi (i = 1, 2, . . .) to denote the extrinsic parameters Ri and Ti for the i-th image. By a slightabuse of notation, we will occasionally use τ0 to represent
the combined transformation of K and D acting on the image domain, i.e., τ0 (·) = KD(·), and use τi (i = 1, 2, . . .)to represent the transforms from the world to individual image planes.

(in brief τi ), with the same intrinsic matrix K and lens distortion kc (in brief τ0 ). In practice, the observed images are
not direct transformed versions of I0 , each may have contained some background or partially occluded regions (saydue to limited field of view of the camera). We use Ei tomodel such error between the original pattern I0 and the ith
observed image Ii with the transformations undone. Thenmathematically we have:
Ii ◦ (τ0 , τi )−1 = I0 + Ei ,(4)where the operator ◦ denotes the geometric transformations.
The task of camera calibration is then to recover τ0 andprobably τi (1 ≤ i ≤ N ), too, from these images.
In general, we assume that we do not know I0 in advance.1 So, we do not have any ground-truth pattern tocompare or correspond with for the images taken. Our goal
is to fully recover the distortion and calibration by utilizing
only the low-rankness of the texture I0 and by establishing
precise correspondence among the N images Ii themselves.
Rectifying deformation via rank minimization. We
draw inspiration from two previous work. Since we know
the pattern is low-rank, from the work on transform invariant low-rank textures (TILT) [17], we can estimate the deformation of each image Ii from I0 by solving the following
robust rank-minimization problem:
min kAi k∗ + λkEi k1 , s.t. Ii ◦ (τ0 , τi )−1 = Ai + Ei , (5)3. Calibration from Low-rank Textureswith Ai , Ei , τi and τ0 as unknowns. The work [17] has
shown that if there is no radial distortion in τ0 , the above
optimization recovers the low-rank pattern I0 up to a translation and scaling in each axis, i.e.,h sx 0 mx i
Ai = I0 ◦ τ, where τ = 0 sy my .
(6)

Our method estimates camera parameters from low-rank
textures. The pattern can be unknown, but is sufficiently
structured, i.e., as a matrix it is sufficiently low-rank (e.g.,
the normally used checkerboard is one such pattern). We
describe our method in two cases; multiple-image andsingle-image cases. From multiple observations of the lowrank textures, our method can fully recover lens distortion,
intrinsics, and extrinsics. In the case of a single image as
input, our method can estimate lens distortion as well as
intrinsics with additional yet reasonable assumptions.
By default we choose the origin of the world coordinate
to be the top-left corner of the image and let the image lie inthe plane Z = 0 and X and Y be the horizontal and verticaldirection, respectively.

However, in our problem, both intrinsic parameters and distortion are present in the deformation. Therefore, a single
image can no longer recover all the unknowns (and we will
discuss in the next section exactly what can be recoveredfrom a single image of low-rank patterns.)
Our hope is that multiple images give us additional information for all the unknown parameters. For that, we needto establish precise point-to-point correspondence among
all the N images. Again, robust rank-minimization techniques offer a good guideline for solving this problem. In
the previous work of RASL [12], the authors have proposed that multiple images can be precisely and efficientlyaligned by solving a robust rank-minimization problem similar to Eq. (5). However, the resulting aligned images could

3.1. Multiple Images of the Same Low-Rank Pattern

1 This is where our method deviates from the classical camera calibra-

Suppose we have images of a certain pattern I0 ∈Rtaken from N different viewpoints R(ωi ) and Tition setting and it makes our method works under broader conditions. We
will discuss by the end of the section what if we do know the pattern inadvance.

m0 ×n00 0

1still differ from the canonical view I0 by an arbitrary linear
transformation, and each individual image as a matrix does
not need to be low-rank.
Simultaneous alignment and rectification. For calibration, we need to align all the N images point-wise, and at
the same time each resulting image should be rectified asa low-rank texture. Or more precisely, we want to find thetransformation τ00 , τi0 such that Ii , 1 ≤ i ≤ N can be expressed asIi ◦ (τ00 ◦ τi0 )−1 = Ai + Ei ,Algorithm 1 (Align Low-rank Textures for Calibration).
Input: A rectangular window Ii ∈ Rmi ×ni in each image, initial extrinsic parameter τi , common intrinsic andlens distortion parameters τ0 , and weight λ > 0.
While not converged Do
step 1: for each image, normalize it and compute
the Jacobian w.r.t. unknown parameters:Ii ◦ (τ0 , τi )−1;
kIi ◦ (τ0 , τi )−1 kFIi ◦ (ζ0 , ζi )−1
∂
Ji0 ←
;
∂ζ0 kIi ◦ (ζ0 , ζi )−1 kF ζ0 =τ0 ,ζi =τi
Ii ◦ (ζ0 , ζi )−1
∂
Ji1 ←
;
∂ζi kIi ◦ (ζ0 , ζi )−1 kF ζi =τi ,ζ0 =τ0
Ii ◦ (τ0 , τi )−1 ←where all Ai are low-rank and equal to each other Ai = Aj .
Therefore, the natural optimization problem associated withthis problem becomesstep 2: solve the linearized convex optimization:
min

N
X

min kD1 k∗ + kD2 k∗ + λkEk1 ,kAi k∗ + kEi k1 ,s.t. Ii ◦ (τ0 , τi )−1 + Ji0 ∆τ0 + Ji1 ∆τi = Ai + Ei ;

i=1

s.t.

Ii ◦ (τ00 ◦ τi0 )−1 = Ai + Ei , Ai = Aj .

(7)

One can use optimization techniques similar to that of
TILT and RASL to solve the above optimization problem, such as the Alternating Direction Method (ADM) usedin [17]. However, having too many constraining terms affects the convergence of these algorithms. In addition, in
practice, due to different illumination and exposure time,
the N images could differ from each other in intensity andcontrast. Hence, in this paper, we propose an alternative,
more effective and efficient way to align the images in thedesired way. The idea is to concatenate all the images assubmatrices of a joint low-rank matrix:.
.
D1 = [A1 , A2 , . . . , AN ], D2 = [AT1 , AT2 , . . . , ATN ],.
E = [E1 , E2 , . . . , EN ].
(8)
We try to simultaneously align the columns and rows of
Ai and minimize its rank by solving the following problem:
min kD1 k∗ + kD2 k∗ + λkEk1 ,s.t.

Ii ◦ (τ0 ◦ τi )−1 = Ai + Ei ,(9)with Ai , Ei , τ0 , τi as unknowns. Notice that, by comparing
to Eq. (7), which introduces in N + N (N2−1) constraints, thenew optimization has just N constraints and hence is easier
to solve. In addition, it is insensitive to illumination andcontrast change across different images. One may view theabove optimization as a generalization for both TILT andRASL: When N = 1, it reduces to TILT; and when there is
no D2 , this reduces to something similar to RASL.
To deal with the nonlinear constraints in Eq. (9), we
linearize the constraints Ii ◦ (τ0 , τi )−1 = Ai + Ei w.r.t
all the unknown parameters τ0 , τi . To reduce the effectof change in illumination and contrast, we normalize Ii ◦step 3: update: τ0 ← τ0 + ∆τ0 , τi ← τi + ∆τi ;End While
Output: Converged solution τi , τ0 .
−1◦(τ0 ◦τi )(τ0 , τi )−1 by its Frobenius norm to kIIii◦(τLet−1 k .
0 ◦τi )
F
−1Ii ◦(τ0 ◦τi )∂0Ji = ∂τ0 kIi ◦(τ0 ◦τi )−1 kF be the Jacobian of the normalized image w.r.t. shared
intrinsic and distortion parameters
Ii ◦(τ0 ◦τi )−1∂1
τ0 and Ji = ∂τi kIi ◦(τ0 ◦τi )−1 kF be the Jacobian w.r.t
extrinsic parameters τi for each image. The local linearizedversion of Eq. (9) becomes

min kD1 k∗ + kD2 k∗ + λkEk1 ,s.t.

Ii ◦ (τ0 , τi )−1 + Ji0 ∆τ0 + Ji1 ∆τi = Ai + Ei , (10)with ∆τ0 , ∆τi , Ai , Ei as unknowns. Notice that this linearized problem is a convex optimization problem andcan be efficiently solved by some of the modern highdimensional optimization methods such as the ADM
method mentioned earlier. To find the global solution to theoriginal nonlinear problem Eq. (9), we only have to incrementally update τ0 and τi by ∆τ0 , ∆τi and iteratively rerun
the above program until convergence. The overall algorithm
is summarized in Algorithm 1.
In general, as long as there is sufficient textural variationin the pattern, the lens distortion parameters kc can always
be accurately estimated by the algorithm once the low-rank
texture of the pattern is fully rectified. This is the case evenfrom a single image2 .
Now the remaining question is, under what conditions
the correct intrinsic parameters K and the extrinsic parame2 although a rigorous mathematical proof for this fact is beyond thescope of this paper.

ters (Ri , Ti ) are the global minimum to the problem Eq. (9),and whether there is still some ambiguity.
Proposition 1. Given N ≥ 5 images of the low-rank pattern I0 taken by a camera with the same intrinsic parameters K under generic viewpoints τi = (Ri , Ti ): Ii =I0 ◦ (τ0 ◦ τi ), i = 1, . . . , N . Then the optimal solution(K 0 , τi0 ) to problem Eq. (9) must satisfy K 0 = K and
Ri0 = Ri .
That is, all the distortion and intrinsic parameters τ0 canbe recovered and so is the rotation Ri of each image. Thereis only ambiguity left in the recovered translation Ti of each
image.
With a known pattern. If the ground-truth I0 is given
and its metric is known, then we may want to align Ii
to I0 directly or indirectly. One possible solution is toslightly modify Algorithm 1 by appending D1 , D2 , E with
A0 , AT0 , E0 , respectively, and adding the constraint I0 =A0 + E0 . Another possible solution would be to align thealready rectified textures Ai to I0 by maximizing the correlation.
In both situations, with knowledge about the metric of
I0 , we can uniquely determine Ti and get exactly the fullset of intrinsic and extrinsic parameters.

3.2. Self-Calibration from a Single Image
With a single plane. For most everyday usage of a camera, people normally do not need to know the full intrinsic
parameters of the camera. For instance, for webcam users,
it normally suffices if we can simply remove the annoying
lens distortion. For such users, asking them to take multiple images and conduct a full calibration might be too muchtrouble. Sometimes, we need to remove the radial distortionof an image but without any access to the camera itself.
Therefore, it would be desirable if we can calibrate thelens distortion of a camera from a single image. Normallythis would be impossible for a generic image. Nevertheless,if the image contains a plane with low-rank pattern rich withhorizontal and vertical lines, then the lens distortion kc canbe correctly recovered using our method.
Given a single image with a single low-rank pattern,since we cannot expect to obtain all the intrinsic parameters correctly, we can make the following simplifying assumptions about K: No skew θ = 0, principal point known(say set at the center of the image), and pixel being square(fx = fy = f ). Although these seem to be somewhat restrictive, they approximately hold for many cameras madetoday. In this circumstance, if the viewpoint is not degenerate, applying the algorithm to the image of this single pattern correctly recovers the lens distortion parameters kc andthe focal length f .

With two orthogonal planes. Very often, an image contains more than one planar low-rank textures, and they satisfy additional geometric constraints. For instance, in a typical urban scene, an image often contains two (orthogonal)
facades of a building. Each facade is full of horizontal andvertical lines and can be considered as a low-rank texture.
In this case, the image encodes much richer information
about the camera calibration: Both the focal length and theprincipal point can be recovered from such an image, giventhat the pixel of the camera is assumed to be square, i.e.,
fx = fy = f , and there’s no skew, i.e., θ = 0.
For simplicity, we let the intersection of these two orthogonal planes be the z-axis of the world frame, and thetwo planes are x = 0 and y = 0, each with a low-rank tex(i)ture I0 , i = 1, 2. We take a photo of the two planes by acamera with intrinsic parameters K and lens distortion kc ,from the viewpoint (R, T ). Denote the photo as I, whichcontains two mutually orthogonal low-rank patterns.
Let ML = [0 Y1 Z1 ]T ∈ R3 be a point on the left
facade, and MR = [X2 0 Z2 ]T ∈ R3 be a point on theright facade, and let mL , mR ∈ R2 be the corresponding
images on I. Then we have:
λ1 m̃L =f 0 ox0 f oy0 0 1



f 0 ox0 f oy0 0 1

[r2 r3 T1 ]h Y1 i
Z1
1

,

(11)

.

(12)

and

λ2 m̃R =[r1 r3 T2 ]h X2 iZ2
1Here we have used a different translation T1 or T2 for eachplane, mainly because otherwise we must exactly find theposition of the intersection of the two planes, which is beyond the scope of this paper. So in this circumstance let
τ0 = [f, ox , oy , kc (1 : 5), ω] and τi = [Ti ] and the optimization problem we need to solve to recover them is:
minAi ,Ei ,τ0 ,τi

kA1 k∗ + kA2 k∗ + λ(kE1 k1 + kE2 k1 ),subject toI ◦ (τ0 , τi )−1 = Ai + Ei .

(13)

With similar normalization and linearization techniques, we
can solve this problem with slight modification to Algorithm 1.
Proposition 2. Given one image of two orthogonal planes
with low-rank textures, taken by a camera from a genericviewpoint (R, T ) with intrinsic parameters K with zeros
skew θ = 0 and square pixels(fx = fy ). If K 0 , R0 , T10 , T20
are solutions to problem Eq. (13), then K 0 = K, R0 = R.
By an argument similar to the multiple-image case, torecover τ0 , we only need to rectify the left and right textureswith a joint parameterization of the rotation.

3.3. Implementation
The initialization of our algorithm is extremely simple
and flexible. The location of the initial window can be obtained from any segmentation methods that approximately
detect the region of the pattern. Or it can be easily specified
by a human. There is no need for the location of the initial
window to be exact or even cover the pattern region. The
proposed method is very robust and can converge precisely
to the pattern.
For initialization, we first run the TILT on each initial
window to approximately extract the homography Hi for
the i-th image. Then we can obtain a rough estimate ofK, R, T as from the vanishing points given by the first twocolumns of τi .3 For lens parameters, even if large lens distortion is present, we set their initial values to be zero.
To make the convergence region of our algorithm large
and to accelerate the algorithm, we employ the conventional
multi-resolution implementation with a proper blurring andpyramid scheme, similar to that described in the work on
TILT [17].

4. Simulations and Experiments
4.1. Calibration from Multiple Images
A. Calibration using a known pattern. In this experiment, we compare our proposed method with the standard
camera calibration toolbox [1]. Normally, the error of calibration can be evaluated by re-projection error of extracted
feature points. But since our method does not involve any
feature extraction and uses only the raw image pixels, thismeasurement of error is no longer suitable here. Instead, we
try to compare the accuracy of estimated camera parametersagainst the average estimates. More precisely, we run multiple experiments with different images by the same camera
and compute the standard deviation for every parameter weestimate. The smaller the deviation, the more stable the estimates are.
In this experiment, we take 50 photos of a known
checker-board pattern using the same camera (a Panasonic
HDC-HS9) and setting, from different viewpoints. In eachexperiment, we randomly select 20 out of the 50 images.
With these selected images, we calibrate the camera withboth the proposed method and the standard toolbox. Notethat we need to manually click the precise location of thefour corners of the checker-board for the toolbox. But forour method, the initialization needs not to be exact at all(several pixels away). See Fig. 2 (b) for examples. We
repeat the experiment 20 times, and calculate the standard
derivation of each parameter4 for each method. The result
3 It is easy to see that the first two columns of τi correspond to thevertical and horizontal directions of the low-rank textures.
4 By default the calibration toolbox disables the estimation of skew.
When turned on, it gives an error. So in this comparison, we do not esti-(a) Initialization for [1] (b) Initialization of our (c) Aligned images by our
methodmethod.

(d) Comparison of intrinsic params(e) Comparison of distortion params

Figure 2. Top: Representative examples of initialization for thetwo methods. Notice that ours can be very flexible. Middle: Precisely aligned images by our algorithm. Bottom: Comparison
with the standard calibration toolbox. Standard deviation in theestimated parameters, in pixels.

is shown in Fig. 2 (d)(e).
From the figure, we can see that our method is more sensitive in the estimation of focal length and principle points
than toolbox, but the performance is comparable.5 The estimation of lens-distortion parameters of our method is almost the same as the toolbox.
So to conclude from this experiment, under noise-free,well-controlled conditions, the performance of our method
is quite comparable to the toolbox. But our method does
not require exact initialization of the point location. In later
experiments, we will see that our method can work undermuch broader conditions: with an unknown pattern, froma single image, and even when significant lens distortion
exists, such as fish-eye images.
In order to verify the accuracy of the remaining experiments, unless otherwise stated, we use the same Panasoniccamera with the same image resolution [960, 540] (directly
down-sampled from its full [1920, 1080] resolution). The
camera parameters estimated from this experiment is:
h 1142.0 0 477.7 i
01138.0 270.3 .
K=
(14)
0

0

1

The camera should have the same set of parameters, except
mate skew either although our method does not have this limitation.
5 We have made no serious attempts to improve the numerical accuracyof our method other than a straightforward implementation of Algorithm 1.
We believe ours can be easily improved by a more careful numerical implementation and post processing in the future.

(a) Input images and windows(b) Rectified and aligned textures

Figure 3. Camera calibration from images of an unknown pattern. The algorithm aligns the low-rank textures precisely despite
specularities in the images. Although aligned and rectified, corresponding pixels do not have to correspond to the same 3D point.

(a) Input image with initial (green) and fi- (b) Rectified texture (from red)nal converted (red) window

Figure 5. Camera calibration using an image of this paper.

(a) Input image and initial windows (b) Rectified image and final windows(a) Input image and window(b) Radial distortion removed

Figure 4. Calibration from a single image in the Toolbox.

for focal length which may change from experiment to experiment.
B. Calibration from an unknown pattern. In this experiment, we take multiple photos of an unknown mosaic wallfrom different viewpoints. By rectifying and aligning themosaic images pixel-wise into a common canonical viewusing Algorithm 1, as shown in Fig. 3, we obtain the camera calibration. The recovered intrinsic matrix is:
h 1138.6 0 482.3 i
01127.8 267.7 .
K̂ =
(15)
0

0

1

C. Calibration from a single pattern. Given just a single image or a regular pattern, to calibrate the camera, wehave to work with fairly strong assumptions, say that the
principal point is known (and simply set as the center of theimage) and the pixel is square. Then from the image, one
can calibrate the focal length as well as eliminating the lensdistortion. Fig. 4 shows an example with an image given in
the standard toolbox. The estimated intrinsic parameters K̂
and the ground-truth K (provided by the calibration toolbox) are respectively:
h 677.1812
i h 661.6700
i
0
319.50000306.0959
0677.1812 239.5000 ,0662.8285 240.78987 .
0

1

0

0

1

(d) Rectified texture of right facade

Figure 6. Calibration from two orthogonal facades of a building.
Notice that the method works despite of specularities and occlusions due to tree branches.

The small error in focal length is mainly due to that the
principle point is approximated with the center of the image.
Nevertheless, we see in Fig. 4(b), the radial distortion iscompletely removed by our algorithm.
To show the flexibility of our method, we further show
another example in Fig. 5 where we took with the Panasoniccamera an image of the frontal page of this paper. With thisimage as input, the recovered calibration matrix is:
h 1229.1 0 479.5 i
01229.1 269.5 .
K̂ =
(16)
0

4.2. Calibration from a Single Image

0(c) Rectified texture of left facade

01

D. Calibration from two orthogonal planes. In this section, we present the results of calibrating a camera from observing two orthogonal facades of a building. The result isshown in Fig. 6 with the estimated calibration parameters:
h 1189.7 0 474.3 i
01189.7 273.4 .
K̂ =
(17)
0

0

1

E. Rectifying fisheye images. Note that the image usedin Fig. 4 is taken from a standard example in the MAT-

mediately useful for many practitioners.

References

Figure 7. Rectify fisheye images with significant lens distortion.
Top: input images with a selected initialization windows (green);
Middle: rectified images with final converged windows (red);
Bottom: rectified low-rank textures.

LAB Calibration Toolbox [1], presumably the one withthe largest radial distortion among all the examples. Our
method can actually handle distortion far beyond that, as weshow in this section with images taken by a typical fisheye
camera (not the Panasonic anymore).
There have been many parametric models proposed forthis kind of images, our method should apply as long as themodel is known. Here to illustrate the basic idea, we makethe simplifying assumptions that there is only 1-D distortion
along the radial direction and hence try to estimate the mapping along the radius before and after distortion r = f (rd ).
We approximate f (·) by polynomials up to degree of four.
In addition, we assume the center of the distortion is thecenter of the image. Moreover, since no prior knowledge
about K, R, T is available, we model the transformationfrom the pattern to the image plane as a general homography transformation H ∈ R3×3 . Some representative results
are shown in Fig. 7.

5. Conclusions
In this paper, we have provided a simple, accurate androbust method for camera calibration based on an advanced
rank minimization technique. Our method naturally avoids
the need of accurate corner or line detections; instead, we
use all pixels in the specified region. This property removes
the necessity of careful human operation, therefore it is im-[1] J.-Y. Bouguet. Camera calibration toolbox for Matlab.
http://www.vision.caltech.edu/bouguetj/
calib_doc/. 1, 2, 6, 8
[2] D. Brown. Close-range camera calibration. Photogrammetric Engineering, 37(8):855–866, 1971. 2[3] B. Caprile and V. Torre. Using vanishing points for camera
calibration. IJCV, 4(2):127–140, Mar 1990. 2[4] R. Cipolla, T. Drummond, and D. Robertson. Camera calibration from vanishing points in images of architecturalscenes. In Proc. of British Machine Vision Conf., volume 2,
pages 382–391, 1999. 2[5] F. Devernay and O. D. Faugeras. Automatic calibration
and removal of distortion from scenes of structured environments. In SPIE Conference on Investigative and Trial Image
Processing, volume 2567, pages 62–72, 1995. 2[6] W. Faig. Calibration of close-range photogrammetry systems: Mathematical formulation. Photogrammetric Engineering and Remote Sensing, 41(12):1479–1486, 1975. 2[7] O. Faugeras. Three-Dimensional Computer Vision: a Geometric Viewpoint. MIT Press, 1993. 2, 3
[8] O. Faugeras, Q. Luong, and S. Maybank. Camera selfcalibration: Theory and experiments. In ECCV, pages 321–
334, 1992. 2[9] L. Grammatikopoulos, G. Karras, E. Petsa, and I. Kalisperakis. A unified approach for automatic camera calibrationfrom vanishing points. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, XXXVI(5), 2006. 2[10] D. Liebowitz and A. Zisserman. Metric rectification for perspective images of planes. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, pages 482–488, June
1998. 2[11] S. Maybank and O. Faugeras. A theory of self-calibration ofa moving camera. IJCV, 8(2):123–152, Aug. 1992. 2[12] Y. Peng, A. Ganesh, J. Wright, W. Xu, and Y. Ma. RASL:
Robust alignment by sparse and low-rank decomposition for
linearly correlated images. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, 2010. 3[13] P. Sturm and S. J. Maybank. A method for interactive 3D reconstruction of piecewise planar objects from single images.
In Proc. of British Machine Vision Conf., 1999. 2[14] R. Tsai. A versatile camera calibration technique for highaccuracy 3D machine vision metrology using off-the-shelf tv
cameras and lenses. IEEE Journal of Robotics and Automation, 3(4):323–344, Aug. 1987. 2[15] Z. Zhang. Flexible camera calibration by viewing a planefrom unknown orientations. In Proc. of Int’l Conf. on Computer Vision, 1999. 2[16] Z. Zhang. Camera calibration with one-dimensional objects.
PAMI, 26(7):892 –899, 2004. 2[17] Z. Zhang, X. Liang, A. Ganesh, and Y. Ma. TILT: Transforminvariant low-rank textures. In Proc. of Asian Conference on
Computer Vision, 2010. 1, 3, 4, 6

A. Proof of Proposition 1: Ambiguities in Calibration with an Unknown Pattern
Proof. Suppose by solving Eq. (9), we have aligned all theimages up to translation and scaling of I0 . To be more specific we have managed to find τi0 = (Ri0 , Ti0 ), τ00 = (K 0 , kc0 )such that
sx 0 mx
00 −1
Ii ◦ (τ0 ◦ τi ) = I0 ◦ τ, with τ =  0 sy my  .
0 01
As all the lines have become straight in the recovered images Ai , the radial distortion parameters kc0 should be exact
kc0 = kc .
Here sx , sy are scaling in the x and y directions of thealigned images Ai w.r.t. the original low-rank pattern I0 .
mx and my are the translations between Ai and I0 . Now let
us consider the mapping between a point M0 on I0 (noticethat the Z-coordinate is zero by default) and its image m̃ ∈R3 (in homogeneous coordinates): λm̃ = K[r1 , r2 , T ]M0 .
As the recovered parameters are consistent with all constraints, the same point and its image satisfy:
sx 0 mx
λ0 m̃ = K 0 [r10 , r20 , T 0 ]  0 sy my  M0 .
0 0
1
So the matrix K[r1 , r2 , T ] must be equivalent toK 0 [sx r10 , sy r20 , mx r10 +my r20 +T 0 ] (i.e., up to a scale factor),so we have
 0−1
Kr1 = ξsx K 0 r10 ,K Kr1 = ξsx r10 ,
⇒
(18)
0 0
Kr2 = ξsy K r2 .
K 0−1 Kr2 = ξsy r20 .
Since r10T r20 = 0, we have
(Kr1 )T K 0−T K 0−1 (Kr2 ) = 0.

(19)This gives one linear constraint for B = K 0−T K 0−1 . Sucha symmetric B has six degrees of freedom. Since each image gives one constraint on B, we need only five generalimages (not in degenerate configurations) to recover B up
to a scale. Since K −T K −1 is a solution too, thus we musthave K 0 = K as the unique solution of the form Eq. (2).
Further from Eq. (18), we have r10 = r1 , r20 = r2 , andsx = sy . That is, once all the images are aligned and rectified, they only differ from the original pattern I0 by a global
scale s = sx = sy and a translation (mx , my ). In addition,
we recovered rotation Ri0 is the correct Ri0 = Ri . But since
we still do not know the exact values of sx , mx , and my ,the recovered Ti0 is not necessarily the correct Ti .
With a similar analysis, we can show that in fact if we individually rectify the images, we still can obtain the correct
K and Ri . The only difference is that sx , sy , mx and myare all different for different images, thus the translations Ti
are even less constrained.

B. Determine Translation from Ground-Truth
If the low-rank pattern I0 is given, we can directly or
indirectly align Ii to I0 . From a derivation similar to theabove, one can show that we can recover sx , mx and mywith respect to the ground truth metric of I0 . Then for each
image the ground-truth translation can be recovered byT =mx r1 + my r2 + T 0
.
sx

(20)

C. Proof of Proposition 2: Ambiguities in Calibration with Two Orthogonal Planes
Proof. Suppose a low-rank texture lies on the left plane
X = 0 and another lies on the right plane Y = 0.
ML = (0, Y1 , Z1 ) is the point on the left plane, and
MR = (X2 , 0, Z2 ) is a point on the right plane. Similarly
we have the image point mL = (xL , yL ) of the left point
and mR = (xR , yR ) of the right point. Then 
f 0 ox
Y1
λ1 mL =  0 f oy  [r2 r3 T ] Z1  ,
(21)
0 0 1
1
and


f
λ2 mR =  0
0

0
f
0


 
ox
X2
oy  [r1 r3 T ]  Z2  .
1
1

(22)

For convenience, we use (x, y) to represent points bothon Y = 0 and on X = 0. Suppose the rectified im(i)age Ai differs from the ground truth I0 by scaling and(i)(i)(i) (i)translation: sx , sy , mx , my . Then the ground truth K,
R = [r1 r2 r3 ] and T , and the recovered parameters K 0 ,R0 = [r10 r20 r30 ] and T10 , T20 are related through the following
formulae:
i
h
(1)
(1)
(1)
(1)
sx Kr2 sy Kr3 K(mx r2 + my r3 + T )
0 0
0 0
0 0
=
i (23)h ξ1 [K r2 K r3 K T1 ] ,
(2)
(2)
(2)
(2)
sx Kr1 sy Kr3 K(mx r1 + my r3 + T )= ξ2 [K 0 r10 K 0 r30 K 0 T20 ] .
This gives
K 0−1 Kh s(2)
x

ξ2

(1)

r1 ,

(1)

i
sx
sy
r2 ,
r3 = [r10 , r20 , r30 ].
ξ1
ξ1

(24)

Knowing that r10 , r20 , r30 are orthogonal to each other, we derive three linear constraints for B = K 0−T K 0−1 , which has
three unknowns. So in general, we can extract unique solution K 0 from B. Note that K 0 = K is one solution too,hence the recovered is the correct solution.
Also, from Eq. (24) we can see that R0 = R, leaving
only Ti being ambiguous.