Automatic Camera and Range Sensor Calibration using a single Shot
Andreas Geiger, Frank Moosmann, Ömer Car and Bernhard Schuster

Abstract— As a core robotic and vision problem, camera
and range sensor calibration have been researched intensely
over the last decades. However, robotic research efforts still
often get heavily delayed by the requirement of setting up
a calibrated system consisting of multiple cameras and range
measurement units. With regard to removing this burden, we
present a toolbox with web interface for fully automatic camerato-camera and camera-to-range calibration. Our system is easy
to setup and recovers intrinsic and extrinsic camera parameters
as well as the transformation between cameras and range
sensors within one minute. In contrast to existing calibration
approaches, which often require user intervention, the proposed
method is robust to varying imaging conditions, fully automatic,and easy to use since a single image and range scan proves
sufficient for most calibration scenarios. Experimentally, wedemonstrate that the proposed checkerboard corner detector
significantly outperforms current state-of-the-art. Furthermore,
the proposed camera-to-range registration method is able todiscover multiple solutions in the case of ambiguities. Experiments using a variety of sensors such as grayscale and color
cameras, the Kinect 3D sensor and the Velodyne HDL-64 laser
scanner show the robustness of our method in different indoor
and outdoor settings and under various lighting conditions.

I. INTRODUCTION AND RELATED WORK
Robots are typically equipped with multiple complementary sensors, which require calibration in order to represent
sensed information in a common coordinate system. Hereby,each sensor can be characterized by its intrinsic (e.g. shape ofthe camera lens) and extrinsic (i.e. pose) parameters. Calibration methods aim at estimating these parameters, often using
checkerboard patterns as targets. Although calibration is anubiquitous problem in robotics and computer vision, current
camera calibration tools such as the widely used Matlab
Camera Calibration Toolbox [1] or OpenCV [2] are not very
robust and often require manual intervention, leading to acumbersome calibration procedure. Furthermore, only little
work on camera-to-range sensor calibration has been done,
and to our knowledge [3] and [4] are the only toolboxes
online available.
In this work, we try to close this gap by proposing a robust
solution to automatically calibrating multiple cameras and
3D range sensors with respect to each other. Our approachrelies on a cheap and simple calibration setup: We attach
multiple printed checkerboard patterns at the walls and thefloor, see Fig. 4 for an illustration. As input, our method
requires a single range or camera image per sensor (which wecall shot in the following), as well as the distance between the
inner checkerboard corners for resolving the scale ambiguity.
All authors are with the Department of Measurement andControl, Karlsruhe Institute of Technology. The first two
authors contributed equally to this work. Primary contact:{geiger,frank.moosmann}@kit.edu.

Fig. 1. Experimental setup. Trinocular camera with Velodyne HDL-64E
laser scanner (left) and binocular camera with Microsoft Kinect (right).

Note that this differs from previous approaches [1], [2], [3],[4] which require multiple (synchronized) images of a single
calibration target presented at different orientations, as wellas the number of checkerboard rows and columns as input.
The only assumption we make is that all sensors return either
intensity or depth images and share a common field of view.
Experimentally we show the robustness of our method
on calibration images from the internet, a binocular camera
setup with the Kinect range sensor and a trinocular camera
setup with the Velodyne HDL-64E laser scanner, illustrated
in Fig. 1. We will also make an online version of our system1
available to the community: After uploading images and 3D
point clouds, the calibration parameters are computed within
one minute.
This work is organized as follows: The next section starts
with a discussion of related work. Sec. III and Sec. IV givea detailed description of the proposed method on camera-tocamera and camera-to-range calibration, respectively. Afteran evaluation in Sec. V the paper is concluded in Sec. VI.
II. RELATED WORK
A. Camera-to-Camera Calibration
The most widely used toolbox for camera calibration
is probably Bouget’s Matlab Camera Calibration Toolbox[1], of which a C++ version has been implemented in the
OpenCV library [2]. It uses the distortion model describedin [1], [5], [6], covering a large variety of lenses, andallows to calibrate up to two video cameras intrinsically andextrinsically by presenting a checkerboard calibration pattern
at multiple orientations. The OpenCV additionally offers anautomatic corner detection method based on quadrangles,
which has been extended and improved upon by Rufli etal. [7]. Due to the method’s sensitivity to clutter in the input
image, Kassir et al. [4] propose a more robust corner detector
1 www.cvlibs.net

based on multi-scale Harris [8] points and an improved
filtering mechanism inspired by [9]. From the recovered
corner points, they are able to detect a single checkerboard
within an image. The problem of multi-camera calibration
for the distortion-free scenario has been tackled in [10].
While all of the existing methods concentrate on the problem of detecting only a single (and often known) calibration
target, our approach finds multiple unknown checkerboard
patterns in an image and automatically matches them between cameras. This facilitates the calibration setup as asingle image is sufficient for calibration and unsynchronizedsensors such as line sweep laser scanners (e.g. rotating SICK)
can be readily employed. Furthermore, our method requires
no manual intervention, and the proposed corner detector
significantly outperforms current state-of-the art [4], [9], as
illustrated by our experiments in Sec. V-A.

(a) Corner Prototype 1(b) Corner Prototype 2(c) Input Image I(d) Corner Likelihood C(e) Orientation & Score(f) Detected Checkerboard Corners

B. Camera-to-Range Calibration
For camera-to-range sensor calibration, all existing works
assume a valid intrinsic calibration of the camera and focus
on estimating the six parameters needed to align the twocoordinate systems. The approaches can be classified into
two groups, based on the type of range sensor used:
The first group of approaches uses full 3D sensors giving
rise to a dense 3D point cloud. Unnikrishnan et al. [3]have developed a toolbox where the user has to mark thecalibration pattern in an interactive GUI. The approach of
Scaramuzza et al. [11] is comparable but specialized toomni-directional cameras. Likewise, correspondences must
be selected manually.
The second and more established group uses one- or fourlayer laser scanners. Since these sensors only measure asmall part of the scene, most approaches rely on manual
selection of correspondences [12], [13], [14]. Only recently
this effort was reduced by either classifying line-segments[4] or by exploiting reflectance measurements from the lidar
scanner [15]. Unfortunately, these approaches are not directly
applicable to 3D range sensors.
Common to all approaches is the use of only one calibration pattern. This requires to take several recordings
with the need for manually moving the calibration pattern.
In this paper we propose a robust method for 3D range
sensors which determines all six extrinsic parameters fully
automatically using a single shot only. Hereby, calibration
efforts are reduced dramatically.
III. CAMERA-TO-CAMERA CALIBRATION
Following [16], we use planar checkerboard patterns as
calibration targets for multiple reasons: They are cheap
to employ, corners can be localized with high sub-pixel
accuracy and structure recovery profits from strong edges
between corners. In contrast to existing methods which
capture multiple images of a single checkerboard, we takea single shot of multiple checkerboards placed at different
locations in the image. This also facilitates the calibration
process, especially with respect to registering the triangulated
checkerboards to a 3D point cloud, as detailed in section IV.

Fig. 2. Corner detection. We filter the input image I using corner prototypes, apply non-maxima-suppression on the resulting corner likelihood C
and verify corners by their gradient distribution. See Sec. III-A for details.

Through not strictly necessary, we assume the prevalent calibration scenario where all sensors have a part of their field ofview in common. Our camera calibration algorithm proceeds
as follows: First, we robustly locate checkerboard corners
in the image (Sec. III-A) and refine them for sub-pixel
accuracy (Sec. III-B). Checkerboard structures are recoveredby optimizing an energy function subject to structural constraints (Sec. III-C). Correspondences between images are
obtained by deterministically sampling affine transformations
and maximizing a fitness function (Sec. III-D). The final
camera parameters are recovered via non-linear optimization,
where we use weak regularizers to avoid degenerate solutions(Sec. III-E). The following subsections give details about
each of these steps.
A. Corner Detection
Harris points [8] or Shi-Tomasi corners [17] are a common
choice for localizing junctions in an image. However, wefound that the following procedure gives more robust resultswith respect to image clutter, blurring artifacts and localization accuracy: In order to locate checkerboard corners in agrayscale image I (Fig. 2(c)), we compute a corner likelihoodat each pixel in the image using two different n × n cornerprototypes: One for axis-aligned corners (Fig. 2(a)) and one
for corners, which are rotated by 45◦ (Fig. 2(b)). Empirically,
we found that these two simple prototypes are sufficient for
detecting corners over a wide range of distortions induced byperspective transformations. Each prototype is composed offour filter kernels {A, B, C, D}, which are convolved with
the input image I. For an ideal corner, the response of {A, B}

should be greater than the mean response of {A, B, C, D},
while the response of {C, D} should be smaller, and viceversa for flipped corners. This fact can be expressed formallyias follows: Let fX
be the filter response of kernel X andprototype i for a particular pixel. The corner likelihood cat this pixel is defined by taking the maximum over all
combinations of prototypes and flippings:c =max(s11 , s12 , s21 , s22 )

si1
si2

i
min(min(fAi , fBi ) − µ, µ − min(fCi , fD
))
i
i
i
i
min(µ − min(fA , fB ), min(fC , fD ) − µ)i0.25 (fAi + fBi + fCi + fD
)

=
=

µ =

(1)

Here, si1 and si2 denote the likelihood of the two possible
flippings for prototype i. Computing this likelihood for every
pixel in the image yields a corner likelihood map C. See
Fig. 2(d) for an illustration. Importantly, note that the above
definition leads to a low likelihood c, if any of the four filter
kernels responds weakly. This is important for removing as
many non-checkerboard style corners as possible from the
hypotheses space. To produce a list of corner candidates,
we apply conservative non-maxima-suppression (with parameters nnms and τnms ) [18] on C, followed by verifying
the candidates by their gradient statistics in a local n × npixel neighborhood, as illustrated in Fig. 2(e): We computea weighted orientation histogram (32 bins) from Sobel filter
responses and find the two dominant modes α1 and α2 using
mean shift [19]. Based on the edge orientations, we constructa template T for the expected gradient strength k∇Ik2 . The
product of T ∗ k∇Ik2 and the corner likelihood in (1) gives
the corner score, which we threshold by τcorner for obtaining
a final list of corner candidates, see Fig. 2(f). Here, ’∗’denotes the normalized cross-correlation operator.
B. Sub-pixel Corner and Orientation Refinement
It is well-known that calibration benefits from sub-pixel
accurate corner locations [20], [21], [2]. In this work we
refine both, the corner location and the edge orientations.
For sub-pixel corner localization, we make use of the factthat at a corner location c ∈ R2 the image gradient gp ∈R2 at a neighboring pixel p ∈ R2 should be approximately
orthogonal to p − c, leading to the optimization problemX2
c = arg min
gpT (p − c0 )(2)0cp∈NI (c0 )where NI is a local 11 × 11 pixel neighborhood around thecorner candidate. Note that neighboring pixels are automatically weighted by the gradient magnitude. This problem isstraightforward to solve in closed form, yielding the solution:
XXc=(
gp gpT )−1(gp gpT )p
(3)
p∈NI

p∈NI

To refine the edge orientation vectors e1 ∈ R2 and e2 ∈ R2 ,we seek to minimize the error in deviation of their normals
with respect to the image gradientsXT
ei = arg min(gpT e0i )2 s.t. e0 i e0i = 1
(4)
0
ei

p∈Mi

(a) Iterative Expansion of Checkerboard Hypotheses from Seed Points(b) Triples(c) Detected Checkerboards on Large-Scale Example

Fig. 3. Structure recovery. We iteratively expand seed points (a) into thedirection of the strongest gradient of an energy function evaluating the local
structuredness (b). Fig. (c) shows final detection result.

where Mi = {p | p ∈ NI ∧ |mTi gp | < 0.25} is the setof neighboring pixels, which are aligned with the gradientmi = [cos(αi ) sin(αi )]T of mode i. The solution to Eq. 4
is obtained, by setting the derivative of its Lagrangian tozero, leading to an eigenvalue problem, with ei being the
eigenvector corresponding to the smallest eigenvalue of!
X gp1 gpT
∈ R2×2
(5)
2 T
g
g
p p
p∈Mi
where gpi denotes the i’th entry of gp .
C. Structure Recovery
Let the set of corner candidates be X = {c1 , .., cN } andlet Y = {y1 , .., yN } be the corresponding set of labels. Here,y ∈ {O} ∪ N2 represents either an outlier detection (O)or the row / column (N2 ) within the checkerboard. For all
checkerboards present in the image our goal is to recover Y
given X . We do this by minimizing the energy function
E(X , Y) = Ecorners (Y) + Estruct (X , Y)

(6)subject to the constraint, that no two labels can explain thesame checkerboard corner. Intuitively, we try to explain as
many corners as possible using a regular structured element(the checkerboard): Ecorners (Y) = −|{y|y 6= O}| is taken
as the negative number of explained checkerboard corners
and Estruct measures how well two neighboring corners iand j are able to predict a third one k, weighted by the
number of explained corners:||ci + ck − 2cj ||2||ci − ck ||2(7)
Here, T denotes the set of all row and column triples of thecurrent checkerboard configuration induced by Y, see Fig.
3(b) for an illustration. In total, we have |T | = m(n − 2) +n(m − 2) triples, with m and n denoting the number of rows
and columns respectively. Importantly note that due to the
local nature of our linearity requirement in Eq. 7, we gain
flexibility and also allow for strongly distorted patterns as
imaged by fisheye lenses, for instance.
Estruct (X , Y) = |{y|y 6= O}| max(i,j,k)∈T

Since the set of possible states Y can take is exponentiallylarge in N , exhaustive search is intractable. Instead, we
employ a simple discrete optimization scheme, which works
well in practice as confirmed by our experiments in Sec. V:
Given a seed corner, we search for its closest neighbors
in the direction of its edges e1 and e2 , yielding an initial
2 × 2 checkerboard hypothesis with an associated energy
value E(X , Y). To optimize E(X , Y), we propose expansion
moves on Y, which expand any of the checkerboard borders
by a single row or column. Amongst all four possibilities,
we select the proposal, which reduces E(X , Y) the most.
Fig. 3(a) illustrates the expansion moves exemplarily.
In order to recover multiple checkerboards in a single
image, we repeat the aforementioned procedure for every
corner in the image as seed, yielding a set of overlapping
checkerboards. Duplicates (having at least one corner index
in common) are removed greedily by keeping only the top
scoring candidates with respect to E(X , Y), starting with thehighest scoring checkerboard first. Fig. 3(c) shows an image
with 11 discovered checkerboards exemplarily.

principal point (cu , cv ) at the center of the image and setting
α = k1 = k2 = k3 = k4 = k5 = 0. In practice wefound this procedure to yield more robust results than closedform solutions, such as the one proposed by Zhang et al.
[24], especially in the presence of only a small number ofcheckerboards. The extrinsic parameters are initialized by averaging the checkerboard-to-image plane homographies [24].
To carry out the final non-linear refinement, we minimize
the sum of squared corner reprojection errors using GaussNewton optimization. Since we found the sixth order radial
distortion coefficient k5 hard to observe, we add a quadratic
regularization term to prevent k5 from getting too large.
Alternatively, individual distortion parameters can be fixed
to 0 in our toolbox.
IV. CAMERA-TO-RANGE CALIBRATION

Having recovered checkerboards in all camera images, weare left with the problem of finding corner correspondences
between cameras. We do this by defining one camera as
the reference camera, and independently match all othercamera images against this reference image. Due to appearance ambiguities, classical feature descriptors such as
SIFT [22] or SURF [23] can not be employed. Instead, we
consider all possible combinations of two checkerboards inboth images (resulting in a loop over four variables), fromwhich we compute the corresponding unique 2D similarity
transformation ϕ(p; A, b) = Ap+b. Here, ϕ takes a point p
in the target image and projects it into the reference image bychanging translation, rotation and scale. Given ϕ, we assign
all checkerboards of the target image to their closest neighbors in the reference image and resolve the two (rectangularpattern) / four (quadratic pattern) fold checkerboard rotation
ambiguity by taking the minimizer of the corner projection
errors independently for each matched checkerboard. Here,
we only assign checkerboards, which agree in the numberof rows and columns and for which the relative projection
error is smaller than τmatch , measured relative to the image
width. From all samples, we choose the final solution as theone which maximizes the number of matched checkerboards.
Fig. 8 shows the final matching results for different scenarios.

Goal of this section is to estimate the 6-DOF rigid transformation parameters θ = (rx , ry , rz , tx , ty , tz )T specifying
the relative pose of the reference camera coordinate systemwrt. the coordinate system of a range sensor. A 3D pointin camera coordinates pc can then be transformed into
range sensor coordinates via pr = Rθ · pc + tθ using thecorresponding rotation matrix Rθ and translation vector tθ .
Input from the range sensor is an unordered set of 3D
points Pr = {(x, y, z)} which keeps the approach as
generic as possible. Similarly, we obtain sets of 3D points
Pcj = {(x, y, z)} from the camera-to-camera calibration
stage, where each set j corresponds to one checkerboard andcomprises the corner locations in 3D. In the following, we
describe how both sets of points can be aligned, giving riseto the transformation parameters θ.
Existing methods [14], [13], [11], [3], [12] simplify
camera-to-laser calibration by manual user intervention andthe fact that several scans are taken with only one board
visible at a time. Our system is designed for easy usage
with the consequence of having a more difficult situation:
We can neither rely on correspondences between the twopoint clouds, nor is it possible to use existing 3D point cloudalignment methods due to the small overlap (full world vs.
checkerboards only) and the unknown initialization.
The proposed algorithm proceeds as follow: First, segmentation is leveraged to identify planes in the range data
(Sec. IV-A). Next, transformation hypotheses are generated
by random plane associations (Sec. IV-B). The best ones
are refined and verified (Sec. IV-C). A final non-maxima
suppression step yields all feasible solutions (Sec. IV-D).

E. Optimization

A. Segmentation

Following [1], [2], we assume a pin-hole camera
model with radial and tangential lens distortion as described in [1], [5], [6]. In total we have 10 intrinsic(fu , fv , cu , cv , α, k1 , k2 , k3 , k4 , k5 ) and 6 extrinsic parameters for each camera / reference camera combination. For
optimization, we extended the Matlab Camera Calibration
Toolbox [1] to handle an arbitrary number of cameras. We
initialize the intrinsic parameters of each camera independently by exhaustively searching for fu and fv , placing theSegmentation aims at finding contiguous regions (or segments) Prj = {(x, y, z)} ⊆ Pr in the range data that
represent a plane and could potentially correspond to acheckerboard mounting. Note that there might be several
checkerboards sticking to the same wall.
As a preprocessing step, we calculate normal vectors nir ∈3
R for each point pir ∈ Pr by principal component analysis
on the K nearest neighbors NK,Pr (pir ). Segmentation iscarried out by greedily growing regions from random seed

D. Matching Checkerboards between Images

Algorithm 1 Global registration
1: GenerateTransformations({sjc }, {sjr }, Pr ):2: G ← {}; S ← {};3: while not enough(G) do
4:(sac , sbc , scc ) ← randomselect({sjc })5:(sar , sbr , scr ) ← randomselect({sjr })6:θ ← minimize(sac , sbc , scc , sar , sbr , scr )7:s ← score(sac , sbc , scc , θ, Pr )8:
G, S ← selectbest(G ∪ θ, S ∪ s)
9: end while
10: return G(a) Input camera image(b) Input range data and calibration result
Fig. 4.

Calibration example. Camera with Velodyne HDL-64E.

points pjr ∈ Pr , each generating the corresponding set Prj .
A point pir is added to the region Prj iff it is a neighbor andits normal vector is similar to the seed’s normal:
i
m
j
iT j
∃ pm
r ∈ NK,Pr (pr ) : pr ∈ Pr ∧ nr nr > τsegment(8)

Each region is grown until it converges, then removed fromPr , and a new seed is selected until no more points areleft in Pr . A final filtering step removes segments which areeither significantly smaller than a checkerboard or not planar
enough.
Note that the above algorithm is nondeterministic: Seedpoints are chosen randomly and each region-expansion isdependent on the seed. Experiments show that the outcome
is nevertheless very stable. Compared to using local decisions
this can guarantee that resulting segments are planar.
B. Global Registration
Given a set of planar point clouds {Prj } and {Pcj } from therange sensor and camera respectively, this section generates a
set of initial transformations G = {θ i }. Therefore, each pointjcloud region Pc/l
is transformed into a disc-shaped surfacejjj
sc/l = (pc/l , nc/l ) with center pjc/l and normal njc/l using
principal component analysis. Next, we repeatedly select andassociate three surfaces from both sets in a random manner,
calculate the transformation, and verify it using a distancemeasure. Algorithm 1 lists these steps in detail.
In line 4 of the algorithm, three surfaces are selected
randomly from the checkerboards. A surface triple (sac , sbc , scc )thereby has the following probability of being selected:p(sac , sbc , scc ) =1T
exp(−nac T nbc − nac T ncc − nbc ncc )Z(9)with Z being a normalizing constant. Intuitively, a surface
triple is selected with higher probability if normals point into
different directions. The probabilites of all possible combinations are computed in advance. Tractability is guaranteedby the limited number of checkerboards present in the scene.

On the contrary, random selection in line 5 is for each
a/b/csr
independent with a uniform probability distribution
over {sjr }.
The transformation θ is computed (line 6) by aligning theselected surfaces. We first estimate the rotation matrix R,
which maximally aligns all normal vectors
X
T
nir R0 nic = VUT(10)
R = argmax
R0i∈{a,b,c}using the singular value decomposition (SVD) of the covariP i iT
ance matrix= UΣVT . Next, we estimate thei nc nrtranslation t by minimizing point-to-plane distancesXt = argmin((R · pic + t0 − pir )T nir )2(11)t0

i∈{a,b,c}

in closed form using least squares. Each transformation isscored (line 7) byXs=−||p̃ic − N1,Pr (p̃ic )||, p̃ic = R · pic + t (12)i∈{a,b,c}which determines the distance from the transformed checkerboard centers to the nearest neighbors in Pr . We return all
solutions for which the score exceeds τscore · max({s}). The
algorithm terminates if either all possible combinations havebeen processed or enough good transformations have beenfound (|G| > τcomb ).
C. Fine Registration
In the previous section, the initial transformation set G has
been calculated from point clouds {Prj }, {Pcj } based on theircentroids and normal vectors solely. Since this is very fast
but less accurate, all transformations θ ∈ G are refined bygradient descent using the following error function:X
E(θ) =||p̃ic − N1,Pr (p̃ic )||2 , p̃ic = R · pic + t (13)pic ∈Pc

This minimizes the sum of point-to-point distances to closest
neighbors, which can be solved with the well-known iterative
closest points algorithm [25]. This results in a set of finely
registered transformations F.

V. EXPERIMENTS
To evaluate our approach, we collected a database of 10
different calibration settings with multiple shots each, as
listed in Table I. Calibration settings differ by baseline, focal
lengths and range sensors employed. For each shot, we varied
the position, orientation and number of calibration targets.
To make our dataset more representative, we additionallycollected 16 random calibration images from the internet,
which are only used to evaluate corner detection. In total, we
used 126 camera images and 55 range measurements in our
experiments. Despite the fact that the proposed frameworkextends to an arbitrary number of sensors, we restrict ourquantitative evaluation to setups involving two cameras anda single range sensor for clarity of illustration, as illustrated
in Fig. 1. As parameters, we empirically picked nnms = 3,
τnms = τcorner = 0.02, τmatch = 0.1, τsegment = 0.9,
τscore = 1.5 and τcomb = 25 and keep them fixed throughout
all experiments. To obtain a modest level of scale invarianceand robustness with respect to blur, we detect corners using
4 × 4, 8 × 8, and 12 × 12 pixel windows in Sec. IIIA and take the maximum of the three scores. Our C++
implementation achieves running times of less than one
minute per calibration scenario, where most time is spentin the camera-to-range point cloud optimization procedures.
A. Corner Detection and Checkerboard Matching
We first evaluate the ability of our method to correctly
locate checkerboard corners in all 126 test images, for which
we annotated all checkerboard corners manually. We compare our approach to several competitive baseline methods:
Harris corners [8] using the Shi-Tomasi criterium [17], areimplementation of the method of Ha et al. [9] and thedetector of Kassir et al. [4]. Note that a fair comparison to themore noise sensitive OpenCV detector [2] and its derivates(Rufli et al. [7]) is not possible due to the fact that they
directly return a single checkerboard per image.
Fig. 5 (left) shows the precision-recall plot obtained byvarying the detection threshold τ for each of the methods. For
Ha et al. [9] we varied the relative number of pixels required
to classify a region as dark or bright while employing a very
conservative Harris threshold. Kassir et al. [4] has been run
using the code and parameters proposed by the authors.
Note that our method significantly outperforms all baselines, especially in terms of recall. The recall of Ha et al.
[9] and Kassir et al. [4] is bound by the (multi-scale) Harris
corner detector, which serves as an input to those methods.

1
0.99

0.8

0.98

0.7

0.97

0.6

0.96

Precision

1
0.9

0.5
0.4
0.3
0.2
0.1
0
0

0.95
0.94
0.93

Our method
Kassir et al. 2010
Ha et al. 2009
Harris et al. 1988
0.2

0.92
0.91

0.4

0.6

0.8

1

0.9
0.9

Our method
Kassir et al. 2010
0.92

0.94

Recall

0.96

0.98

1

Recall5000
4000#Occurences

The transformation set F might contain several transformations that are very similar. Hence, we employ non-maxima
suppression in order to suppress similar transformations in a
local neighborhood based on their energy (13).
Under typical conditions, we retain exactly one solution
after this process. If not, there exist ambiguities that result
from the constellation of the checkerboards, i.e. orthogonality
as in Fig. 6. In this case, an automatic view is rendered foreach transformation and the final selection is left to the user.

Precision

D. Solution Selection

3000
2000
1000

0

0

0.2

0.4
0.6
0.8
Reprojection Error [Pixels]

1

1.2

Fig. 5. Left: Precision-recall curves for different corner detection methods.
Right: Close-up for the range {precision, recall} ∈ [0.9...1.0]. Bottom:
Corner reprojection errors of our method after fitting the model parameters.

Qualitative results of our detector are illustrated in Fig. 8
and at www.cvlibs.net.
B. Camera-to-Camera Calibration
In this section we evaluate the camera-to-camera calibration accuracy by comparing the estimated baselines and focal
lengths to ground truth in all 10 calibration settings. On
average, we obtain a corner reprojection error of 0.18 pixels
indicating high sub-pixel accuracies and a good model fit.
This is also illustrated by the Poisson distribution of the
reprojection errors shown in Fig. 5 (right). Further results
are depicted in Table I: Here, the fifth and eighth column
are the ground truth focal length and baseline, followed by
the mean and standard deviation of the estimated values
respectively. In general, the errors of our fully automatic
calibration system are small. The largest ones occur for
setting 7 and 8. This can be explained as those are theoutdoor setups, which are more difficult since fewer corners
have been matched as a consequence of cast shadows anddifficult lighting conditions. Also note that the provided
ground truth itself is subject to slight measurement errors,
as the values have been measured manually or simply havebeen read from the objective lens.
C. Camera-to-Range Calibration
Assuming a precise camera-calibration, this section evaluates the robustness of camera-to-range calibration. Since
range sensors are independent from lighting conditions andthe proposed approach does not assume any initially given
parameters, the only two factors of influence are the constellation of the checkerboards and the noise within the range
data. The former is covered by the collected dataset (seeTable I) that contains a varying number of checkerboards
in different constellations. For the latter, we added Gaussian
noise N (0, σ 2 I3 ) to the point cloud Pr for varying values of
σ and carry out calibration. The final result R, t is comparedagainst ground truth Rg , tg , which was determined with theTranslation error (m)Rotation error (deg)

150

100

50

2.5
2
1.5
1
0.5
0

0
0

0.005

0.01

0.015

0

0.02

(a) Input data: camera image and range image from the Kinect sensor(a) Kinect: Rotation error

0.015

0.02

(b) Kinect: Translation error
Translation error (m)Rotation error (deg)

100
80
60
40
20

10
8
6
4
2
0

0

Fig. 6.

0.01

12

120

(b) Camera-to-Range calibration result: Three solutions were detected,
automatically rendered images help selecting the appropriate solution.

0.005

Noise (m)Noise (m)

0

0.1

0.2

0.3

0

0.1

0.2

0.3

Noise (m)Noise (m)(c) Velodyne: Rotation error(d) Velodyne: Translation error

Calibration example. Ambiguities are automatically detected.

fine registration (Sec. IV-C) on the original point cloud in asupervised way. Errors are given independently for rotationand translationeter= ||t − tg ||
−1

= ∠(R

Rg )

(14)
(15)

where er is the smallest rotation angle around a rotation
axis that represents the rotation difference. This process isrepeated 20 times for each setting, and the statistics over etand er are gathered for the Velodyne and the Kinect sensor
independently. The results, depicted in Fig. 7, indicate thatthe approach is sufficiently robust to noise, but highly dependent on the calibration setting: Only configurations where thecheckerboards constrain the problem sufficiently well lead tolow errors. This is the case when the checkerboards covermost parts of the image and they are presented at various
distances and orientations. The feasible higher noise level for
the Velodyne sensor in comparison with the Kinect sensor
is thereby due to sparser range data and roughly five times
more distant checkerboards.
VI. CONCLUSIONS
We have proposed a toolbox for automatic camera andrange sensor calibration and shown its effectiveness undervarious conditions. The main limiting assumption of ourapproach is a common field of view of the camera andrange sensors. While this remains a useful scenario forapplications such as generating stereo or scene flow groundtruth, augmenting images with depth values or colorizing apoint cloud, we believe that extending our method to handle
partly overlapping fields of view will further increase therange of applications.
VII. ACKNOWLEDGMENTS
We thank the Karlsruhe School of Optics and Photonics
for their support. We also thank the Research Center for

Fig. 7. Robustness of camera-to-range calibration. Calibration errors
when adding Gaussian noise N (0, σ 2 I3 ) to the range data. On each box,
the central mark is the median, the edges of the box are the 25th and75th percentiles, the whiskers extend to the most extreme data points notconsidered outliers, and outliers are plotted individually.

Information Technology Karlsruhe (FZI) for providing uswith a Kinect range sensor for our experiments.
R EFERENCES[1] J.-Y. Bouguet, Camera Calibration Toolbox for Matlab, 2010.
[Online]. Available: http://www.vision.caltech.edu/bouguetj/calib doc[2] G. Bradski, OpenCV, 2011. [Online]. Available: http://opencv.
willowgarage.com[3] R. Unnikrishnan and M. Hebert, “Fast extrinsic calibration of a laser
rangefinder to a camera,” Robotics Institute, Pittsburgh, PA, Tech. Rep.
CMU-RI-TR-05-09, July 2005.
[4] A. Kassir and T. Peynot, “Reliable automatic camera-laser calibration,”in Australasian Conference on Robotics and Automation, 2010.
[5] J. Heikkila and O. Silven, “A four-step camera calibration procedure
with implicit image correction,” in Computer Vision and Pattern
Recognition, 1997.
[6] A. S. of Photogrammetry, C. Slama, C. Theurer, and S. Henriksen,
Manual of photogrammetry. American Society of Photogrammetry,
1980.
[7] M. Rufli, D. Scaramuzza, and R. Siegwart, “Automatic detection ofcheckerboards on blurred and distorted images,” in Intelligent Robots
and Systems, 2008.
[8] C. Harris and M. Stephens, “A combined corner and edge detector,”
in Fourth Alvey Vision Conference, 1988.
[9] J.-E. Ha, “Automatic detection of chessboard and its applications,”
Optical Engineering, vol. 48, no. 6, June 2009.
[10] J. Pilet, A. Geiger, P. Lagger, V. Lepetit, and P. Fua, “An all-in-one
solution to geometric and photometric calibration,” in International
Symposium on Mixed and Augmented Reality, October 2006.
[11] D. Scaramuzza, A. Harati, and R. Siegwart, “Extrinsic self calibrationof a camera and a 3d laser range finder from natural scenes,” inIntelligent Robots and Systems, 2007.
[12] Q. Zhang and R. Pless, “Extrinsic calibration of a camera and laser
range finder (improves camera calibration),” in Intelligent Robots andSystems, 2004.
[13] H. Zhao, Y. Chen, and R. Shibasaki, “An efficient extrinsic calibrationof a multiple laser scanners and cameras’ sensor system on a mobile
platform,” in Intelligent Vehicles Symposium, 2007.
[14] L. Huang and M. Barth, “A novel multi-planar lidar and computer vision calibration procedure using 2d patterns for automated navigation,”in Intelligent Vehicles Symposium, 2009.

Fig. 8. Structure Recovery and Checkerboard Matching Results. Top row: Random monocular wide-angle and fisheye calibration images from [16]and Google Image Search. Middle row: Binocular camera setup and Kinect range sensor. Last row: Trinocular camera setup with Velodyne HDL 64 laser
scanner. Additional results are available at: www.cvlibs.net.
Setting
0
1
2
3
4
5
6
7
8
9

#Shots
5
5
3
3
4
9
6
5
6
9

Range sensor
kinectkinectkinectkinectkinectkinectvelodyne
velodyne
velodyne
velodyne

Environment
indoorindoorindoorindoorindoorindoorhall
hall / outdoor
outdoorgarage#Checkerboards
3
3
3
3
3
3
7
4–7
4–5
8–12

Focal length f
6.00 mm
9.50 mm
9.50 mm4.00 mm4.00 mm11.50 mm4.00 mm8.00 mm4.00 mm4.00 mm

Mean(fˆ)
6.10 mm
9.56 mm
9.68 mm
4.12 mm
4.11 mm
11.61 mm
4.53 mm8.25 mm
4.32 mm
4.23 mmStd(fˆ)0.01 mm0.13 mm
0.01 mm
0.01 mm
0.01 mm0.19 mm0.11 mm0.29 mm0.11 mm0.02 mmBaseline b
11.50 cm
11.50 cm
20.70 cm
20.70 cm
6.20 cm
6.20 cm
53.60 cm
53.60 cm
53.60 cm
53.60 cm

Mean(b̂)11.45 cm
11.11 cm
20.98 cm
20.76 cm
6.19 cm
6.04 cm
52.27 cm
56.24 cm
54.10 cm
53.17 cm

Std(b̂)0.01 cm
0.14 cm
0.39 cm
0.05 cm
0.10 cm
0.21 cm
0.93 cm
2.60 cm
2.76 cm
0.38 cm

TABLE IE XPERIMENTAL SETTINGS WITH CAMERA - TO - CAMERA CALIBRATION RESULTS[15] O. Naroditsky, A. Patterson, and K. Daniilidis, “Automatic alignmentof a camera with a line scan lidar system,” in International Conferenceon Robotics and Automation, 2011.
[16] D. Scaramuzza and A. Martinelli, “A toolbox for easily calibrating
omnidirectional cameras,” in International Conference on Intelligent
Robots and Systems, 2006.
[17] J. Shi and C. Tomasi, “Good features to track,” in Computer Vision
and Pattern Recognition, 1994.
[18] A. Neubeck and L. J. V. Gool, “Efficient non-maximum suppression,”in International Conference on Pattern Recognition, 2006.
[19] D. Comaniciu and P. Meer, “Mean shift: a robust approach toward
feature space analysis,” Transactions on Pattern Analysis and MachineIntelligence, 2002.
[20] L. Lucchese and S. Mitra, “Using saddle points for subpixel featuredetection in camera calibration targets,” in Asia-Pacific Conference on
Circuits and Systems, 2002.
[21] J. Lavest, M. Viala, and M. Dhome, “Do we really need an accurate calibration pattern to achieve a reliable camera calibration?” inEuropean Conference on Computer Vision, 1998.
[22] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, 2004.
[23] H. Bay, T. Tuytelaars, and L. V. Gool, “Surf: Speeded up robustfeatures,” in European Conference on Computer Vision, 2006.
[24] Z. Zhang, “Flexible camera calibration by viewing a plane fromunknown orientations,” in International Conference on Computer
Vision, 1999.
[25] P. Besl and H. McKay, “A method for registration of 3-d shapes,”
Transactions on Pattern Analysis and Machine Intelligence, 1992.