Automatic Camera and Range Sensor
Calibration using a single Shot
Andreas Geiger Frank Moosmann Ömer Car Bernhard Schuster

I NSTITUTE OF M EASUREMENT AND C ONTROL S YSTEMSK ARLSRUHE I NSTITUTE OF T ECHNOLOGYMotivation

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

2/19

Overview

Cameras / Microsoft Kinect

Cameras / Velodyne HDL-64

Setup: Video cameras (+ range sensor)

Goals
1

Calibrate cameras intrinsically and extrinsically

2

Register range sensor to cameras

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

3/19

Overview

Cameras / Microsoft Kinect

Cameras / Velodyne HDL-64

Setup: Video cameras (+ range sensor)

Goals
1

Calibrate cameras intrinsically and extrinsically

2

Register range sensor to cameras

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

3/19

Overview

Cameras / Microsoft Kinect

Cameras / Velodyne HDL-64

Setup: Video cameras (+ range sensor)

Goals
1

Calibrate cameras intrinsically and extrinsically

2

Register range sensor to cameras

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

3/19

Proposed Method
Contributions
Automatic camera and range sensor calibration
Our method requires only one image / scan per sensor
Processing time < 5 minutes

Assumptions
Planar checkerboards (presented at different poses)
Overlapping field of view (e.g., stereo)

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

4/19

Proposed Method
Contributions
Automatic camera and range sensor calibration
Our method requires only one image / scan per sensor
Processing time < 5 minutes

Assumptions
Planar checkerboards (presented at different poses)
Overlapping field of view (e.g., stereo)

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

4/19

Camera Calibration

Goals
1

Calibrate cameras intrinsically and extrinsically

2

Register range sensor to cameras

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

5/19

Camera: Related Work
Bouget’s Camera Calibration Toolbox for Matlab / OpenCV

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

6/19

Camera: Related Work
Bouget’s Camera Calibration Toolbox for Matlab / OpenCV

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

6/19

Related Work: Camera Calibration
Bouget’s Camera Calibration Toolbox for Matlab / OpenCV

Requires ≥ 10 synchronized images
No automatic corner detection and matching
Time consuming
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

7/19

Related Work: Camera Calibration
Bouget’s Camera Calibration Toolbox for Matlab / OpenCV

Requires ≥ 10 synchronized images
No automatic corner detection and matching
Time consuming
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

7/19

Related Work: Camera Calibration
Bouget’s Camera Calibration Toolbox for Matlab / OpenCV

Requires ≥ 10 synchronized images
No automatic corner detection and matching
Time consuming
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

7/19

Camera: Corner Detection
Corner detection and subpixel refinement
Compute cornerness score for each pixel
Non-maximum suppression, sub-pixel refinement

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

8/19

Camera: Corner Detection
Corner detection and subpixel refinement
Compute cornerness score for each pixel
Non-maximum suppression, sub-pixel refinement

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

8/19

Camera: Corner Detection
Corner detection and subpixel refinement
Compute cornerness score for each pixel
Non-maximum suppression, sub-pixel refinement

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

8/19

Camera: Corner Detection
Corner detection and subpixel refinement
Compute cornerness score for each pixel
Non-maximum suppression, sub-pixel refinement

87#occurences

6
5
4
3
2
1
0
0

0.5

1

1.5
2
orientation (rad)2.5

3

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

8/19

Camera: Corner Detection
Corner detection and subpixel refinement
Compute cornerness score for each pixel
Non-maximum suppression, sub-pixel refinement

Intensity
Score

Gradient
Score

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

8/19

Camera: Corner Detection
Corner detection and subpixel refinement
Compute cornerness score for each pixel
Non-maximum suppression, sub-pixel refinement

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

8/19

Camera: Finding Checkerboards

Parameters
Corner locations: X = {c1 , .., cN }, ci ∈ R2
Corner labels: Y = {y1 , .., yN }, yi ∈ {O} ∪ N3
Energy E(X , Y) = Ecorners (Y) + Estruct (X , Y)
Ecorners = − number of explained corners
Estruct = − corner collinearity
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

9/19

Camera: Finding Checkerboards

Parameters
Corner locations: X = {c1 , .., cN }, ci ∈ R2
Corner labels: Y = {y1 , .., yN }, yi ∈ {O} ∪ N3
Energy E(X , Y) = Ecorners (Y) + Estruct (X , Y)
Ecorners = − number of explained corners
Estruct = − corner collinearity
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

9/19

Camera: Finding Checkerboards

Parameters
Corner locations: X = {c1 , .., cN }, ci ∈ R2
Corner labels: Y = {y1 , .., yN }, yi ∈ {O} ∪ N3
Energy E(X , Y) = Ecorners (Y) + Estruct (X , Y)
Ecorners = − number of explained corners
Estruct = − corner collinearity
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

9/19

Camera: Finding Checkerboards

Parameters
Corner locations: X = {c1 , .., cN }, ci ∈ R2
Corner labels: Y = {y1 , .., yN }, yi ∈ {O} ∪ N3
Energy E(X , Y) = Ecorners (Y) + Estruct (X , Y)
Ecorners = − number of explained corners
Estruct = − corner collinearity
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

Collinearity
9/19

Camera: Finding Checkerboards
Exponential complexity O(|X ||L| ) ⇒ Search space pruning
For each corner as seed do:
Incrementally add neighboring corners with lowest energy

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

10/19

Camera: Finding Checkerboards
Exponential complexity O(|X ||L| ) ⇒ Search space pruning
For each corner as seed do:
Incrementally add neighboring corners with lowest energy

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

10/19

Camera: Finding Checkerboards
Exponential complexity O(|X ||L| ) ⇒ Search space pruning
For each corner as seed do:
Incrementally add neighboring corners with lowest energy

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

10/19

Camera: Finding Checkerboards
Exponential complexity O(|X ||L| ) ⇒ Search space pruning
For each corner as seed do:
Incrementally add neighboring corners with lowest energy

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

10/19

Camera: Finding Checkerboards
Exponential complexity O(|X ||L| ) ⇒ Search space pruning
For each corner as seed do:
Incrementally add neighboring corners with lowest energy

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

10/19

Camera: Finding Checkerboards
Exponential complexity O(|X ||L| ) ⇒ Search space pruning
For each corner as seed do:
Incrementally add neighboring corners with lowest energy

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

10/19

Camera: Finding Checkerboards
Exponential complexity O(|X ||L| ) ⇒ Search space pruning
For each corner as seed do:
Incrementally add neighboring corners with lowest energy

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

10/19

Camera: Finding Checkerboards
Exponential complexity O(|X ||L| ) ⇒ Search space pruning
For each corner as seed do:
Incrementally add neighboring corners with lowest energy

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

10/19

Camera: Finding Checkerboards
Exponential complexity O(|X ||L| ) ⇒ Search space pruning
For each corner as seed do:
Incrementally add neighboring corners with lowest energy
Keep lowest energy solutions in case of overlaps

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

10/19

Camera: Matching and Optimization
Match checkerboards across images
Sample 2 checkerboards per image
Compute similarity transformationfrom center of checkerboards
Maximize number of inliers
Parameter Optimization
Parameters: {f, c, α, k1 , .., k5 }, {r, t}
Non-linear least squares(Gauss-Newton)

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

11/19

Camera: Matching and Optimization
Match checkerboards across images
Sample 2 checkerboards per image
Compute similarity transformationfrom center of checkerboards
Maximize number of inliers
Parameter Optimization
Parameters: {f, c, α, k1 , .., k5 }, {r, t}
Non-linear least squares(Gauss-Newton)

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

11/19

Results: Checkerboard Matching

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

12/19

Results: Corner Detection
1
0.9
0.8

Precision

0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0

Our method
Kassir et al. 2010
Ha et al. 2009
Harris et al. 1988
0.2

0.4

0.6

0.8

1

RecallPrecision-Recall computed from
150 images taken under various conditions
80.000 checkerboard corners
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

13/19

3D Range Sensor Registration

Goals
1

Calibrate cameras intrinsically and extrinsically

2

Register range sensor to cameras

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

14/19

3D Range Sensor: Related Work
[Unnikrishnan and Hebert 2005]
Interactive GUI for laser-to-camera registration
User marks calibration object manually[Scaramuzza, Harati, Siegwart 2007]
Registration of omnidirectional camera and laserscanner
Manual correspondence selection required

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

15/19

3D Range Sensor: Related Work
[Unnikrishnan and Hebert 2005]
Interactive GUI for laser-to-camera registration
User marks calibration object manually[Scaramuzza, Harati, Siegwart 2007]
Registration of omnidirectional camera and laserscanner
Manual correspondence selection required

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

15/19

3D Range Sensor: Segmentation
Segments point cloud into planar pieces
Compute normal vector nir ∈ R3 for each point pir ∈ R3 byprincipal component analysis on k-nearest neighbors
Grow regions from random seeds
Stop growing when normal gets too dissimilar from seed
Repeat until segmentation covers all points

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

16/19

3D Range Sensor: Segmentation
Segments point cloud into planar pieces
Compute normal vector nir ∈ R3 for each point pir ∈ R3 byprincipal component analysis on k-nearest neighbors
Grow regions from random seeds
Stop growing when normal gets too dissimilar from seed
Repeat until segmentation covers all points

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

16/19

3D Range Sensor: Segmentation
Segments point cloud into planar pieces
Compute normal vector nir ∈ R3 for each point pir ∈ R3 byprincipal component analysis on k-nearest neighbors
Grow regions from random seeds
Stop growing when normal gets too dissimilar from seed
Repeat until segmentation covers all points

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

16/19

3D Range Sensor: Segmentation
Segments point cloud into planar pieces
Compute normal vector nir ∈ R3 for each point pir ∈ R3 byprincipal component analysis on k-nearest neighbors
Grow regions from random seeds
Stop growing when normal gets too dissimilar from seed
Repeat until segmentation covers all points

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

16/19

3D Range Sensor: Segmentation
Segments point cloud into planar pieces
Compute normal vector nir ∈ R3 for each point pir ∈ R3 byprincipal component analysis on k-nearest neighbors
Grow regions from random seeds
Stop growing when normal gets too dissimilar from seed
Repeat until segmentation covers all points

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

16/19

3D Range Sensor: Segmentation
Segments point cloud into planar pieces
Compute normal vector nir ∈ R3 for each point pir ∈ R3 byprincipal component analysis on k-nearest neighbors
Grow regions from random seeds
Stop growing when normal gets too dissimilar from seed
Repeat until segmentation covers all points

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

16/19

3D Range Sensor: Segmentation
Segments point cloud into planar pieces
Compute normal vector nir ∈ R3 for each point pir ∈ R3 byprincipal component analysis on k-nearest neighbors
Grow regions from random seeds
Stop growing when normal gets too dissimilar from seed
Repeat until segmentation covers all points

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

16/19

3D Range Sensor: Segmentation
Segments point cloud into planar pieces
Compute normal vector nir ∈ R3 for each point pir ∈ R3 byprincipal component analysis on k-nearest neighbors
Grow regions from random seeds
Stop growing when normal gets too dissimilar from seed
Repeat until segmentation covers all points

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

16/19

3D Range Sensor: Segmentation
Segments point cloud into planar pieces
Compute normal vector nir ∈ R3 for each point pir ∈ R3 byprincipal component analysis on k-nearest neighbors
Grow regions from random seeds
Stop growing when normal gets too dissimilar from seed
Repeat until segmentation covers all points

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

16/19

3D Range Sensor: Segmentation
Segments point cloud into planar pieces
Compute normal vector nir ∈ R3 for each point pir ∈ R3 byprincipal component analysis on k-nearest neighbors
Grow regions from random seeds
Stop growing when normal gets too dissimilar from seed
Repeat until segmentation covers all points

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

16/19

3D Range Sensor: Segmentation
Segments point cloud into planar pieces
Compute normal vector nir ∈ R3 for each point pir ∈ R3 byprincipal component analysis on k-nearest neighbors
Grow regions from random seeds
Stop growing when normal gets too dissimilar from seed
Repeat until segmentation covers all points

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

16/19

3D Range Sensor: Segmentation
Segments point cloud into planar pieces
Compute normal vector nir ∈ R3 for each point pir ∈ R3 byprincipal component analysis on k-nearest neighbors
Grow regions from random seeds
Stop growing when normal gets too dissimilar from seed
Repeat until segmentation covers all points

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

16/19

3D Range Sensor: Registration
Sample a set of plausible hypotheses
Draw 3 checkerboards
Draw 3 segments
Estimate optimal rotation R
Estimate optimal translation t(minimize center to plane)
Score hypothesis using
euclidean distance
Repeat until with probability p thecorrect solution has been foundRefine best hypotheses using robust Point-to-Point ICPP
Minimize: i wi kpi − N (pi )k2 with wi ∈ {0, 1}
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

17/19

3D Range Sensor: Registration
Sample a set of plausible hypotheses
Draw 3 checkerboards
Draw 3 segments
Estimate optimal rotation R
Estimate optimal translation t(minimize center to plane)
Score hypothesis using
euclidean distance
Repeat until with probability p thecorrect solution has been foundRefine best hypotheses using robust Point-to-Point ICPP
Minimize: i wi kpi − N (pi )k2 with wi ∈ {0, 1}
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

17/19

3D Range Sensor: Registration
Sample a set of plausible hypotheses
Draw 3 checkerboards
Draw 3 segments
Estimate optimal rotation R
Estimate optimal translation t(minimize center to plane)
Score hypothesis using
euclidean distance
Repeat until with probability p thecorrect solution has been foundRefine best hypotheses using robust Point-to-Point ICPP
Minimize: i wi kpi − N (pi )k2 with wi ∈ {0, 1}
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

17/19

3D Range Sensor: Registration
Sample a set of plausible hypotheses
Draw 3 checkerboards
Draw 3 segments
Estimate optimal rotation R
Estimate optimal translation t(minimize center to plane)
Score hypothesis using
euclidean distance
Repeat until with probability p thecorrect solution has been foundRefine best hypotheses using robust Point-to-Point ICPP
Minimize: i wi kpi − N (pi )k2 with wi ∈ {0, 1}
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

17/19

3D Range Sensor: Registration
Sample a set of plausible hypotheses
Draw 3 checkerboards
Draw 3 segments
Estimate optimal rotation R
Estimate optimal translation t(minimize center to plane)
Score hypothesis using
euclidean distance
Repeat until with probability p thecorrect solution has been foundRefine best hypotheses using robust Point-to-Point ICPP
Minimize: i wi kpi − N (pi )k2 with wi ∈ {0, 1}
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

17/19

3D Range Sensor: Registration
Sample a set of plausible hypotheses
Draw 3 checkerboards
Draw 3 segments
Estimate optimal rotation R
Estimate optimal translation t(minimize center to plane)
Score hypothesis using
euclidean distance
Repeat until with probability p thecorrect solution has been foundRefine best hypotheses using robust Point-to-Point ICPP
Minimize: i wi kpi − N (pi )k2 with wi ∈ {0, 1}
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

17/19

3D Range Sensor: Registration
Sample a set of plausible hypotheses
Draw 3 checkerboards
Draw 3 segments
Estimate optimal rotation R
Estimate optimal translation t(minimize center to plane)
Score hypothesis using
euclidean distance
Repeat until with probability p thecorrect solution has been foundRefine best hypotheses using robust Point-to-Point ICPP
Minimize: i wi kpi − N (pi )k2 with wi ∈ {0, 1}
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

17/19

3D Range Sensor: Registration
Sample a set of plausible hypotheses
Draw 3 checkerboards
Draw 3 segments
Estimate optimal rotation R
Estimate optimal translation t(minimize center to plane)
Score hypothesis using
euclidean distance
Repeat until with probability p thecorrect solution has been foundRefine best hypotheses using robust Point-to-Point ICPP
Minimize: i wi kpi − N (pi )k2 with wi ∈ {0, 1}
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

17/19

3D Range Sensor: Registration
Sample a set of plausible hypotheses
Draw 3 checkerboards
Draw 3 segments
Estimate optimal rotation R
Estimate optimal translation t(minimize center to plane)
Score hypothesis using
euclidean distance
Repeat until with probability p thecorrect solution has been foundRefine best hypotheses using robust Point-to-Point ICPP
Minimize: i wi kpi − N (pi )k2 with wi ∈ {0, 1}
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

17/19

3D Range Sensor: Registration
Sample a set of plausible hypotheses
Draw 3 checkerboards
Draw 3 segments
Estimate optimal rotation R
Estimate optimal translation t(minimize center to plane)
Score hypothesis using
euclidean distance
Repeat until with probability p thecorrect solution has been foundRefine best hypotheses using robust Point-to-Point ICPP
Minimize: i wi kpi − N (pi )k2 with wi ∈ {0, 1}
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

17/19

3D Range Sensor: Registration
Sample a set of plausible hypotheses
Draw 3 checkerboards
Draw 3 segments
Estimate optimal rotation R
Estimate optimal translation t(minimize center to plane)
Score hypothesis using
euclidean distance
Repeat until with probability p thecorrect solution has been foundRefine best hypotheses using robust Point-to-Point ICPP
Minimize: i wi kpi − N (pi )k2 with wi ∈ {0, 1}
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

17/19

3D Range Sensor: Registration
Sample a set of plausible hypotheses
Draw 3 checkerboards
Draw 3 segments
Estimate optimal rotation R
Estimate optimal translation t(minimize center to plane)
Score hypothesis using
euclidean distance
Repeat until with probability p thecorrect solution has been foundRefine best hypotheses using robust Point-to-Point ICPP
Minimize: i wi kpi − N (pi )k2 with wi ∈ {0, 1}
A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

17/19

3D Range Sensor: Results

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

18/19

3D Range Sensor: Results

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

18/19

Thank you!

www.cvlibs.net

A. Geiger – Camera and Range Sensor Calibration – www.cvlibs.net

19/19