General, Single-shot, Target-less, and Automatic
LiDAR-Camera Extrinsic Calibration Toolbox
Kenji Koide1 , Shuji Oishi1 , Masashi Yokozuka1 , and Atsuhiko Banno1
Abstract— This paper presents an open source LiDARcamera calibration toolbox that is general to LiDAR and camera projection models, requires only one pairing of LiDAR andcamera data without a calibration target, and is fully automatic.
For automatic initial guess estimation, we employ the SuperGlue image matching pipeline to find 2D-3D correspondences
between LiDAR and camera data and estimate the LiDARcamera transformation via RANSAC. Given the initial guess, we
refine the transformation estimate with direct LiDAR-camera
registration based on the normalized information distance, amutual information-based cross-modal distance metric. For a
handy calibration process, we also present several assistancecapabilities (e.g., dynamic LiDAR data integration and userinterface for making 2D-3D correspondence manually). The
experimental results show that the proposed toolbox enables
calibration of any combination of spinning and non-repetitive
scan LiDARs and pinhole and omnidirectional cameras, andshows better calibration accuracy and robustness than those of
the state-of-the-art edge-alignment-based calibration method.

Spinning LiDAR

Standard FoV camera

Non-repetitive scan LiDAR

Depth camera

Ultra-wide FoV camera

Omnidirectional camera(a) Supported LiDAR and camera models

I. I NTRODUCTION
LiDAR-camera extrinsic calibration is the task of estimating the transformation between the coordinate frames of a
LiDAR and a camera. It is necessary for LiDAR-camera
sensor fusion and is required for many applications, including
autonomous vehicle localization, environmental mapping,and surrounding-object recognition.
Although LiDAR-camera calibration has been actively
studied over the last decade, the robotics community stilllacks a handy and complete LiDAR-camera calibration toolbox. The existing LiDAR-camera calibration frameworks require preparing a calibration target that is sometimes difficult
to create [1], taking many shots of LiDAR-camera datathat results in a large amount of effort [2], or choosing ageometry-rich environment carefully [3]. Furthermore, theyrarely support various LiDAR and camera projection models,such as spinning and non-repetitive scan LiDARs and ultrawide FoV and omnidirectional cameras. We believe that thelack of an easy-to-use LiDAR-camera calibration method has
been a long-standing barrier to the development of LiDARcamera sensor fusion systems.
As a benefit to the robotics community, herein we present
a new complete LiDAR-camera calibration toolbox that has
the following features:• Generalizable: The proposed toolbox is sensor-modelindependent and can handle various LiDAR and cam*This work was supported in part by a project commissioned by the New
Energy and Industrial Technology Development Organization (NEDO).
1 All the authors are with the Department of Information Technology and
Human Factors, the National Institute of Advanced Industrial Science andTechnology, Tsukuba, Ibaraki, Japan, k.koide@aist.go.jp(b) Calibration result (left to right: LiDAR to camera intensity)

Fig. 1: We present a complete LiDAR-camera calibration
framework that can handle various LiDAR and camera
models and calibrate the transformation between them fromonly a single pairing of a LiDAR point cloud and a camera
image. The pixel-level direct alignment algorithm enables
high-quality LiDAR-camera data fusion.
era projection models, including spinning and nonrepetitive scan LiDARs, and pinhole, fisheye, and omnidirectional projection cameras, as shown in Fig. 1 (a).
• Target-less: The proposed calibration algorithm does
not require a calibration target but uses the environment
structure and texture for calibration.
• Single-shot: At a minimum, only one pairing of a
LiDAR point cloud and a camera image is requiredfor calibration. Optionally, multiple LiDAR-camera data
pairs can be used to further improve the calibration
accuracy.
• Automatic: To make the calibration process fully automatic, the system includes an initial guess estimation algorithm with cross-modal 2D-3D correspondence
matching based on SuperGlue [4].
• Accurate and robust: A pixel-level direct LiDARcamera registration algorithm is employed to robustly
and accurately perform LiDAR-camera calibration inenvironments without rich geometrical features, where
existing edge alignment-based algorithms [3] would fail,as long as there exists mutual information between
LiDAR and camera data.
To our knowledge, there is no open implementation that has
all the above features, and we believe the release of thistoolbox will be beneficial to the robotics community.
The main contributions of this paper are as follows:• We present a robust initial guess estimation algorithm
based 2D-3D correspondence estimation. To take advantage of the recent graph neural network-based image
matching [4], we generate a LiDAR intensity image
with a virtual camera and find correspondences between
the LiDAR intensity image and the camera image. An
estimate of the LiDAR-camera transformation is thengiven by RANSAC and reprojection error minimization.
• For robust and accurate calibration, we combined a direct LiDAR-camera fine registration algorithm based on
the normalized information distance (NID), a mutualinformation (MI)-based cross-modal distance metric,
with a view-based hidden points removal algorithm thatfilters out points that are occluded and should not be
visible from the viewpoint of the camera.
• The entire system was carefully designed to be general
to LiDAR and camera projection models so that it canbe applied to various sensor models.
• We released the code of the developed method as open
source to benefit the community 1 .
II. R ELATED W ORK
LiDAR-camera extrinsic calibration methods are categorized into three approaches: 1) target-based, 2) motion-based,
and 3) scene-based.
A. Target-based calibration
As in the well-known camera intrinsic calibration process, the target-based approach is the most natural way for
LiDAR-camera extrinsic calibration. Once we obtain the
3D coordinates of points on a calibration target and theircorresponding 2D coordinates projected in the image, we caneasily estimate the LiDAR-camera transformation by solving
the perspective-n-point problem. The challenge here is that
it is often difficult to design and create a calibration target
that can robustly and accurately be detected by both the
LiDAR and the camera. Several studies used a 3D structured
calibration target that is not as easy to create as the wellknown chessboard pattern [5], [6]. Although several otherworks used a planar pattern that is easy to create, theyrequired manual annotation of LiDAR data [7] and multiple
data acquisitions, resulting in a large amount of effort [8].
B. Motion-based calibration
As in the hand-eye calibration problem, we can estimate
the transformation between two frames on a rigid body
based on their motions [9]. This approach can easily handle
heterogeneous sensors and does not need an overlap between
1 The code is available at https://github.com/koide3/
direct_visual_lidar_calibration

sensors [10]. However, it requires careful time synchronization, which is not always possible when, for example, we use
an affordable web camera. Furthermore, we need to estimate
the per-sensor motion as accurately as possible for better
calibration results.
C. Scene-based calibration
Scene-based methods estimate the LiDAR-camera transformation by considering the consistency between pairs of
LiDAR point clouds and camera images. LiDAR points areprojected in the image space, and then their consistency ismeasured with pixel values based on some metrics.
Similar to the visual odometry estimation problem, there
are two major approaches for LiDAR-camera data consistency evaluation: indirect and direct.
The indirect approach first extracts feature points (e.g.,edge points) from both the camera and LiDAR data, andcomputes the reprojection error between 2D-3D corresponding points [3], [11], [12]. While it exhibits good convergence
due to the discriminative features and robust correspondence
creation, it requires the environment to be geometry-rich to
extract sufficient feature points. Because a majority of texture
information for surfaces in the environment is discarded inthis approach, it is less accurate compared with the direct
approach.
The latter approach compares the pixel and point intensities directly. Because LiDAR and cameras usually exhibit
very different intensity distributions, simply taking the difference between them does not work in practice. To overcome
the difference in modalities, MI-based metrics have beenused for LiDAR-camera data comparison [13], [14]. Because
they do not use the difference between LiDAR and image
intensities but instead consider their co-occurrence, they
can robustly measure the consistency between LiDAR andcamera intensity values. Following [15], [16], we used the
NID metric derived from MI such that it satisfies the metricspace axioms and becomes more robust than MI [15].
III. M ETHODOLOGY
A. Overview
Fig. 2 shows an overview of the proposed LiDAR-camera
calibration toolbox.
To handle various LiDAR models with a unified processing pipeline, we first create a dense point cloud by merging
multiple LiDAR frames. For non-repetitive scan LiDARs(e.g., Livox Avia), we simply accumulate points to densify
the point cloud. For spinning LiDARs (e.g., Velodyne andOuster LiDARs), we use a dynamic LiDAR point integration
technique based on continuous-time ICP [17].
From a paired densified point cloud and camera image,
we then obtain a rough estimate of the LiDAR-camera transformation based on 2D-3D correspondences estimated with
SuperGlue [4]. We also provide an intuitive user interface
for creating manual correspondence for fail-safe. Given the
2D-3D correspondences, we perform RANSAC and reprojection error minimization to obtain an initial estimate of the
LiDAR-camera transformation.

Preprocessing

Spinning
LiDAR

Non-repetitive
scan LiDAR

Dynamic LiDAR
integration

Static LiDAR
integration

Camera(a) Single-scan point cloud

Histogram equalization

Refinement

Initial guess

Dense point cloud & image pairs

Automatic correspondence
estimation (SuperGlue)

Manual correspondence
estimation (GUI)RANSAC + Reprojection error minimization

View-based hidden points removal
NID-based fine LiDAR-camera registration

Fig. 2: Overview of proposed LiDAR-camera calibration
process. Input point clouds are merged to create dense point
clouds using static and dynamic LiDAR point integrators.
Given the densified point cloud and camera image, we find
2D-3D correspondences using the SuperGlue pipeline. We
also provide an easy-to-use manual correspondence estimation tool. Given the 2D-3D correspondences, a roughestimate of the LiDAR-camera transformation is obtained
via RANSAC and reprojection error minimization. Finally,
we perform fine LiDAR-camera registration based on NID
minimization.
Given the initial estimate of the LiDAR-camera transformation, we apply view-based hidden points removal toremove LiDAR points that should not be visible from theviewpoint of the camera. We then refine the LiDAR-cameratransformation estimate via fine LiDAR-camera registration
based on NID minimization.

B. Notation
Our goal is to estimate the transformation between LiDAR
and camera coordinate frames C TL from pairings of LiDAR
point clouds Pi = [L p1 , · · · , L pN ] with point intensities
Li = [l1 , · · · , lN ] and camera images Ii (xj ) = yj , wherexj ∈ R2 are the pixel coordinates and yj is the pixel
intensity. A point in the LiDAR frame L pj is transformed
into the camera coordinate frame as C pj = C TL L pj andprojected into the image space using a projection function π;
xj = π C pj . In this paper, we mainly use the conventional
pinhole camera model with plumb-bob lens distortion and theomnidirectional equirectangular camera model [18] as the
projection function. Note that other major camera models,
including ATAN [19], fisheye [20], and unified omnidirectional camera models [21], have been implemented and aresupported in the developed toolbox.

(b) Densified point cloudFig. 3: Point cloud densification for spinning LiDARs. LiDAR integration enables the creation of a dense point cloudfrom a few seconds of dynamic LiDAR data. The densified
point cloud exhibits rich geometrical and surface texture
information.
C. Preprocessing
Spinning LiDARs (e.g., Ouster OS1-64) exhibit a sparseand repetitive scan pattern, and it is difficult to extract
meaningful geometrical and texture information from only asingle scan (see Fig. 3 (a)). For such a LiDAR, we move the
LiDAR in the up-down direction for a few seconds and accumulate points while compensating for the viewpoint change
and point cloud distortion. To estimate the LiDAR motion,
we use the CT-ICP algorithm, which jointly optimizes the
LiDAR poses at the scan beginning and end by minimizing
the distance between the current LiDAR scan and a model
point cloud with the interpolated LiDAR pose. To efficiently
create the target point cloud from past observations, we use
the linear iVox [22] structure, which simply keeps points
in a linear container for each voxel. Based on the estimated
LiDAR scan beginning and end poses, we correct the motion
distortion on the input point cloud and create a dense point
cloud by accumulating all points in the coordinate frameof the first scan. Fig. 3 (b) shows a densified point cloudafter application of this dynamic LiDAR point integration
process. We can see that the densified point cloud exhibitsrich geometrical and texture information that was difficult tosee in the single-scan point cloud.
For LiDARs with a non-repetitive scan mechanism (e.g.,
Livox Avia), we simply accumulate all scans into one frame,
resulting in a dense point cloud, as shown in Fig. 1 (b).
For the densified point cloud and the camera image, weapply histogram equalization because the NID metric usedin the fine registration step works best with uniform intensity
distributions.
D. Initial Guess Estimation
To obtain a rough estimate of the LiDAR-camera transformation, we first obtain 2D-3D correspondences between the
input images and point clouds and then estimate the LiDARcamera transformation via RANSAC and reprojection error
minimization.

Algorithm 1 Rotation-only RANSAC(a) Ouster OS1-64 (estimated FoV: 178.6°)

K
Input: 2D keypoints K = [xK
1 , · · · , xM ] and corresponding
L K
L K
3D coordinates D = [ p1 , · · · , pM ]1: function E STIMATE ROTATION RANSAC(K, D)
π −1 (xKj )C▷ Bearing vectors of xK2:
dj ← ∥π−1 xKj( j )∥
L

3:

L

pK

▷ Bearing vectors of L pKj

dj ← ∥L pjK ∥jiterationfor i ∈ [1, · · · , N] do
Randomly sample two correspondences j0 and j1
C
RL ← F IND ROTATION LSQ(j0 , j1)K
N ← count of xij s.t. |π C RL L pK
j − xj | < α∗8:if i = 1 or N > N then9:N∗ ← NC ∗10:
RL ← C RL∗
11:
return C RL
12: function E STIMATE ROTATION LSQ(j0 , j1 ) [25]13:
A ← [C dj0 C dj1 ], B ← [L dj0 L dj1 ]14:
U ΣV ∗ = AB
▷ SVD
15:
s ← 1 if |U ||V | >= 0 else −1
16:
return U diag([1, 1, s])V ∗
4:
5:
6:
7:

(b) Livox Avia (estimated FoV: 76.2°)

Fig. 4: LiDAR intensity images rendered with virtual cameras (images are cropped due to space limitations). Either
the pinhole or equirectangular projection model is selected
depending on the FoV of the LiDAR.

Fig. 5: SuperGlue can find correspondences between LiDAR
and camera images in different modalities with a very low
sensitive matching threshold setting. The result, however,contains many false correspondences that need to be filteredout before pose estimation (green: inliers, red: outliers).
To take advantage of the graph neural network-based
image matching pipeline [4], we generate LiDAR intensity
images from dense point clouds with a virtual camera model.
To select the best projection model for rendering the entire
point cloud, we first estimate the FoV of the LiDAR. We
extract the convex hull of the input point cloud using thequickhull algorithm [23] and then find the point pair withthe maximum angle distance in the convex hull using abrute-force search. If the estimated FoV of the LiDAR issmaller than 150°, we create a virtual camera with thepinhole projection model. Otherwise, we create a virtual
camera with the equirectangular projection model. With thevirtual camera, we render the point cloud with intensity
values to obtain LiDAR intensity images, as shown in Fig.
4. Along with the intensity images, we also generate pointindex maps to efficiently look up pixel-wise 3D coordinatesin the subsequent pose estimation step. Note that while
we simply render each point without interpolation and gap
filling, rendering results exhibit good appearance quality
thanks to the densely accumulated point clouds.
To find correspondences between the LiDAR and camera
intensity images, we use the SuperGlue pipeline [4]. It firstdetects keypoints on images using SuperPoint [24] and thenfinds correspondences between the keypoints using a graphneural network. The weights pretrained on the MegaDepth
dataset is used in this work. While SuperGlue can findcorrespondences between images in different modalities, wefound that the matching threshold needs to be set to a verysmall value (e.g., 0.05) to obtain a sufficient number ofcorrespondences. However, with this setting, we observedthat many false correspondences are created, as shown in
Fig. 5.
K
Given 2D keypoint coordinates K = [xK
1 , · · · , xM ]and corresponding 3D coordinates in the LiDAR frame
L K
D = [ L pK
1 , · · · , pM ], we first perform the rotation-only
RANSAC described in Alg. 1 to robustly deal with outlier
correspondences. Given the estimated rotation as an initial
guess, we then obtain the 6 DoF LiDAR-camera transformation C T̃L by minimizing the reprojection errors of all
keypoints using the Levenberg-Marquardt optimizer:
C

T̃L = arg min
CT
L

M
X



2
ρ ∥π C TL L pK
− xK
,
j
j ∥

(1)

j=1

where ρ is the Cauchy robust kernel.
E. NID-based Direct LiDAR-Camera Registration
Some points in the LiDAR point cloud can be occluded
and not visible from the camera due to the viewpoint difference. If we simply project all the LiDAR points, such points
can cause false correspondences and affect the calibration
result, as discussed in [3]. To avoid this problem, we apply
efficient view-based hidden point removal to filter out LiDAR
points that should not be visible from the viewpoint of thecamera. With the current estimate of the LiDAR-cameratransformation, we project the LiDAR points in the image
and keep only the point with the minimum distance for eachpixel (i.e., depth buffer testing). From the projected image

TABLE I: Calibration errors for Ouster OS1-64 and pinhole
camera(a) Ouster OS1-64(b) Livox Avia

Fig. 6: Sensor configurations for LiDAR-camera calibration
experiments.

Data

Init. guess

ProposedTrans. [m]Rot. [°]

Edge-based [3]Trans. [m]Rot. [°]

00
01
02
03
04
05
06
07
08
09
10
11
12
13
14

✓
✗
✗
✓
✗
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓

0.019
0.028
0.014
0.006
0.029
0.023
0.033
0.017
0.048
0.007
0.064
0.011
0.325
0.010
0.015

0.688
0.348
0.180
0.663
0.400
0.450
0.613
0.392
0.386
0.251
0.210
0.146
0.530
0.137
0.217

0.043
0.029
0.035
0.056
0.158
0.170
0.136
0.033
0.520
0.317
0.191
0.384
0.226
1.211
0.034

0.135
0.685
0.490
0.377
1.499
2.334
1.034
0.154
1.458
1.254
1.463
0.768
0.524
7.344
0.421

Avg.

12 / 15

0.043

0.374

0.236

1.329

✓and ✗respectively represent the success and failure of the initial guess.

Fig. 7: Reference LiDAR-camera transformations were measured using high-reflectivity sphere targets; 2D and 3D
positions of the targets were manually annotated and thetransformation was estimated by minimizing their reprojection errors.
that retains only points visible from the camera, we obtaina 3D point cloud and use it for fine registration.
We then perform direct LiDAR-camera registration basedon the NID metric [15]. To compute the NID, we transform
LiDAR points L pj ∈ Pi in the camera frame and project
them into the image space xj = π C TL L pj . From the
LiDAR point intensities lj and corresponding pixel intensities Ii (xj ), we create P(Li ), P(Ii ), and P(Li , Ii ), which are
marginal and joint histograms of LiDAR and pixel intensities,
and calculate their entropies H(Li ), H(Ii ), and H(Li , Ii ) as
follows:
X
H (X) = −p(x) log p(x),(2)
x∈X

where x is each bin in the histogram. The NID between Li
and Ii is then defined as follows:
H (Li , Ii ) − MI (Li ; Ii ),H (Li , Ii )MI (Li ; Ii ) = H (Li ) + H (Ii ) − H (Li , Ii ) ,NID (Li , Ii ) =

(3)
(4)

where MI (Li ; Ii ) is the mutual information between Li and
Ii . By using the Nelder-Mead optimizer, we find the LiDARcamera transformation that minimizes Eq. 3.
Because LiDAR points visible from the camera can change
as the LiDAR-camera transformation is updated, we iterate the view-based hidden points removal and NID-based
LiDAR-camera registration until the displacement of thetransformation update converges.
IV. E XPERIMENT
We evaluated the proposed calibration toolbox with all
four combinations of the spinning and non-repetitive scan
LiDARs (Ouster OS1-64 and Livox Avia) and pinhole and

TABLE II: Calibration errors for Livox Avia and pinhole
cameraData

Init. guess

ProposedTrans. [m]Rot. [°]

Edge-based [3]Trans. [m]Rot. [°]

00
01
02
03
04
05
06
07
08
09
10
11
12
13
14

✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓

0.047
0.162
0.028
0.098
0.124
0.027
0.032
0.026
0.031
0.054
0.071
0.029
0.046
0.080
0.032

0.478
0.885
0.356
0.757
1.430
0.466
0.410
0.273
0.270
0.665
0.887
0.412
0.297
0.645
0.452

1.054
0.152
1.587
0.200
0.065
0.105
0.128
0.216
0.358
0.117
0.214
0.085
0.181
0.170
0.210

6.964
1.336
14.278
3.480
0.933
3.852
1.577
3.609
4.450
1.522
1.590
0.651
2.133
2.152
3.934

Avg.

15 / 15

0.059

0.579

0.323

3.497

✓and ✗respectively represent the success and failure of the initial guess.

omnidirectional cameras (Omron Sentech STC-MBS202POE
and Kodak PixPro 4KVR360) shown in Fig. 6. For eachcombination, we recorded 15 pairs of LiDAR point clouds
and camera images in indoor and outdoor environments, andwe ran the proposed calibration process for each pair (i.e.,single-shot calibration).
As a reference, we estimated the LiDAR-camera transformation using survey-grade high-reflectivity sphere targets, as
shown in Fig.7. We manually annotated the 2D and 3D positions of the targets to find the LiDAR-camera transformations
that minimized the target reprojection errors. From visualinspection, we confirmed that the estimated transformations
closely describe the projection of the cameras. We used theestimated transformations as the “pseudo” ground truth.
Table I summarizes the calibration results for the combination of the Ouster LiDAR and pinhole camera. For the initial
guess estimation, if the translation and rotation errors are
smaller than 0.5 m and 1.0°, respectively, we consider the initial guess estimation as successful. As shown in Table I, theproposed algorithm provided a reasonable initial guess for 12
out of 15 data correlations. Fig. 8 shows the correspondencesof a failure case (Dataset 02). We can see that the estimatedTABLE III: Calibration errors for Ouster OS1-64, Livox
Avia, and omnidirectional camera
Data
00
01
02
03
04
05
06
07
08
09
10
11
12
13
14
Avg.

Ouster OS1-64 & Omnidirectional
Init.
Trans. [m]Rot. [°]

Livox Avia & Omnidirectional
Init.
Trans. [m]Rot. [°]

✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓

0.123
0.111
0.085
0.045
0.061
0.033
0.057
0.038
0.035
0.087
0.087
0.062
0.079
0.095
0.044

1.018
0.355
0.747
0.744
0.664
0.632
0.540
0.410
0.356
0.349
0.428
0.769
0.497
2.939
0.415

✓
✗
✓
✓
✗
✗
✗
✓
✓
✓
✗
✓
✓
✓
✓

0.029
0.059
0.071
0.037
0.067
0.113
0.070
0.056
0.079
0.407
0.161
0.098
0.212
0.033
0.028

0.515
0.902
0.518
0.958
1.265
1.447
0.312
1.466
0.635
1.164
0.753
0.681
0.936
0.409
0.149

15 / 15

0.069

0.724

10 / 15

0.101

0.807

✓and ✗respectively represent the success and failure of the initial guess.

TABLE IV: Multi-data calibration errors
PinholeTrans [m]Rot [°]
Ouster
Livox

0.034
0.010

0.414
0.132

Omnidirectional
Trans [m]Rot [°]
0.082
0.011

0.425
0.400

correspondences contain many false correspondences dueto the flat and repetitive environment structure. Note that
this kind of data pairing, where correct correspondences
are difficult to find, can automatically be filtered out by
RANSAC in multiple-data calibration and does not affectthe estimation result. For data that resulted in a failed initial
guess, we manually annotated 2D-3D correspondences andre-estimated the transformation for evaluation of the fineregistration algorithm.
The proposed fine registration algorithm worked well for
all the data and achieved 0.043 m and 0.374° calibration
errors on average. As a baseline, we also applied the stateof-the-art edge alignment-based calibration method [3]. For
this method, we used the reference LiDAR-camera transformation as the initial transformation, and thus it was evaluatedwith an almost ideal initial guess. However, as can be seenin Table I, it showed large calibration errors for several data
TABLE V: Processing time
LiDAR

CameraPreprocessing

Init. guess

Calibration

Total

OusterOusterLivox
Livox

PinholeOmnidir.
PinholeOmnidir.

56.6 s56.3 s9.3 s
9.2 s

10.3 s11.2 s10.6 s9.7 s28.8 s181.5 s54.5 s84.3 s95.7 s249.0 s74.4 s103.2 scorrelations, resulting in worse average calibration errors(0.236 m and 1.329°). This was because both image andpoint cloud edge extraction in [3] were very sensitive toenvironment changes and could not properly extract edge
points in several environments.
As shown in Table II, for the combination of the Livox
LiDAR and pinhole camera, the initial guess estimation successfully provided good initial LiDAR-camera transformations for all data pairings, and the fine registration algorithm
achieved average calibration errors of 0.069 m and 0.724°,
which were better than those for the edge-based calibration[3] (0.323 m and 3.497°).
Table III summarizes the calibration results for the omnidirectional camera. While it showed a good initial guess
success rate (15 / 15) and low calibration errors for the
Ouster LiDAR (0.069 m and 0.724°), both the initial guessestimation and the fine LiDAR-camera registration tended
be degraded for the combination of the Livox LiDAR andomnidirectional camera (10 / 15 success rate, 0.101 m
and 0.807°). This is because only a small portion of theomnidirectional camera images was used for calibration due
to the very different FoVs of the LiDAR and camera, and theresolution of the images was not sufficient to represent fineenvironmental details with this limited FoV. We think thatthe calibration accuracy can be improved by using higherresolution or multiple images.
Finally, we performed multi-data calibration with all data
pairings and evaluated the calibration errors for all the
LiDAR-camera combinations. Table IV summarizes the calibration errors. We can see that good calibration results wereobtained even for the combination of the Livox and omnidirectional camera (0.011 m and 0.400°). For the combinationof the Ouster LiDAR and pinhole camera, we observed
the best calibration accuracy (0.010 m and 0.132°). This isbecause in this combination, the LiDAR and camera have
similar FoVs and most of input data were used for calibration
while some points and image regions are not used in othercombinations due to the difference of FoV.
Table V shows the processing times for each calibration
step. Depending on the combination of the LiDAR and camera models, it took from 74 to 249 s to calibrate the LiDARcamera transformation from 15 data pairings. Although the
combination of the Ouster LiDAR and omnidirectional camera took a longer time because both the sensors have 360°
FoVs and need projection of most of the points, we consider
it is in a reasonable level for offline calibration.
V. C ONCLUSION

Fig. 8: Failure case for initial guess estimation (Ouster 02).
(green: inliers, red: outliers)

We developed a general LiDAR-camera calibration toolbox. For a fully automatic calibration process, we used image
matching-based initial guess estimation. The initial estimate
was then refined by a NID-based direct LiDAR-camera registration algorithm. The experimental results showed that the
toolbox can accurately calibrate the transformation between
spinning and non-repetitive scan LiDARs and pinhole andomnidirectional cameras.

R EFERENCES[1] Y. Xie, L. Deng, T. Sun, Y. Fu, J. Li, X. Cui, H. Yin, S. Deng,
J. Xiao, and B. Chen, “A4lidartag: Depth-based fiducial marker for
extrinsic calibration of solid-state lidar and camera,” IEEE Robotics
and Automation Letters, vol. 7, no. 3, pp. 6487–6494, July 2022.
[2] D. Tsai, S. Worrall, M. Shan, A. Lohr, and E. Nebot, “Optimising
the selection of samples for robust lidar camera calibration,” in IEEE
International Intelligent Transportation Systems Conference. IEEE,
Sept. 2021, pp. 2631–2638.
[3] C. Yuan, X. Liu, X. Hong, and F. Zhang, “Pixel-level extrinsic
self calibration of high resolution LiDAR and camera in targetlessenvironments,” IEEE Robotics and Automation Letters, vol. 6, no. 4,
pp. 7517–7524, Oct. 2021.
[4] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “SuperGlue: Learning feature matching with graph neural networks,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition.
IEEE, June 2020, pp. 4938–4947.
[5] J. Beltran, C. Guindel, A. de Escalera la, and F. Garcia, “Automatic
extrinsic calibration method for LiDAR and camera sensor setups,”
IEEE Transactions on Intelligent Transportation Systems, pp. 1–13,
Mar. 2022.
[6] C. Fang, S. Ding, Z. Dong, H. Li, S. Zhu, and P. Tan, “Singleshot is enough: Panoramic infrastructure based calibration of multiple
cameras and 3d LiDARs,” in IEEE/RSJ International Conference onIntelligent Robots and Systems. IEEE, Sept. 2021, pp. 8890–8897.
[7] Q. Zhang and R. Pless, “Extrinsic calibration of a camera and laser
range finder (improves camera calibration),” in IEEE/RSJ International
Conference on Intelligent Robots and Systems. IEEE, Sept. 2004, pp.
2301–2306.
[8] L. Zhou, Z. Li, and M. Kaess, “Automatic extrinsic calibration ofa camera and a 3d LiDAR using line and plane correspondences,” inIEEE/RSJ International Conference on Intelligent Robots and Systems.
IEEE, Oct. 2018, pp. 5562–5569.
[9] R. Tsai and R. Lenz, “A new technique for fully autonomous andefficient 3d robotics hand/eye calibration,” IEEE Transactions on
Robotics and Automation, vol. 5, no. 3, pp. 345–358, June 1989.
[10] R. Ishikawa, T. Oishi, and K. Ikeuchi, “LiDAR and camera calibration
using motions estimated by sensor fusion odometry,” in IEEE/RSJ
International Conference on Intelligent Robots and Systems. IEEE,
Oct. 2018, pp. 7342–7349.
[11] X. Zhang, S. Zhu, S. Guo, J. Li, and H. Liu, “Line-based automatic
extrinsic calibration of LiDAR and camera,” in IEEE International
Conference on Robotics and Automation. IEEE, May 2021, pp. 9347–
9353.
[12] X. Liu, C. Yuan, and F. Zhang, “Targetless extrinsic calibration ofmultiple small FoV LiDARs and cameras using adaptive voxelization,”

IEEE Transactions on Instrumentation and Measurement, vol. 71, pp.
1–12, May 2022.
[13] G. Pandey, J. McBride, S. Savarese, and R. Eustice, “Automatictargetless extrinsic calibration of a 3d lidar and camera by maximizing mutual information,” AAAI Conference on Artificial Intelligence,
vol. 26, no. 1, pp. 2053–2059, Sept. 2021.
[14] G. Pandey, J. R. McBride, S. Savarese, and R. M. Eustice, “Automatic
extrinsic calibration of vision and lidar by maximizing mutual information,” Journal of Field Robotics, vol. 32, no. 5, pp. 696–722, Sept.
2014.
[15] A. Stewart, “Localisation using the appearance of prior structure,”
Ph.D. dissertation, Oxford University, UK, 2016.
[16] J. Jeong, Y. Cho, and A. Kim, “The road is enough! extrinsic
calibration of non-overlapping stereo camera and LiDAR using roadinformation,” IEEE Robotics and Automation Letters, vol. 4, no. 3,
pp. 2831–2838, July 2019.
[17] P. Dellenbach, J.-E. Deschaud, B. Jacquet, and F. Goulette, “CT-ICP:
Real-time elastic LiDAR odometry with loop closure,” in International
Conference on Robotics and Automation. IEEE, May 2022, pp. 5580–
5586.
[18] A. Torii, M. Havlena, and T. Pajdla, “From google street view to 3dcity models,” in IEEE International Conference on Computer Vision
Workshops. IEEE, Sept. 2009, pp. 2188–2195.
[19] F. Devernay and O. Faugeras, “Straight lines have to be straight,”
Machine Vision and Applications, vol. 13, no. 1, pp. 14–24, Aug.
2001.
[20] J. Kannala and S. Brandt, “A generic camera model and calibration
method for conventional, wide-angle, and fish-eye lenses,” IEEE
Transactions on Pattern Analysis and Machine Intelligence, vol. 28,
no. 8, pp. 1335–1340, Aug. 2006.
[21] D. Scaramuzza, A. Martinelli, and R. Siegwart, “A toolbox foreasily calibrating omnidirectional cameras,” in IEEE/RSJ International
Conference on Intelligent Robots and Systems. IEEE, Oct. 2006, pp.
5695–5701.
[22] C. Bai, T. Xiao, Y. Chen, H. Wang, F. Zhang, and X. Gao, “FasterLIO: Lightweight tightly coupled lidar-inertial odometry using parallelsparse incremental voxels,” IEEE Robotics and Automation Letters,
vol. 7, no. 2, pp. 4861–4868, Apr. 2022.
[23] C. B. Barber, D. P. Dobkin, and H. Huhdanpaa, “The quickhull
algorithm for convex hulls,” ACM Transactions on Mathematical
Software, vol. 22, no. 4, pp. 469–483, Dec. 1996.
[24] D. DeTone, T. Malisiewicz, and A. Rabinovich, “SuperPoint: Selfsupervised interest point detection and description,” in IEEE/CVF
Conference on Computer Vision and Pattern Recognition Workshops.
IEEE, June 2018, pp. 224–236.
[25] S. Umeyama, “Least-squares estimation of transformation parameters
between two point patterns,” IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 13, no. 4, pp. 376–380, Apr. 1991.