Tame a Wild Camera: In-the-Wild Monocular Camera Calibration
Authors: Shengjie Zhu
Abhinav Kumar
Masa Hu
Xiaoming Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Monocular Camera Calibration; Camera Pose Estimation; Image Editing
TL;DR: We calibrate 4 DoF intrinsic with monocular image. We demonstrate robust in-the-wild performance. We include intriguing applications.

Abstract: 3D sensing for monocular in-the-wild images, e.g., depth estimation and 3D object detection, has become increasingly important.
However, the unknown intrinsic parameter hinders their development and deployment.
Previous methods for the monocular camera calibration rely on specific 3D objects or strong geometry prior, such as using a checkerboard or imposing a Manhattan World assumption.
This work instead calibrates intrinsic via exploiting the monocular 3D prior.
Given an undistorted image as input, our method calibrates the complete 4 Degree-of-Freedom (DoF) intrinsic parameters.
First, we show intrinsic is determined by the two well-studied monocular priors: monocular depthmap and surface normal map.
However, this solution necessitates a low-bias and low-variance depth estimation.
Alternatively, we introduce the incidence field, defined as the incidence rays between points in 3D space and pixels in the 2D imaging plane.
We show that: 1) The incidence field is a pixel-wise parametrization of the intrinsic invariant to image cropping and resizing.
2) The incidence field is a learnable monocular 3D prior, determined pixel-wisely by up-to-sacle monocular depthmap and surface normal.
With the estimated incidence field, a robust RANSAC algorithm recovers intrinsic.
We show the effectiveness of our method through superior performance on synthetic and zero-shot testing datasets.
Beyond calibration, we demonstrate downstream applications in image manipulation detection \& restoration, uncalibrated two-view pose estimation, and 3D sensing.

Tame a Wild Camera: In-the-Wild Monocular Camera Calibration
Shengjie Zhu, Abhinav Kumar, Masa Hu, and Xiaoming Liu Department of Computer Science and Engineering, Michigan State University, East Lansing, MI, 48824 (zhusheng, kumarab6, huynshen] @ msu.edu, liuxm @ mss.euu
Abstract
3D sensing for monocular in-the-wild images, e.g., depth estimation and 3D object detection, has become increasingly important. However, the unknown intrinsic parameter hinders their development and deployment. Previous methods for the monocular camera calibration rely on specific 3D objects strong geometry prior, such as using a checkerboard or imposing a Manhattan World assumption. This work instead calibrates intrinsic via exploiting the monocular 3D prior. Given an undistorted image as input, our method calibrates the complete 4 Degree-of- Freedom (DoF) intrinsic parameters. First, show intrinsic is determined by the two well-studied monocular priors: monocular depthmap and surface normal map. However, this solution necessitates low-bias and low-variance depth estimation. Alternatively, we introduce the incidence field, defined as the incidence rays be- tween points in 3D space and pixels in the 2D imaging plane. We show that: 1) The incidence field is a pixel-wise parametrization of the intrinsic invariant to image cropping and resizing. 2) The incidence field is a learnable monocular 3D prior, determined pixel-wisely by up-to-sacle monocular depthmap and surface normal. With the estimated incidence field, robust RANSAC algorithm recovers intrinsic. We show the effectiveness of our method through superior performance on synthetic and zero-shot testing datasets. Beyond calibration, we demonstrate downstream applications in image manipulation detection & restoration, uncalibrated two-view pose estimation, and 3D sensing. Codes/models are held here.
Introduction
Camera calibration is typically the first step in numerous vision and robotics applications [27,46] that involve 3D sensing. Classic methods enable accurate calibration by imaging specific 3D) structure such checkerboard [49]. With the rapid growth of monocular 3D vision, there is an increasing focus on 3D sensing from in-the-wild images, such as monocular depth estimation, 3D object detection [35, 36], and 3D reconstruction [42, 43]. While 3D sensing techniques over in-the-wild images are developed, camera calibration for such in-the-wild images continues to pose significant challenges.
Classic methods for monocular calibration use strong geometry prior, such as using a checkerboard. However, such 3D structures not always available in in-the-wild images. As solution, alternative methods relax the assumptions. For example, [29] and [23] calibrate using common objects such as human faces and objects' 3D bounding boxes. Another significant line of research [7, 19, 37,3,, 62, 66] is based on the Manhattan World assumption [15], which posits that all planes within a scene either parallel
perpendicular each other. This assumption further relaxed [08,30, 64] to estimate the lines that are either parallel or perpendicular to the direction of gravity. The intrinsic parameters are recovered by determining the intersected vanishing points of detected lines, assuming a central focal point and an identical focal length.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(8)
de
(0)
GT Intrinsic Noisy Intrinsic
Mono- Mono-Callbaation
Incidence Ray
Incidence Vector
Intrinsic
Noisy Intrinsic
Canera
Intrinsic
Figure 1: In (a), our work focuses on monocular camera calibration for in-the-wild images. recover the
intrinsic from monocular 3D- 3D-prior In (c) (e), an estimated depthmap is converted to surface normal using a groundtruth and noisy intrinsic individually. Noisy intrinsic distorts the point cloud, consequently leading to inaccurate surface normal. In (e). the normal presents different color (c). Motivated by the observation, develop solver that utilizes the consistency between the two to recover the intrinsic. However, this solution exhibits numerical instability. We then propose to learn the incidence field as an alternative 3D monocular prior. The incidence field is the collection of the pixel-wise incidence ray, which originates from a 3D point, targets at a 2D pixel, and crosses the camera origin, as shown in (b). Similar to depthmap and normal, noisy intrinsic leads to noisy incidence field, as in (e). By same motivation, develop neural network to learn mn-the-wild incidence field and develop RANSAC algorithm recover intrinsic from the estimated incidence field.
While the assumptions relaxed, they may still not hold true for in-the-wild images. This creates a contradiction: although we develop robust models to estimate in-the-wild monocular depthmap, generating its 3D point cloud remains infeasible due the missing intrinsic. A similar challenge arises monocular 3D object detection, as we face limitations in projecting the detected 3D bounding boxes onto the 2D image. In AR/VR applications, the absence of intrinsic precludes placing multiple reconstructed 3D objects within a canonical 3D space. The absence of a reliable, assumption-frm monocular intrinsic calibrator has become bottleneck in deploying these 3D sensing applications.
Our method is motivated by the consistency between the monocular depthmap and surface normal map. In Fig. 1 (c) (c), incorrect intrinsic distorts the back mnok-proeetted 3D point cloud from the depthmap, resulting in distorted surface normals. Based on this, intrinsic optimal when the estimated monocular depthmap aligns consistently with the surface normal. We present solution to recover the complete 4 DoF intrinsic by leveraging the consistency between the surface normal and depthmap. However, the algorithm is numerically ill-conditioned as its computation depends on the accurate gradient of depthmap. This requires depthmap estimation with low bias and variance.
To resolve it, we propose alternative approach by introducing novel 3D monocular prior in complementation to the depthmap and surface normal map. We refer to this as the incidence field, which depicts the incidence ray between the observed 3D point and the projected 2D pixel on the imaging plane, as shown in Fig. (b). The combination of the incidence field and the monocular depthmap describes 3D point cloud. Compared to the original solution, the incidence field is direct pixel-wise parameterization of the camera intrinsic. This implies that a minimal solver based on the incidence field only needs to have low bias. We then utilize deep neural network to perform the incidence field estimation. A non-learning RANSAC algorithm is developed to recover the intrinsic parameters from the estimated incidence field.
We consider the incidence field also a monocular 3D prior. Similar to depthmap and surface normal, the incidence field is invariant to the image cropping or resizing. This encourages its generalization over in- wild images. Theoretically, we show the incidence field is pixel-wisely determined argument,
by depthmap and normal map. Empirically, to support datasets into comprehensive dataset with diverse indoor, outdoor, and object-centric images captured by different imaging devices. We further boost the variety of intrinsic by resizing and cropping the images in a similar manner as [23]. Finally, we include zero-shot testing samples to benchmark real-world monocular camera calibration performance.
combine multiple public
We showcase downstream applications that benefit from monocular camera calibration. addition to the aforementioned 3D sensing tasks, we present two intriguing additional applications. One detecting and restoring image resizing and cropping. When an image is cropped or resized, disrupts the assumption of central focal point and identical focal length. Using the estimated

---

Table 1: Camera Calibration Methods from Strong Relaxed Assumptions. Non-learning methods [23, 29, 34,37, 38, 51,62, 71-74] rely on strong assumptions. Learning-based methods [28,30, 38] relax the assumptions to using gravity-aligned panorama images during training. Our method makes no assumptions except for undistorted images. This enables training with any calibrated images and calibrates 4 DoF intrinsic.
[2A,29 29,29,88711-4
[4437,31,62]
28, 38
30
Ours
Degree of Freedom (DoF)
Assumption
Specifie-Objects
Manhattan
Train with Panorama Images
Undistorted Images
intrinsic parameters, restore the edited image by adjusting its intrinsic to a regularized form. The other application involves two-view uncalibrated camera pose estimation. With established image corresonndece, fundamental matrix [25] is determined. However, there does not exist injective mapping between the fundamental matrix and camera pose [27]. This raises a counter-intuitive fact: inferring the pose from two uncalibrated images infeasible. But our method enables uncalibrated two-view pose estimation by applying monocular camera calibration.
We summarize our contributions follows:
o Our approach tackles monocular camera calibration from a novel perspective by relying on monoc- ular 3D priors. Our method makes assumption for the to-be-calibrated undistorted image. o Our algorithm provides robust monocular intrinsic estimation for in wnll images, accompanied by extensive benchmarking and comparisons against other baselines.
We demonstrate its benefits on diverse and novel downstream applications.
2 Related Works
Monocular Camera Calibration with Geometry. One line of work [7, 34, 37, 52, 66] assumes the Manhattan World assumption [15], where all planes in 3D space are either parallel or perpendicular. Under the assumption, line segments in the image converge at the vanishing points, from which the intrinsic is recovered. LSD [61] and EDLine [2] develop robust line estimators. Others jointly estimate the horizon line and the vanishing points [40, 54, 70]. In Tab. 1, recent learning- based panorama images during
methods [28, 30,38,0,66,0 (8,63,64] [3,64] relax the assumption by using the gravity- training. Such data provides the groundrtuth vanishing point and horizon lines which are originally computed with the Manhattan World assumption. Still, the assumption constrains [28, 38, 63, 64] in modeling intrinsic 1 DoF camera. Recently, [30] relaxes the assumption 3 DoF via regressing the focal point. In comparison, our method makes no assumption except for undistorted images. This enables us to calibrate DoF intrinsic and train with any calibrated images.
Monocular Camera Calibration with Object. Zhang's method [73] based a checkerboard pattern is widely regarded as the standard for camera calibration. Several works generalize this method other geometric patterns such 1D objects [74], line segments [72], and spheres [71]. Recent works [29] and [23] extend camera calibration to real-world objects such as human faces. Optimizers, including BPnP [12] and PnP [56] developed. However, the usage of specific objects restricts their applications. In contrast, our approach applicable to any image.
Image Cropping and Resizing. Detecting photometric image manipulation [5, 17,24] is extensively rescarched. But few studies detecting geometric manipulation, e.g., resizing and cropping. On resizing, [18] regresses the image aspect ratio with a deep model. On cropping. a proactive method [67] is developed. We demonstrate that calibration also detects geometric manipulation. Our method does not need to encrypt images, complementing photometric manipulation detection.
Uncalibrated Two-View Pose Estimation. With the fundamental matrix estimated, the two-view camera pose is determined up to a projective ambiguity if images are uncalibrated. Alternative solutions [31, 13,,47.5 58] exist by employing deep networks to regress the pose. However, regression hinders the usage of geometric constraints, which proves crucial in calibrated two-view pose estima- tion [60, 75, Other works [20, [6] use more than two uncalibrated images for pose estimation. Our work complements prior studies by enabling an uncalibrated two-view solution.
Learnable Monocular 3D Priors. Monocular depth [39,766 76] and surface normals [6] are two established 3D priors. Numerous studies have shown learnable mapping between monocular images and corresponding 3D priors. Recently, [30] introduces the perspective field monocular 3D
3

---

A
Esimated KANSAC
(c)
w/o Asm. RANSAC Sampling
w/ Asm. Enumerate
Eq. (16)
Eq. (18)
(5t)
Eq.
(ox
(Px())}
Eq. (19)
(pz(') Px(-)}
Max
Max
Max
RANSAC
K
We illustrate the framework for the proposed monocular camera calibration algorithm. In (a), deep network maps the input image I the incidence field V. RANSAC algorithm recovers intrinsic from V. In (b), we visualize a single iteration RANSAC. An intrinsic is computed with two incidence vectors randomly sampled at red pixel locations. From Eq. (2), an intrinsic determines the incidence vector at a given location. The optimal intrinsic maximizes the consistency with the network prediction (blue and orange). Subfigure (c) details the RANSAC algorithm. Different strategies not assumed, we independently compute (fx, 6x) and (fy, by). ). assumed, there is only DoF of intrinsic. proceed by enumerating the focal length within predefined range to determine the optimal value.
Figure 2:
applied depending if simple camera is assumed. If
prior for inferring gravity direction. In this work, we suggest the incidence field is also a learnable monocular 3D prior. A common characteristic of monocular 3D priors is their exclusive relationship with 3D structures, making them invariant to 2D manipulations, egg., cropping, and resizing. In contrast, camera intrinsic changes on spacial manipulation. Unlike other monocular priors, computing the groundtruth incidence field is straightforward, relying solely on intrinsic and image coordinates.
3 Method
Our work focuses intrinsic estimation from in-the-wild monocular images captured by modern imaging devices. Hence we assume an undistorted image input. In this section, we first show how estimate intrinsic parameters by using monocular 3D priors, such the surface normal map and monocular depthmap. We then introduce the incidence field and show it learnable monocular 3D prior invariant to 2D spacial manipulation. Next, we describe the training strategy and the network used to learn the incidence field. After estimating the incidence field, we present RANSAC algorithm recover the DoF intrinsic parameters. Lastly, we explore various downstream applications the proposed algorithm. Fig. 2 shows the framework of the proposed algorithm.
3.1 Monocular Intrinsic Calibration
Our method aims use generalizable monocular 3D priors without assuming the 3D scene geometry. Hence, we start with monocular depthmap D and surface normal map N. Assume there exists learnable mapping from the input image to depthmap D and normal map N: D, N (I),), where Th is learned network. We denote the intrinsic Kiimle. K, and its inverse K-1 as:
0
w1/2
[2
0
b
1/Sa
0
by/ f
K simple
0
f
h/2
K =
0
fg
by
K-1 =
1/f
by/ fy
(1)
0
0
0
0
The notation Ksimlle suggests simple camera model with the identical focal length and central focal
point assumption. Given 2D homogeneous pixel location p p= y 1]" and its depth value d D(p), the corresponding 3D point P
[X
y
2) is defined as:
R-
X Y Z
T
Ux
P
d K
y
d
d
Ugy
d
fo
where the vector is an incidence ray, originating from the 3D point P, directed towards the 2D pixel p. and passing through the camera's origin. The incidence field V is the collection of incidence rays associated with all pixels at location p, where v V(p).
3.2 Monocular Intrinsic Calibration with Monocular Depthmap and Surface Normal
In this section, we explain how to determine the intrinsic matrix using the estimated surface normal map N and depthmap D. Given the estimated depth d D(p) and normal N(p) at 2D pixel

---

(a) GT Depthmup
(b) Depthmap Converted Surface Normal
La fv t fo Fav]
225 2/v 0 N/A]
(c) RGB Image
(d) Incidence Field V
Figure 3: In (a) and (b), we highlight the ground truth depthmap of smooth surface, such table's side. Even with the ground truth depthmap, the resulting surface normals exhibit noise patterns due the inherent high variance. This makes the intrinsic solver based on the consistency of the depthmap and surface normals numerically unstable. Further, (a)-(d) demonstrate a scaling and cropping operation applied to each modality. In (c). the intrinsic changes per operation, leading to ambiguity if a network directly regresses the intrinsic values. Meanwhile, the FoV is undefined after cropping. In comparison, the incidence field remains invariant image editing, the same as the surface normal and depthmap.
location p, a local 3D plane is described as:
n d +=
By taking derivative in a-axis and y-axis directions, we have:
n V v) =0, v 0.
(4)
Note the bias of the 3D local plane is independent of the camera projection process. Without loss of generality, we show the case of our method for x-direction. Expanding Eq. (4), we obtain:
bx fa
y by V. (d) n13 (d) 0,
n1 v (d.
T12
(5)
fy
where (d(d) represents the gradient the depthmap D in the z-axis and can be computed, for example, using Sobel filter [33]. Next, re-parametrize the unknowns in Eq. (5) get:
a1 JzJy + azJaby aaSyba + aasy + u552 0.
(6)
Divide both sides of the equation by fa to get:
a1fy + a2by + agrb, + air + a5 = 0,
where F. By stacking Eq. (7) with N 2 randomly sampled pixels, we acquire a linear system:
ANx4 X4x1 Bax1
(
where the intrinsic parameter to be solved is stored in vector X1x1 1fn by rb r|_ This solves the other intrinsic parameters as:
fo = Sy. by by Sa. Sy x
rbe
by
The known constants are stored in matrix A and B4x1- we choose N 4 in Eq. (8), we obtain minimal solver where the solution X is computed by performing Gauss-Jordan Elimination. Conversely, when > 4, the linear system is over-determined, and X is obtained using a least squares solver. The above suggests the intrinsic is recoverable from the monocular 3D prior.
3.3 Monocular Incidence Field as Monocular 3D Prior
Eq. (9) relies on the consistency between the surface normal and depthmap gradient, which may require a low-variance depthmap estimate, From Fig. 3, even groundtruth depthmap leads to spurious normal due to its inherent high variance. Thus, minimal solver in Eq. (9) can lead to poor solution.

---

As solution, we propose to directly learn the incidence field V as monocular 3D prior. In Eq. (2) and Fig. the combination of the incidence field V and the monocular depthmap D creates a 3D point cloud. In Eq. (3), the incidence field V can measure the observation angle between a 3D plane and the camera. Similar to depthmap D and surface normal map N, the incidence field V invariant to the image cropping and resizing. Consider an image cropping and resizing described as:
A fx
0
CCz
AK X, AK
A
Agy
K' = AKK,
(10)
where K' is the intrinsic after transformation. The surface normal map N and depthmap D after transformation is defined as:
(x(x') = N(x) = N(AK 1x), D'(x') = D(x) = D-AK 'x).
(11)
Similarly, the incidence field after transformation is:
V'(x) = (K') x K K- (AK) x' K- x x v= Vxx..
(12)
Eq. (12) shows that the incidence field is parameterization of the intrinsic matrix that is invariant to image resizing and image cropping. Other invariant parameterizations of the intrinsic matrix, such as the camera field of view (FoV), rely on the central focal point assumption and only cover a 2 DoF intrinsic matrix. An illustration is shown in Fig. 3.
Next, we prove the incidence field V a monocular 3D prior pixel-wisely determined by monocular depth D and surface normal N. Expanding Eq. (4) using the incidence vector v in Eq. (2), we get:
B1 V 2 (d) + nil d ) n2 V (d) + ns V (d) = O.
FT
+2
(13)
Combined with the axis-y constraint, the 2 DoF incidence vector [U' By 11 is uniquely solved. Eq. (13) gives identical solutions when depthmap D is adjusted by a scalar. This implies that the incidence field V is pixel-wise determined by the up-to-scale depth map and surface normal map, indicating a learnable mapping from the monocular image I to the incidence field V exists.
Given the strong connection between the monocular depthmap D and camera incidence field V. we adopt NewCRFs [69], a neural network used in monocular depth estimation, for incidence field estimation. We change the last output head to output a dimee-iimesiionll normalized incidence field V with the same resolution as the input image I. We adopt cosine similarity loss defined as:
N
V = (I),,,
L- N
Σ
VI(x))V
(14)
We normalize the last dimension of the incidence field to one before feeding to the RANSAC algorithm. That is to say, V(x,) 101 02/03 1]
jui
11a
3.4 Monocular Intrinsic Calibration with Incidence Field
Since the network inference executes on GPU device, we adopt a GPU-end RANSAC algorithm to recover the intrinsic K from the incidence field V. Unlike CPU-based RANSAC, we perform fixed K, iterations of RANSAC without termination. RANSAC, we use the minimal solver to generate
Ke candidates and select the optimal one that maximizes scoring function (see Fig. 2).
RANSAC w.o Assumption. From Eq. (2), the incidence vector v relates to the intrinsic K as:
K x
(15)
f=
fx
From Eq. (15), a minimal solver for intrinsic is straightforward. In the incidence field, randomly sample two incidence vectors v [0]
by
1] and 2
[02
uy
The intrinsic is:
ff x² n'=シ²
fz b
(16)
+ 2 v2fz)
bb = (x
+ 22f
Similarly, the scoring function is defined in a-axis and y-axis, respectively:
P= br. (v})
Ny. Σ
(1=t-- - by w < kz). PolSs. by (x). (v))
(1'0~ ubg-ba b,
r < kg) a7)
f=
1 Sb

---

Table 2: In-the-Wild Monocular Camera Calibration. We benchmark in-the- wild monocular camera calibration. We use the first datasets for training, and test on all 14 datasets. Except for MegaDepth, we synthesize novel intrinsic by cropping and resizing during training. Note the synthesized images violate the focal point and focal length assumption. [Key: ZS = Shot, Asm. Assumptions, Syn. Synthesized]
Dutaset
Calibration
Scene
ZS
Syn.
Perspective [30]
Ours
Ours
Asm.
Cr
E1
66
C/
Cy
NuScenes [T0]
Calibrated
Driving
0.378
0.286
0.102
0.087
0.402
0.400
KITTI [22]
Calibrated
Driving
0.631
0.279
0.111
0.078
0.383
0.368
Cityscapes [14]
Calibrated
Driving
X
V
0.624
0.316
0.108
0.110
0.387
0.367
NYUv2 [53]
Calibrated
Indoor
X
0.261
0.348
0.086
0.174
0.376
0.379
ARKitScenes [8]
Calibrated
Indoor
X
0.325
0.367
0.140
0.243
0.400
0.377
SUN3D [65]
Calibrated
Indoor
0.260
0.385
0.113
0.205
0.389
0.383
MVImgNet [68]
SIM
Object
X
V
0.838
0.272
0.101
0.081
0.108
0.072
Objectron [1]
SfM
Object
X
V
0.601
0.311
0.078
0.070
0.088
0.079
MegaDepth [4]
SIM
Outdoor
X
X
0.319
0.000
0.137
0.046
0.109
0.000
Waymo [58]
Calibrated
Driving
X 0.444
0.020
0.210
0.053
0.157
0.020
RGBD [55]
Pre-defined
Indoor
X
0.166
0.000
0.097
0.067
0.000
ScanNet [16]
Calibrated
Indoor
X 0.189
0.010
0.128
0.041
0.109
0.070
MVS [21]
Pre-defined
Indoor
X
0.185
0.000
0.170
0.028
0.127
0.000
Scenesl I [11]
Pre-defined
Synthetic
X
0.211
0.000
0.170
0.044
0.117
0.000
Table 3: Comparisons to Monocular Camera Calibration with Geometry on GSV dataset [4]. follow the
training and testing protocol of [38]. For a fair comparison, we convert the estimated intrinsic to camera FoV on the y-axis direction, following [30,38] and report our results w/ and w/o the assumptions.
(")
Upright [37]
Perceptual [28]
CTRL-C [38]
Perspective [30]
Ours
PAMPIS
EVPR1s
Kev
w/o Asm.
w/ Asm
Mean
9.47
4.37
3.59
3.07
2.49
2.47
Median
4.42
3.58
2.72
2.33
1.96
1.92
where N₂ and / by are the number of sampled pixels and the threshold for axis-z/ yyy directions.
RANSAC w/ Assumption. If simple camera model is assumed, iee, intrinsic only has an unknown focal length, it only needs to estimate 1-DoF intrinsic. We enumerate the focal length candidates as:
S = [Smin + 8 Ny
(Jmax Smin) |osisN))
(18)
The scoring function under the scenario is defined as the summation over z-axis and y-axis: plJ, (x), {v})= Pa w/2, (x), (v)) + Py(Sp. h/2, (v)).
(19)
3.5
Downstream Applications
Image Crop & Resize Detection and Restoration. Eq. (10) defines crop and resize operation:
x' AA K' AKK, I'xx) I'(A I(x).
(20)
When modified image I' is presented, our algorithm calibrates its intrinsic K' and then:
Case 1: The original intrinsic K is known. E.g. we obtain K from the image-associated EXIF file [3]. Image manipulation is computed as AK K' 1-1, A manipulation is detected if AK deviates from an identity matrix. The original image restores as I(x) ('( Interestingly, the four corners of image I' are mapped to a bounding box in original image under manipulation AK. We thus quantify the restoration by measuring the bounding box. See Fig. 4.
Case 2: The original intrinsic K is unknown. We assume the genuine image possess an identical focal length and central focal point. Any resizing and cropping are detected when matrix K' breaks this assumption. Note, the rule cannot detect aspect ratio preserving resize or centered crop. We restore the original image by defining an inverse operation AK restore K' to an intrinsic fits the assumption.
3D Sensing Related Tasks. Intrinsic estimation can enable multiple applications. E.g., depthmap to point cloud, uncalibrated two-view pose estimation, etc.
4 Experiments
Implementation Details Our network is trained using the Adam optimizer [32] with a batch size of 8. The learning rate is 1e and the training process runs for 20, 000 steps. Our RANSAC algorithm
7

---

Table 4: Comparisons Monocular Camera Calibration with Object. We compare the recent Face-
Calib [29], which calibrates the camera using video containing human faces. We report our results w/ and w/
assuming a simple camera model. We perform zero-shot testing using the same model benchmarked in Tab. 2.
Methods
BIWIRGBD-ID [48]
CAD-120 [59]
64
6/.
%
Louraki [45]
0.682
0.602
0.602
0.357
0.222
0.732
0.732
0.732
0.255
0.180
Fetzer [20]
0.845
0.845
0.845
0.001
0.005
0.679
0.679
0.679
0.001
0.005
BPaP [12] everre
0.675
0.675
0.675
0.322
0.179
1.178
1.178
1.178
0.103
0.129
FaceClalib [29]
0.133
0.026
0.022
0.063
Ours
0.034
0,029
0.018
0.020
0.011
0.018
0.137
0.137
0.054
0.042
0.042
0.005
Assumptions
0.019
0.019
0.019
0.000
0.000
0.000
0.047
0.047
0.047
0.000
0.000
0.000
executed on a GPU, performing K, = 2, 0448 iterations, with each iteration including a random sample of Ke 20, 000 incidence vectors. During both training and testing, the image is resized to
resolution of 480 x 640. The threshold for determining RANSAC inliers is k2 ky = 0.02.
Evaluation Metrics We follow [29] in assessing camera intrinsic estimation using the metrics:
112 fall,
|r Sell
Eb. 16 ball
1 blll
ff=-
Eg, S
Cby
2
(21)
Jx
w
We additionally consolidate the performance along the x-axis and y-axis into comprehensive mea- surements, Ey = max(ef,, er,) and e, = max Cb,.).
4.1 Monocular Camera Calibration In-The-Wins
Our method is trained whenever a calibrated intrinsic is provided, making applicable to wide range of publicly available datasets. In Tab. 2, we incorporate datasets of different application scenarios, including indoor, outdoor scenes, driving, and object-centric scenes. Many of the datasets utilize only single type of camera for data collection, resulting in scarcity of intrinsic variations. Similar to [38]. we employ random resizing and cropping to synthesize more intrinsic, marked in first resize all images
Datasets.
Tab. 2 column "Syn.". In augmentation, We then uniformly random resize up to two times its size and subsequently crop to a resolution of 480 x 640. As MegaDepth [41] collects images captured by various cameras from the Internet, we disable its augmentation. We document the intrinsic parameters of each dataset in Supp.
resolution of 480 x 640.
In Tab. 2 column "Calibration". assess intrinsic quality into various levels. "Calibrated" suggests accurate calibration with a checkerboard. "Pre-defined" is less accurate, indicating the default intrinsic provided by the camera manufacturer without a calibration process. "SfM" signifies that the intrinsic is computed via an SfM method [52].
In-The Wild Monocular Camera Calibration. We benchmark in-the-wild monocular calibration performance on Tab. 2. For trained datasets, except for MegaDepth, we test synthetic data using random cropping and resizing. For the unseen test dataset, we refrain from applying any augmentation to better mimic real-world application scenarios.
We compare the recent baseline [30], which regresses intrinsic via deep network. Note, [30] can not train on arbitrary calibrated images as requiring panorama images in training. A fair comparison using the same training and testing images is in Tab. 4 and See. 4.2. [30] provides models with two variations: one assumes a central focal point, and another does not. We report with the former model whenever the input image fits the assumption. From Tab. 2, our method demonstrates superior generalization across multiple unseen datasets. Further, the result w/ assumption outperforms w/o assumption whenever the input images the assumption. Tested on an RTX-2080 GPU, the combined network inference and calibration algorithm runs average in 87 ms per image.
4.2 Monocular Camera Calibration with Geometry
Methods in this line of research hold a Manhattan World assumption, positing that images consist of planes that either parallel perpendicular each other. Stated in Tab. 1, baselines (28,30, 38] relax the assumption to training data. Our method imposes no other assumption except for undistorted images in both training and testing.
This brings three benefits. First, the assumption restricts their training to panorama images. In contrast, our model is trainable with any calibrated images. This yields improved generalization,
8

---

(a) Input Incidence
(b) Est. Restoration
(c) GT. Restoration
(d) Original Image
Figure 4: Image Crop & Resize Detection and Restoration. Image editing, including cropping and resizing
changes intrinsic. As Sec. 3.5, monocular calibration is applicable to detect and restore image manipulations. We visualize the zero-shot samples on ScanNet and Waymo. More examples in Supp.
Table 5: Image Crop and Resize Restoration. Stated in Sec. 3.5, method also encompasses the restoration image manipulations. Use model reported in Tab. 2, we conduct evaluations on both seen and unseen datasets.
Methods
mIOU
22
Acc
mlOU
[53v2[53]
Ace
mlOU
ARKiSScenes
Acc
mIOU
Waymo
[58]
Acc
mlOU
RGBDD]55]
Acc
mIOU
ScanNet
Acc
[6]]
mlOU
MVST2]
Acc
Baseline
0.686
0.795
0.021
0.710
0.586
0.518
0.581
0.721
0.036
0.681
0.597
0.811
0.595
0.667
0.779
0.856
0.691
0.837
0.681
0.796
0.709
as shown in Tab. Second, constrains the baselines simple camera parameterized by FoV. We consider the proposed incidence field more generalizable and invariant parameterization of intrinsic. E.g., while FoV remains invariant image resizing, still changes after cropping. However, the incidence field is unaffected in both cases. In Tab. 3, the substantial improvement we achieved (0.60 3.07 2.47) over the recent SoTA [30] empirically supports our argument. Third, our method calibrates the 4 DoF intrinsic with non-learning RANSAC algorithm. Baselines instead regress the intrinsic. This renders our method more robust and interpretable. In Fig. (b), the estimated intrinsic quality is visually discerned through the consistency achieved between the two incidence fields.
4.3 Monocular Camera Calibration with Objects
We compare to the recent object-based camera calibration method FaceCalib [29]. The baseline employs a face alignment model to calibrate the intrinsic over a video. Both [29] and ours perform zero-shot prediction. We report performance using Tab. 2 model. Compared to [29], method is more general as it does not assume human face present the image. Meanwhile, [29] calibrates video, while ours is monocular method. For fair comparison, results as an averaged error over the videos. We report results w/ and w/o assuming a simple camera model. Since the tested image has central focal point, when the assumption applied, the error of the focal point diminished. In Tab. 4, we outperform SoTA substantially.
report the iideo-based
4.4 Downstream Applications
Image Crop & Resize Detection and Restoration. In Sec. 3.5, our method also addresses the detection and restoration of geometric manipulations in images. Using the model reported in Tab. 2, we benchmark its performance in Tab. 5. Random manipulations following Sec. 4.1 contribute to 50% of both train and test sets, and the other 50% are genuine images. In Tab. 5, we evaluate restoration with mIOU and report detection accuracy (i.e. binary classification of genuine vs edited images). From the table, our method generalizes to the unseen dataset, achieving an averaged mIOU of 0.680. Meanwhile, we substantially outperform the baseline, which trained to directly regresses the intrinsic. The improvement suggests the benefit of the incidence field as an invariant intrinsic parameterization. Beyond performance, our algorithm is interpretable, In Fig. 4, the perceived image geometry is interpretable for humans.

---

OmmaSD [o] NY our iumisic
OmmmL3 [S]
Ocan3 P nf our intrinsic
Figure 5: In-the- Wild Camera Calibration Benefits In-the-Wild 3D Object Detection [9]. Our estimated intrinsic improves 2D projection of estimated bounding boxes [9]. highlighted arrows.
Restreed T
Cropped Tj
Original Image
Cropped To
Restored T2
Figure 6: Broader Impact. Our method detects geometric manipulation and helps press image authenticity, In the ligure, our method restores cropping and the aspect ratio. Orange arrows mark estimated incidence field.
Table 6: Uncalibrated Two-View Camera Pose Estimation. use model reported Tab. and assume distinct camera models for both frames. During calibration, we apply the simple camera assumption. The last two rows ablate the performance using GT intrinsic and our estimated intrinsic.
Methods
Calibrated
ScanNer [16]
MegaDepth [41]
@5
@10
@20
@5
@10
@20'
SuperGlue [50] EVPR11
16.2
33.8
51.8
42.2
61.2
75.9
DRC-Net [44] ICASSP222
7.7
17.9
30.5
27.0
42.9
58.3
LoFTR [57] evenn
22.0
40.8
57.6
52.8
09.2
81.2
Former [13] Rocv12
25.6
46.0
63.3
55.3
71.5
83.1
PMatch [78] evee22
29.4
50.1
67.4
61.4
75.7
85.7
PMatch [78] CVPR223
X
11.4
29.8
49.4
16.8
30.6
47.4
Uncalibrated Two-View Camera Pose Estimation. With correspondence between images, one can infer the fundamental matrix [25]. However, the pose between two uncalibrated images is determined by projective ambiguity. Our method eliminates the ambiguity with monocular camera calibration. In Tab. 6, we benchmark the uncalibrated two-view pose estimation and compare it to recent basclines. The result reported using the model benchmarked in Tab. 2 and assumes unique intrinsic for both images. We perform zero-shot in ScanNet. For MegaDepth, it includes images collected over the Internet with diverse intrinsics. Interestingly, in ScanNet, our uncalibrated method outperforms calibrated one [44]. In Supp. we plot the curve between pose performance and intrinsic quality. The challenging setting suggests itself an ideal task to evaluate the intrinsic quality.
In-the- Wild Monocular 3D Sensing. In-the-wild 3D sensing, e.g., monocular 3D object detec- tion [9]. requires intrinsic to connect the 3D and 2D space. In Fig. 5, despite 3D bounding boxes being accurately estimated, incorrect intrinsic produces suboptimal 2D projection. Our calibration results suggest monocular incidence field estimation is another essential monocular 3D sensing task.
5 Conclusion
We calibrate monocular images through a novel monocular 3D prior referred as incidence field. The incidence field is pixel-wise parameterization of intrinsic invariant image resizing and cropping. A RANSAC algorithm developed to recover intrinsic from the incidence field. We extensively benchmark our algorithm and demonstrate robust in-the-wild performance. Beyond calibration, show multiple downstream applications that benefit from our method.
Limitation. In real application, whether to apply the assumption still requires human input.
Broader Impacts. Our method detects image spacial editing. If the Exif files are provided, it further localizes the modification region. Shown in Fig. 6, it helps improve press image authenticity.
10

---

References
[1] Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. Objectron:
A large scale dataset object-centrie videos the wild with pose annotations. In CVPR, 2021.
[2] Cuncyt Akinlar and Cihan Topal. EDLines: A real-time line segment detector with false detection control. Pattern Recognition Letters, 2011.
[3] Paul Alvarez. Using extended file information (exif) file headers in digital evidence analysis. IDDE 2004.
[4] Dragomir Anguelov, Carole Dulong, Daniel Filip, Christian Fruch, Stéphane Lafon, Richard Lyon, Abhijit Ogale, Luc Vincent, and Josh Weaver. Google street view: Capturing the world street level. Computer, 2010. 7
[5] Vishal Asnani, Xi Yin, Tal Hassner, Sijia Liu, and Xiaoming Liu. Proactive image manipulation detection. In CVPR, 2022. 3
[6] Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Estimating and exploiting the aleatoric uncertainty
in surface normal estimation. In ICCV, 2021. 3
[7] Olga Barinova, Victor Lempitsky, Elena Tretiak, and Pushmeet Kohli. Geometric image parsing in man-made environments. In ECCV, 2010. 1,3
[8] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARKitScenes- a diverse real-world dataset for 3D indoor scene understanding using mobile RGB-D data. In NeurlPS Dataxets and Benchmarks Track, 2021. 7. 9
[9] Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3D: A large benchmark and model for 3D object detection in the wild. In CVPR, 2023. 10
[10] Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal datasct for autonomous driving. In
CVPR, 2020. 7
[11] Angel Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3D model repository. arXiv preprint arXiv:1512.0000 2015. 7
[12] Bo Chen, Tat-Jun Chin, and Nan Li. BPnP: Further empowering end-to-end learning with back-propagatabbe geometric optimization. arXiv: 1909. 1909.06043, 2019. 8,8
[13] Hongkai Chen, Zixin Luo, Lei Zhou, Yurun Tian, Mingmin Zhen, Tian Fang. David Mckinnon. Yanghai Tsin, and Long Quan. ASpanFormer: Detector-free image matching with adaptive span transformer. In ECCV, 2022. 10
[14] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urhan scene understanding.
In CVPR. 2016. 7
[15] James M Coughlan and Alan I. Yuille. Manhattan world: Compass direction from single image by bayesian inference. In ICCV, 1999. 1.3
[16] Angela Dai, Angel Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annnn annotated 3D reconstructions of indoor scenes. In TPAMI, 2017. 7,9, 10
[17] Hao Dang, Feng Liu, Joel Stchouwer, Xiaoming Liu, and Anil Jain. On the detection of digital face manipulation. In CVPR, 2020. 3
[18] Bibhash Das, Mrutyunjay Biswal, Abhranta Panigrahi, and Manish Okade, CNN based image resizing
detection and resize factor classification for forensic applications. In ICORT, 2021. 3
[19] Patrick Denis, James Elder, and Francisco Estrada. Efficient edge-based methods for estimating manhattan
frames in urban imagery. In ECCV, 2008. 1,
[20] Torben Fetzer, Gerd Reis, and Didier Stricker. Stable intrinsic auto-calibration from fundamental matrices of devices with uncorrelated camera parameters. In WACV, 2020. 3, 8
[21] Simon Fuhrmann, Fabian Langguth, and Michael Goesele. Mve-a multi-view reconstruction environment.
In GCH, 2014. 7.9
[22] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The KITTI
dataset. IJRR, 2013. 7.9
[23] Alexander Grabner, Peter Roth, and Vincent Lepetit. GPºC: Geometric projection parameter consensus for joint 3D pose and focal length estimation in the wild. In ICCV, 2019. 1,2,3
[24] Xiao Guo, Xiaohong Liu, Zhiyuan Ren. Steven Grosz, Iacopo Masi, and Xiaoming Liu. Hierarchical
fine-grained image forgery detection and localization. In CVPR, 2023. 3
[25] Richard Hartley. In defense of the eight-point algorithm. TPAMI, 1997. 3.
[26] Richard Hartly. Kruppa's equations derived from the fundamental matrix. TPAMI, 1997.
[27] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university
press, 2003. 1,
11

---

[28] Yannick Hold- Hlol-Geeofrroy Kalyan Sunkavalli, Jonathan Eisenmann, Matthew Fisher, Emiliano Gambaretto, Sunil Hadap, and Jean-François Lalonde. perceptual measure for deep single image camera calibration.
In CVPR, 2018 3,3,7, 7.8
[29]
Masa Hu, Garrick Brazil, Nanxiang Li, Liu Ren, and Xiaoming Liu. Camera self-calibration using human
faces. In FG, 2023. 3, 8,
[30] Linyi Jin, Jianming Zhang, Yannick Noof-Geofrrry Oliver Wang, Kevin Blackbura-Matzen. Matthew Sticha, and David F Fouhey. Perspective fields for single image camera calibration. In CVPR, 2023. 1, 3,
7, 8, 9.
[31] Fadi Khatib, Yuval Margalit, Meirav Galun, and Ronen Basri. GRelPose: Generalizable end-to-end relative
camera pose regression. arXiv preprint arXiv:2211. 149500 2022. 3
[32]
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2023.
[33] Josef Kittler. On the accuracy of the sobel edge detector, Image ond Vision Computing, 1983. 5
[34] Jana Košecká and Wei Zhang. Video compass. In F.C.V, 2002. 1,3
[35] Abhinav Kumar, Garrick Brazil, Enrique Corona, Armin Parchami, and Xiaoming Liu. Deviant: Depth
equivariant network for monocular 3d object detection. In ECCV, 2022. 1
[36] Abhinav Kumar, Garrick Brazil, and Xiaoming Liu, Groomed-nm Grouped mathematically differentiable
nms for monocular 3d object detection. In CVPR, 2021.
[37] Hyunjoon Lee. Eli Shechtman, Jue Wang. and Seungyong Lee, Automatic upright adjustment of pho-
tographs with robust camera calibration. TPAMI, 2013. 1, 3.
[38] Jinwoo Lee. Hyunsung Go, Hyunjoon Lee, Sunghyun Cho, Minhyuk Sung. and Junho Kim. Ctrl-C: Camera calibration transformer with cissciaasiiiiaatin. In ICCV, 2021. 3, 7, 8
[39] Jin Han Lee, Myung Kyu Han, Dong Wook Ko, and Il Hong Suh. From big small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv:1907. (00,I12607. 2019.
[40] Haoang Li. Ji Zhao, Jcan- -harrhhh Bazin, Wen Chen, Zhe Liu, and Yun-Hu Liu. Quasi-globally optimal
and efficient vanishing point estimation in manhattan world. In ICCV, 2019. 3
[41] Zhenggi Li and Noah Snavely. MegaDepth: Learning single-view depth prediction from internet photos.
In CVPR, 2018. 7, 8, 10
[42] Feng Liu and Xiaoming Liu. Voxel-based detection and reconstruction of multiple objects from a single image. In NeurlPS, 2021.
[43] Feng Liu and Xiaoming Liu. 2d gans meet unsupervised single-view 3d reconstruction. In ECCV, 2022. 1
[44] Jinjiang Liu and Xueliang Zhang. DRC-NET: Densely connected recurrent convolutional neural network
for speech dereverberation. In ICASSP, 2022. 10
[45] Manolis Lourakis and Rachid Deriche. Camera self-calibratinn using the singular value decomposition of the fundamental matrix: From point correspondences to 3D measurements. PhD thesis, INRIA, 1999. 8 [46] Yi Ma, Stefano Soatto, Jana Košecká, and Shankar Sastry. An invitation to 3D vision: from images to geometric models, volume 26. Springer, 2004.
[47] Taroslav Melekhov, Juha Ylioinas, Juho Kannala, and Esa Rahtu. Relative camera pose estimation using convolutional neural networks. In AC/VS, 2017.
[48] Cosimo Patruno, Roberto Marani, Grazia Cicirelli, Ettore Stella, and Tiziana D'Orazio. People re- identification using skeleton standard posture and color descriptors from RGB-D data. Pattern Recognition,
2019. 8
[49]
Simon Placht, Peter Fürsattel, Etienne Assoumou Mengue, Hannes Hofmann, Christian Schaller, Michael Balda, and Elli Angelopoulou. Rochade: Robust checkerboard advanced detection for camera calibration. In ECCV, 2014. 1
[50] Paul-Edouan Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGiuc: Learning
feature matching with graph neural networks. In CVPR, 2020. 10
[51] Grant Schindler and Frank Dellaert. Atlanta world: An expectation maximization framework for simul- taneous low-level edge grouping and camera calibration in complex man-made environments. In CVPR, 2004. 1,3
[52]
Johannes Schonberger and Jan-Michae Frahm. fimactre-from-mootie revisited. In CVPR, 2016. 8 Nathan Silherman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support
[53]
inference from rgbd images. ECCV, 2012. 7,9
[54] Gilles Simon, Antoine Fond, and Maric-Odile Berger. A-Contrario horizon-first vanishing point detection
using second-order grouping laws. In ECCV, 2018. 3
[55] Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of RGBD SL.AM systems. In IROS, 2012. 7,9
[56] Peter Sturm. Multi-view geometry for general camera models. CVPR, 2005.
[57] Jiaming Sun, Zehong Shen, Yuang Wang. Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-fre free local
feature matching with transformers. In CVPR, 2021. 10
[58] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens,
12

---

Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020. 7.9
[59] J Sung, Hema Koppula, Bart Selman, and Ashutosh Saxena. Cornell activity datasets: CAD-60 & CAD-120.
8
[60]
Zachary Teed and Deng. VeD: Video depth with differentiable structure from motion. In ICLR. 2020. 3
[61] Rafacl Von Gioi, Jeremie Jakubowicz, Jean- Michel Morel, and Gregory Randall. L.DD: fast line segment detector with false detection control. TPAMI, 2008,
[62]
Horst Wildenauer and Allan Hanbury. Robust camera self-calibration from monocular images of manhattan worlds. In CVPR, 2012. 1.3
[63] Scott Workman, Connor Greenwell, Menghua Zhai, Ryan Baltenberger, and Nathan Jacobs. Deepfocal: A method for direct focal length estimation. In ICIP, 2015.
[64] Scntt Workman, Menghua Zhai, and Nathan Jacobs. Horizon lines in wil. BMVC, 2016. 1, Jianxiong Xiao, Andrew Owens, and Antonio Torralba. Sun3D: A datahase of big spaces reconstructed
[65]
using SFM and object labels. In ICCV, 2013. 7
[66]
Yiliang Xu, Sangmin Oh, and Anthony Hoogs. A minimum error vanishing point detection approach for uncalibrated monocular images of man-made environments. In CVPR, 2013. 1,
[67]
Qichao Ying. Hang Zhou, Zhenxing Qian, Sheng Li, and Xinpeng Zhang. Robust image protection
countering cropping manipulation. arXiv preprint vrXiv:2206.02405, 2022. 3
[68]
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan. Chenming
Zhu, Zhangyang Xiong. Tianyou Liang, Guanying Chen, Shuguang Cui, and Xiaoguang Han. MelmgNet:
A large-scale dataset of multi-view images. In CVPR, 2023. 7
[69] Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. NeW CRFs: Neural window fully-connected CRFs for monocular depth estimation. CVPR, 2022.
[70] Menghua Zhai, Scott Workman, and Nathan Jacobs. Detecting vanishing points using global image context
in manhatantt world. In CVPR, 2016.
[71] Hui Zhang, Wong Kwan-yee, and Guoqiang Zhang. Camera calibration from images of spheres. TPAMI, 2007. 3
[72] Yueqiang Zhang, Langming Zhou, Haibo Liu, and Yang Shang. A flexible online camera calibration using line segments. Journal of Sensors.
[73] Zhengyou Zhang. flexible new technique for camera calibration. TPAMI, 2000.
[74]
Zhengyou Zhang, Camera calibration with one- me-dimensional objects. TPAMI, 2004, 3
[75] Wang Zhao, Shaohui Liu, Yezhi Shu, and Yong-Jin Liu. Towards better generalization: Joint depth-pose
learning without posenet. CVPR, 2020.
[76] Shengjie Zhu, Garrick Brazil, and Xiaoming Liu. The edge of depth: Explicit constraints between
segmentation and depth. CVPR, 2020. 3
[77] Shengjie Zhu and Xiaoming Liu. LightedDepth: Video depth estimation in light of limited inference view
angles. In CVPR, 2023. 3
[78] Shengjie Zhu and Xiaoming Liu. PMatch: Paired masked image modeling for dense geometric matching. In CVPR, 2023. 10
13


Reviews:
[{"review_id": "5K6gUjJA3l", "reviewer": "Reviewer_TGwc", "summary": "The authors propose a novel learning based approach for 4 DOF camera intrinsics calibration from a single image in the wild. They propose to use a neural network to predict the incidence rays for each pixel in an image from which the camera intrinsics can be recovered. The authors motivate the prediction of the incidence rays as an alternative for predicting the depth and normals, as the computation of the intrinsics from these signals involves derivatives it is very succesible to noise. The evaluation of the authors shows great potential of this method to generalize to images in the wild and show potential applications like uncalibrated 2-view pose estimation and image transformation detection.", "questions": "Please add the metrics to the tables, introduce them and provide a reference for the definition.\nAdd the description of the baseline in 4.4.\nHow does the performance depend on the image content and the sampling of pixels? E.g. what samples are usually providing the best hypotheses and what would intuitively be a minimal image content that would allows the model to predict the incident rays? The same as for depth and normal prediction?", "limitations": "Similar to depth and normal prediction there are probably cases where the image content is not sufficient to predict incidence rays. The authors did not discuss this case as far as I can see.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The idea of leveraging the relationship betwen the scene depth and normals and the incidence rays is very interesting.\nThe explanations are great and the neccesary derivations are all there. There are comparisons to both monocular methods using geometry and methods using known objects.", "weaknesses": "Most of the tables (2, 4 & 6) with the quantitative results do not provide a metric unit and while the results are compared to existing methods, the metrics should be introduced properly. For 4.4 and table 5 the description and or reference of the baseline is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Response to R1 TGwc**\n\nWe sincerely appreciate the valuable feedback provided by the reviewer to enhance our manuscript quality. We also agree on the importance of properly introducing metric units in tables and providing clear references for baselines. We will address these aspects appropriately. We offer the following responses to the raised questions:\n\n**R1, Q1: Missing table metric units.**\n  \nThank you for bringing this to us!\nTabs. $2$ and $4$ evaluation metrics are at L$31$ of the supplementary.\nFor Tab. $6$, we follow the evaluation protocols of baseline LoFTR. The reported AUC of the pose error is measured at thresholds (5°, 10°, 20°), with the pose error defined as the maximum angular error in rotation and translation. Please refer to Sec. $4.2$ of LoFTR for the detailed evaluation protocols.\nWe have revised our main paper to incorporate these metrics.\n\n**R1, Q2: Missing reference to Sec. $4.4$ and Tab. $5$ baselines.**\n  \nWe apologize for the missing details. In Tab. $5$, the baseline shares an identical network structure, with the last layer modified to a fully connected layer for direct regression of the $4$ degrees of freedom (DoF) intrinsic. During training and testing, we normalize the intrinsic matrix by resizing the image height and width in the range [0, 1], which aids model convergence. \nWe compare the baseline and our method's estimated intrinsics using the same evaluation protocol.\n\nThe baselines in Tab. $6$ of Sec. $4.4$ follow LoFTR in estimating image correspondence. With the estimated correspondences, they apply an OpenCV-based five-point algorithm with RANSAC to estimate the two-view camera pose.\n\n**R1, Q3: Calibration performance w.r.t. sampling strategy.**\n\nThat is a very insightful question! Monocular camera calibration eventually relies on image projective distortion. This leads us to think sampling around image edges could enhance results, as projective distortion is most discernible near borders. We run the following ablation to verify the reviewer's insight.\n\nWith normalized image coordinates ranging from [-1, 1], we ablate calibration performance through sampling: initially around the image border and then spanning the entire image. We define the sampling area w.r.t. threshold $k$ as:\n\n {$ (x, y) \\mid  |x| \\geq k, |y| \\geq k, x \\in [-1, 1], y \\in [-1, 1] $} \n\nwhere $k$ closer to $1$ refers to sampling around the image edges, while $k=0.0$ spans the entire image.\n \nWe evaluate the MegaDepth, ScanNet, and Waymo datasets, reporting calibration performance with the simple camera assumption applied in Tab. C. The performance at $k=0.0$, thus, aligns with the results in Tab. $2$ of the main paper.\n\nTable C. Calibration performance with sampling strategy.\n\n| $e_f$ \\ $k$ |  0.7  |  0.6  |  0.5  |  0.4  |  0.3  |  0.2  |  0.1  |  0.0  |\n|:-----------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n|  MegaDepth  | **0.106** | 0.107 | 0.107 | 0.107 | 0.108 | 0.108 | 0.109 | 0.109 |\n|   ScanNet   | **0.098** | 0.104 | 0.108 | 0.108 | 0.107 | 0.108 | 0.108 | 0.109 |\n|    Waymo    | **0.149** | 0.151 | 0.154 | 0.157 | 0.158 | 0.157 | 0.157 | 0.157 |\n\nInterestingly, we observe performance improvement when sampling near the border $(k=0.7)$, which partially supports our argument. For instance, the improvement in ScanNet is from $0.109$ to $0.098$. \n\n**R1, Q4: Minimal image content to predict the incidence field.**\n\nWe believe that estimating the incidence field is easier on images with projective distortions. Following this reasoning, we consider object-centric images challenging, as the camera is too close to observe the projective distortion.\n\nYet, a trade-off exists. Although estimating the incidence field in object-centric images is challenging, its accuracy has less impact to 3D sensing. In other words, an accurate 3D structure remains feasible despite noisy intrinsic parameters. A supporting fact is that many 3D object and 3D face modeling works use predefined intrinsic or weak projection to render their 3D model onto 2D images. We give two examples:\n- [img2pose: Face Alignment and Detection via 6DoF, Face Pose Estimation, CVPR'21]\n- [Fully understanding generic objects: Modeling, segmentation, and reconstruction, CVPR'21]\n\n**R1, Q5: Limitation in image hard to predict Incidence Rays.**\n\nWe agree! An uncertainty measurement about the incidence field prediction will be beneficial!"}, {"review_id": "DOuii4Q82O", "reviewer": "Reviewer_wjnN", "summary": "An in-the-wild monocular camera calibration method is proposed. It allows to estimate the focal lengths $f_x$ and $f_y$ as well as the optical center $b_x$, $b_y$ without any additional information such as a checkerboard or the Manhattan world assumption. The proposed method consists in employing a neural network to predict the incidence field, followed by a RANSAC. Several experiments are performed to demonstrate the ability of the proposed method to perform in-the-wild calibration.", "questions": "After carefully reading the paper, I recommend to reject it. I believe the contributions are not sufficient for a conference like NeurIPS and not interesting for the broader NeurIPS community. I suggest to submit the paper to another conference like 3DV or CVPR.", "limitations": ".", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed approach is simple and efficient.\n2. Training a network to predict the incidence field seems straightforward (the authors took a depth prediction network architecture and simply changed the last layer and the loss). ", "weaknesses": "1. The idea of using a sota depth prediction network to predict the incidence field (section 3.4) seems new, but this is a rather weak contribution. If I am not mistaken, it essentially consists in taking the network from a github page, change the last layer and the cost function and retrain the network on the same data. This contribution seems weak for a conference like NeurIPS.\n2. The idea of using the incidence field to estimate the calibration parameters (section 3.5) seems new, but the derivation is trivial: here the minimal solver is the solution of a trivial linear system (it takes a few minutes to derive, compared to Grubner-basis-based minimal solvers that we encounter in essential matrix estimation). In terms of contribution, I believe this is not strong enough for NeurIPS. Btw, I believe eq.17 and 18 do not correspond to a RANSAC since you simply quantize the focal length space and test each value. \n3. I believe section 3.2 is not a contribution, is it ? It is said that this calibration method will not produce good results but I could not find any experiment using it. I suggest to include this experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Response to R2 wjnN**\n\nWe thank the reviewer for the positive comments on the soundness and presentation of the paper. We greatly appreciate the valuable, constructive criticism. Your concern helps us to further address the missing connection between Sec. $3.2$ and other sections in the manuscript. We address the concerns in detail.  \n\n**R2, Q1: Using the SoTA depth network to predict the incidence field is a weak contribution.**\n\nWe do not claim any contribution in learning the incidence field. Therefore, we briefly discuss learning the incidence field in a short paragraph (Sec.$3.4$) of our methodology section.\n\n**R2, Q2 - Part1: Monocular Camera Calibration with incidence field provides insufficient contributions to the community.**\n\nWe restate our contributions in monocular camera calibration:\n\n1. **Assumption-free method for undistorted in-the-wild images.** \n\tMonocular image-based 3D sensing is gaining rapid attention, and most of these sensing methods require known camera intrinsics. Therefore, an intrinsic estimation method for monocular images is the need of the hour. In Fig. A of the one-page rebuttal PDF, we enhance the monocular 3D object detection results of a recent CVPR 2023 work, Omni3D, by replacing their predefined intrinsics with estimated intrinsics from our method. \n\n2. **First to infer the $4$ DoF intrinsics without extrinsics from monocular 3D prior.** Prior works always infer intrinsics alongside the camera extrinsics, either as camera pose ([25]) or perspective angles ([24, 26, 30, 31]).\n\n3. **Simple and Neat solution with competitive performance**. Our method eliminates the need for extrinsic estimation in camera calibration, simplifying the algorithm and enhancing performance.\n\n**R2, Q3: Sec. $3.2$ is not a contribution.**\n\nWe respectfully disagree. While the algorithm in Sec. 3.2 is not finally adopted and is not a contribution, the following insights we learned from this algorithm are contributions.\n\n- First, Sec. $3.2$ suggests the one can infer the intrinsics from monocular 3D prior.\n\n\t To the best of our knowledge, we are the first to show this. \n\t Prior works always infer intrinsics alongside camera extrinsics. Some use 3D-2D correspondence sets with a predefined 3D template [25] (Tab. $4$ baseline). Others jointly solve intrinsics with the camera roll and pitch angle [24, 26, 30, 31] (Tab. $3$ baselines). Sec. $3.2$ provides one solution which **avoids** extrinsic estimation by directly solving $4$ DoF intrinsic from monocular 3D prior. The disentanglement also simplifies the design of the minimal solver, representing a significant contribution of our work.\n  \n- Second, Sec. $3.2$ suggests the incidence field is a 3D prior.\n\n  Proving the incidence field as a 3D prior is crucial, as the incidence field is not a typical 3D prior. \n  Monocular depthmap and surface normal are image texture dependent, which changes w.r.t. image content. In comparison, the incidence field is less texture dependent. For a given image with size $H \\times W$, its incidence field is identical over different images. However, the incidence field still depends on image content. Given an image, after cropping and resizing, the incidence field changes accordingly, as the Fig. $3$ of the main paper. In Sec. $3.2$, Eqn. $4$ suggests the incidence vector is uniquely determined by the surface normal and monocular depthmap. The incidence vector $\\mathbf{v}$ in Eqn. $4$ has $2$ DoF, uniquely determined by the $2$ constraints provided in Eqn. $4$. In summary, Sec. 3.2 suggests the incidence field is a monocular 3D prior, uniquely determined by the well-defined monocular 3D prior surface normal and depthmap. The fact implies that incidence field learning can generalize to different scenes in a similar way to surface normal and monocular depthmap. \n\n- Third, we make up the Sec. $3.2$ baseline performance in the Tab. A of the one-page rebuttal pdf. From Tab. A, the Sec. 3.2 minimal solver gives poor calibration results. This rationalizes the proposal of the incidence field for camera calibration. Thanks for suggesting this!\n\n**R2, Q2 - Part2: The proposed minimal solver is naive.**\n\n- First,  following **R2, Q3**, using the RANSAC algorithm to recover the intrinsics is just a part of our contribution. \n\n- Second, Our incidence field-based solution is neat, yet a benefit.\n\tFrom **R2, Q3**, our incidence field-based solution is neat as it avoids extrinsic estimation. Further, our neat solution facilitates rapid implementation and strong performance. Together with our efforts in creating a dataset and defining evaluation metrics, our work serves as a benchmark for other in-the-wild monocular image calibration works. Please also see the comments from the **R4**, who acknowledges the benefit of a neat solution.  \n\t\nRegarding the Eqn.$17$ and $18$,  we will modify the subtitle in L$179$ to \"Enumerate w/ Assumption\". Thank you for pointing this out!"}, {"review_id": "NbaA8yiHnr", "reviewer": "Reviewer_PGSe", "summary": "This paper introduces incidence field, a per-pixel representation for single image camera intrinsics calibration. The incidence field is defined as a 2D vector, pointing to the principal point of the image, normalized by the focal length. It is invariant to cropping and resizing. The paper suggests using a neural network to regress the incidence field given an input image. Finally, RANSAC is used to get the 4DoF intrinsics (fx, fy, cx, cy). ", "questions": "See weakness.", "limitations": "In real application, whether to apply the assumption still waits for human input.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper connects the incidence fields with surface normals and depthmaps. Thus, making it a reasonable representation that can be predicted by a network such as NewCRF, which is originally designed for depth estimation. \n\nThe incidence field formulation is simple yet powerful. It has desirable properties such as invariant to arbitrary cropping and resizing, as shown in S3.3; and can be derived from surface normals and depth, as shown in S3.2. \n\nIncidence fields are independent of camera extrinsic, contrasting Perspective Fields [26], which requires camera roll and pitch for formulation. This feature enables incidence fields to be trained on extensive data where the camera is calibrated but the pose is unknown.\n\nThe paper demonstrates SOTA performance on various datasets, both qualitatively and quantitatively. \n", "weaknesses": "- The paper claims in its contribution that “Our method meaks no assumption for the to-be-calibrated image”, while in fact, it presumes no distortion in the image (L118). \n\n- Table 1 needs revision, as it inaccurately describes the baselines and the comparisons are not apple-to-apple. Contrary to the claims made, [24, 26] do not assume Manhattan data during training and can be trained on arbitrary scenes, such as natural scenes. Furthermore, it is unfair to state that your method requires no assumption on the training data when comparing it with others [24, 31, 26], since they also predict camera roll and pitch while your method does not. \n\n- There are missing details in Table 2. It is unclear what the unit for the error metric is, and which dataset [26] is trained on. If [26] is solely trained on GSV data, then it implies a zero-shot scenario for [26] on all the listed datasets in Table 2. If camera extrinsics are provided, you should be able to train [26] on the same training set as your method, ensuring a fair comparison.\n\n- Including a baseline that predicts intrinsics from depth and surface normal predictions (as described in Sec 3.2) would be beneficial. This would empirically validate the derivation and give an idea of the method’s accuracy in Sec 3.2. Furthermore, it would support the argument in L150 that “Minimal solver in Eq. (9) can lead to a poor solution”, therefore further justifying the contribution of the method in Sec 3.4. \n\n- Typo in L218 and L233: There is no [26] in Tab 4. It should be Tab 3 instead. \n\n- In Table 5 and Sec 4.4, the baseline method is not clarified.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Response to R3 PGSe**\n\nWe sincerely thank for the exceptional comments and for giving our paper an \"Accept.\" We are particularly grateful for recognizing the validity of parametrizing intrinsic as an incidence field. We next delve into your specific feedback.\n\n**R3, Q1 Undistorted Image Assumption.**\n\nGreat Catch! We apologize for the imprecise argument. We assume an undistorted image and will clarify our claim and update the abstract, introduction, related works, and Table $1$ in the revision.  \nHowever, we humbly suggest that the undistorted image assumption is commonly upheld in all compared baselines ([25][30][24][31][26]), particularly in line with the Manhattan World assumption [31, 32, 47, 63], where detecting image lines inherently relies on an undistorted image.\n\n**R3 Q2&3: Fair Comparison in Tabs. $1$ and $2$.**\n\nThere is a misunderstanding in baselines. Baseline [26] and other baselines [24, 30, 31] can **NOT** be trained using ground truth camera extrinsics. Instead, they assume a gravity-aligned camera during training. \nThe gravity direction is unknown in most public datasets, such as ScanNet.\nIn other words, the known extrinsic only establishes the camera pose relative to the first camera.\nIn contrast, the orientation of the first camera relative to gravity remains unknown.  \nNow, we address the concerns comprehensively:\n\n1. **Manhattan data assumption in training.**  \n\nThank you for bringing this to our attention!  \nThe Manhattan World assumption eventually gives the camera's roll, pitch, and focal length. We illustrate the process briefly below:\n\n\tManhattan World assumption\n\t↳ Parallaized Line Sets,\n\t↳ Two (or three) Vanishing Points,\n\t↳ Camera's roll, pitch, and focal length.\n\nRecent learning-based methods [24][31][26] directly acquire the groundtruth camera's roll, pitch, and focal length from panorama images. We term this as the 'Manhattan data assumption,' which ultimately obtains groundtruth derived from the Manhattan World assumption. It does not imply the presence of horizontal and vertical lines in the images. We will further clarify this point with an additional description in our Tab. $1$ and Sec. $2$.\n\n2. **Unfair comparison to [24, 31, 26] in Tab. $3$.**\n\nGreat point! We view Tab. 3 as a fair comparison since our reformulation eliminates the need for perspective calibration. Following the Manhattan World illustration above, traditional baseline methods [31, 32, 47, 63] require simultaneous perspective calibration and intrinsic calibration. This also applies to the SoTA learning-based methods. For instance, [26] necessitates line detection before camera intrinsic estimation, while [31] estimates camera intrinsic using an inferred camera perspective field. In short, perspective calibration is indispensable for camera calibration in baselines.\n\nNext, we extend our method and conduct experiments to jointly apply perspective and intrinsic calibration to address your concern in Tab. $B$ of the one-page rebuttal pdf.  We modify our network to jointly regress the incidence and perspective fields (defined in [26]).\n\nInterestingly, with the incidence field estimated, we can directly estimate the camera roll and pitch from the perspective field. The incidence vector $\\mathbf{v}$ and perspective up vector $\\mathbf{u}\\_x$ (Eqn. $1$ of [26]) determine a 3D plane where the gravity vector (cross-focal point) resides. The intersection of two such planes in 3D space gives the gravity direction. We solve gravity using this as a minimal solver with a RANSAC algorithm. Finally, we determine the horizon line by averaging over $\\varphi_{\\mathbf{x}}$ (Eqn. $2$). Tab. B shows that our method also outperforms the SoTA methods in perspective calibration.\n\t\n3. **Unfair comparison to [26] in Tab. $2$.**\n\t\nWe now compare with the most recent checkpoint (released after the NeurIPS submission deadline) in Tab. $A$ in the one-page PDF.  \nThe newer released checkpoint [26] was trained on bigger 360Cities and EDINA datasets. However, we cannot train our model on 360Cities since the dataset is not public. From Tab. $A$, our method maintains SoTA performance with a solid margin over [26].\n\n**R3, Q4 Sec. 3.2 baseline.**\n\nThank you for your valuable input! In the one-page rebuttal PDF Tab. $A$, we incorporate a baseline with Sec. 3.2 minimal solver. \nWe utilize ZoeDepth and [Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation, ICCV, 2021] to initialize monocular depthmap and surface normal. These experiments further justify our idea of parametrizing intrinsic as the incidence field.\n\n**R3, Q5 Typos.**\n\nWe appreciate for pointing out the typos! We have corrected them in the revised version.\n\n**R3, Q6 Baselines in Tab. $5$ and Sec. $4.4$.**\n\nApologies for the oversight.  \nThe baselines in Tab. $5$ adopt an identical network structure, with only the last layer modified to a fully connected layer after average pooling for direct regression of the $4$ degrees of freedom (DoF) intrinsic. Additionally, all these models normalize the intrinsics by resizing the image height and width to [0, 1], which aids model convergence. We compare the baseline and our method’s estimated intrinsics using the same evaluation protocol. \n\nThe baselines in Tab. $6$ of Sec. $4.4$ follow LoFTR in estimating image correspondence. With the estimated correspondences, they apply an OpenCV-based five-point algorithm with RANSAC to estimate the two-view camera pose.\n"}, {"review_id": "9R6SxniPdy", "reviewer": "Reviewer_Ujoz", "summary": "This paper proposes a method for single-image camera calibration using a 3D prior the authors refer to as an incidence field. The incidence field is the collection of rays originating from some 3D point towards the camera origin, that are incidental to the image plane. The authors describe how the incidence field can be used to recover the camera intrinsics. Next, a method is proposed for recovering the intrinsics by first estimating an incidence field from an image (using a recently introduced neural network architecture, NewCRFs) and then applying RANSAC. An extensive evaluation shows that the proposed method achieves superior performance to recent methods on numerous benchmarks.  Additionally, several example applications are demonstrated.", "questions": "My initial rating is a weak accept. The proposed approach is simple, explained well, has novelty, and the results are compelling. I think this will be of interest to the community.\n\nSuggestions:\n\n- Full editing pass to address minor language issues. Improve aesthetics of figures (e.g., Figure 2, left).\n\n- Implementation details should be in the main document instead of supplemental.\n\n- L233 should reference Table 3. Text should introduce the dataset.", "limitations": "Limitations minorly touched on in the manuscript.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Addresses an important and interesting problem. Single image camera calibration \"in the wild\" has received a lot of attention lately and is a fundamental vision task. Knowledge of camera intrinsics is essential for numerous applications.\n\n- The proposed approach has technical novelty. In particular, shows how the concept of an incidence field (backprojected rays in 3D space) are related to camera intrinsics, then introduces a method to estimate them (method of [62] plus RANSAC). As far as I know, this has not been done before. Often in this space, a novel take on how to represent something leads to downstream performance benefits.\n\n- The method is extremely simple, which I consider a positive. \n\n- The evaluation is extensive, achieving superior results to recent methods on several benchmarks.\n\n- Numerous interesting applications presented, such as detecting image resizing and cropping.\n\n- Code provided and will be released with data and models.", "weaknesses": "- The evaluation should have descriptions of the metrics and how they are calculated (found in supplemental).\n\n- The quality of the writing is poor in certain aspects. For example, L14 in the abstract: \"With the estimated incidence field, a robust RANSAC algorithm recovers intrinsic.\" Ultimately, the manuscript just needs a solid editing pass top-to-bottom.\n\n- The quality of the figures, though parseable, could be improved from an aesthetic sense.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Response to R4 Ujoz**\n  \nWe appreciate the reviewer's suggestions to make our paper accessible to a wider audience.  We extend special gratitude to the reviewer for recognizing the technical novelty of our approach and its potential contribution to the community. We will adhere to the reviewer's suggestions in the revision.\n\n**R4, Q1 Missing metrics in tables.**  \n  \nThanks for the suggestion! We will move the evaluation metrics and implementation details from the supplementary to the main paper.  \n  \n**R4, Q2 Improve the quality of writing.**  \n  \nAlthough we have partially revised our manuscript based on the reviewer's feedback, we will do a solid editing pass over the paper. \n We include a list of revisions made so far. We will continue improving the manuscript.  \n  \n- L$14$: ''With the estimated incidence field, a robust RANSAC algorithm recovers intrinsic.'' -> \"We apply a robust RANSAC algorithm to recover intrinsic from the estimated incidence field.\"  \n- Fig.$2$c, \"w/ Assum.\" to \"w/o Assum.\", and change the other one accordingly.  \n- Tab.$3$ \"erceptual\" to \"perceptual\"  \n- L$233$, Tab.$4$ reference to Tab.$3$ reference.\n- Additional description of Tab. $5$ and Tab. $6$ baselines.\n  \n**R4, Q3 Quality of Fig. 2.**  \n  \nWe re-plotted Fig. $2$, improving its layout and removing the ``L\" shape illustration boxes. We do not place this in the one-page rebuttal pdf because of space constraints.  \n  \n**R4, Q4 L233 should reference Table 3. Text should introduce the dataset.**  \n\nWe fixed these issues in our manuscript.  \n"}, {"review_id": "pFuynJg95O", "reviewer": "Reviewer_iokN", "summary": "The submission #3482, entitled \"Tame a Wild Camera: In-the-Wild Monocular Camera Calibration\" presents a novel self-calibration strategy where the regression of the incidence field of the camera is predicted via a deep neural network before using a RANSAC to filter outliers and regress the intrinsic parameters of the camera. ", "questions": "- 1. One of my main grips on this paper is the claim for novelty. While the reviewer should admit that using the monocular normal and the depth prediction to estimate the camera's intrinsic is new. The \"incidence field\" regression is not novel as it has already been described in \"Deep Single Image Camera Calibration with Radial Distortion, CVPR, 2019\". In this paper, the authors obtained mixed results using such a strategy. However, one of the significant advantages is to be able to deal with optical aberration, which can be challenging to model parametrically. Unfortunately, this manuscript does not integrate camera distortion, which would be meaningful in this context. I believe that introducing this non-parametric approach, is not fully justified, maybe by training [24] on the exact same cropped data would lead to similar results? (I have not seen its detail in the experiment part). \n- 2. In this regard, it would also be beneficial to integrate \"DeepCalib: A Deep Learning Approach for Automatic Intrinsic Calibration of Wide Field-of-View Cameras, CVMP 2018\" in the literature as it deals with distortion and also explicitly tested on cropped images (maybe in supplementary). In their case, the estimation on cropped images fails because they have yet to train on these particular cases. So it may support your narrative. Another missing paper in the literature is the recent T-PAMI \"A Perceptual Measure for Deep Single Image Camera and Lens Calibration\" which might contain some interest in recent developments on deep learning-based calibration of cameras from a single image.\n- 3. Would it be possible to also test on non-cropped images too? I feel it is unfair to test with many cropped images as, in reality, most images in the wild are uncropped\n- 4. Since the approach has been trained on perspective images only, what would happen in case of a large field of view camera with a field of view outside of the training set distribution? In this regard it would be good to have some statistics regarding the distribution of the field on view used for the training.\n- 5. To really work in the wild, integrating radial distortion seems necessary, how would you integrate it in your framework for training and inference?", "limitations": "I have already expressed all my concerns regarding this work in the previous sections of this review. For all the reasons above mentioned, I would like to issue a rather mixed opinion regarding the acceptance of this work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Without being impeccable, the paper conveys the main idea behind the proposed technique.\n- Combining the incidence field with more traditional methods to regress the camera intrinsic is \"relatively\" novel.\n- The approach is supposed to work even when the images are cropped", "weaknesses": "- The literature is quite incomplete\n- The camera distortion is not considered\n- The approach is not really novel, as incident field regression has already been done in the past", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Response to R5 ioKN**\n\nWe sincerely thank R5 for the detailed comments. We deeply appreciate your feedback and aim to address your concerns thoroughly.\n\nBefore responding to the concerns, we believe that there could be a possibility of misunderstanding in our approach:  \nDifferent from [24, 26, 30, 31], [Deep Single], and [DeepCalib], our approach does not need perspective images in training. As in the main paper Tab. $1$, our approach avoids camera perspective calibration for intrinsic estimation. Except for Tab. $3$, all experiments use non-perspective images in training.  \n  \nWe next delve into concerns in detail.  \n  \n**R5, Q1 Novelty of Incidence Field.**  \n  \nThanks for bringing up the related work [Deep Single]! There exist several differences.  We summarize in Tab. D.\n\nTable D: Comparison of methods\n\nMethod | Intrinsic DoF | Persp. Images Reqd | Network Outputs |  \n-------|----| ------------ | ----|  \nDeep Single | 1 | Yes | $4 \\times 1$ Vector|  \nOurs | 4 | No | $H \\times W  \\times 3$ Incidence Field|\n\nWe elaborate on the differences further:\n  \n1. On Intrinsic Parameterization: \n\n\t[Deep Single] does not parameterize intrinsic as the incidence field. Instead, they parameterize the intrinsic as $1$ DoF camera vertical FoV (see [Deep Single] Fig. $2$).\n\t\n2. On Network Inference:\n\n\t [Deep Single] regresses $1 \\times 4$ vector. We regress the $H \\times W \\times 3$ pixel-wise incidence field. Next, we use a RANSAC algorithm to retrieve the $4$ DoF intrinsic. \n\n3. On Training Data: \n\n\t[Deep Single] trains with perspective images. Our method trains with perspective and non-perspective images.\n\n4. Potential Confusion caused by [Deep Single] Bearing Loss:\n\n  The confusion possibly arises at the Bearing Loss in [Deep Single], depicted in Fig. 5. Our incidence field learning loss Eqn. 13 shares some similarities with the Bearing Loss, with the distinction that the latter requires camera perspective angles in computation. We acknowledge the similarity.\n  However, incidence field learning itself takes a brief paragraph in our work. Our novelty is more on parametrizing intrinsics as incidence field, and the accompanied RANSAC algorithm. The similarity does not impact our novelty.\n\n5. Train [24] on cropped images.\n\n\t[24] parametrizes intrinsic as $1$ DoF vertical FoV. This limits [24] training on cropped images since cropped images have $4$ DoF camera intrinsic. \n  \n**R5 Q2&5: Integration of Image Undistortion.** \n  \nWe appreciate this exciting suggestion of discussing image undistortion works.  We will update our reference to include the suggested three pieces of literature.\n  \nWe omit image undistortion within the scope of our paper based on three reasons:  \n  \n1. Based on our observation, most public datasets and images on the internet are undistorted. Thus, most monocular camera calibration works, including all compared baselines ([25][30][24][31][26]), assume undistorted images as input. \n2. Excellent solutions in learning-based image undistortion already exist. e.g., [Blind Geometric Distortion Correction on Images Through Deep Learning, CVPR, 2019]. \n3. In our setting, the image undistortion becomes an open-classification problem, where whether an image is distorted needs to be verified first. However, we consider this a non-trivial problem requiring another elaborate algorithm to solve and is currently beyond the scope of this paper.  \n\n**Integrating Image Undistortion**\nWe do agree that integrating the image undistortion enhances the generalizability of our model. \nHence, we  discuss the following potential solutions for a distorted image:\n\n1. Synthesize image distortion in the training dataset.  \n  \n2. Estimate image distortion as a coordinate remapping similar to [Blind], together with the incidence field. This suggests estimating a pixel-wise vector:  \n$[v_1, v_2, v_3, \\Delta x, \\Delta y]$, where $[v_1, v_2, v_3]$ is the incidence field, and $[ \\Delta x, \\Delta y]$ is the distortion flow following [Blind].  \n  \n3. Recover image distortion using Hough Transform following [Blind].\n  \n4. Recover the camera intrinsic using the updated coordinates with distortion corrected.  \n  \n**R5, Q3 Test on uncropped images.**  \n  \nWe test primarily on uncropped images:\n  \n- **All** test images in Tab. $3$ are uncropped.\n  \n- **Lower** half of Tab. $2$ tests real uncropped images from datasets such as MegaDepth, Waymo, RGBD, ScanNet, MVS, and SceneNet.  \n  \n**R5, Q4 Statistics and performance of camera FoV.**\n  \n**FoV Statistics** We report our training/testing data FoV statistics Fig.$3$ of the Supp material.\nThe figure shows that our training data primarily includes $30-80$ degrees horizontally and $15-60$ degrees vertically.\nFurther, our training and testing data does contain some particularly large (over $60$ degrees) or small FoV (below $20$ degrees) images.  Most of our training data is not perspective images.\n\n**Performance Statistics** We thank you for suggesting this insightful analysis!\nTherefore, we recompute Tab.$2$ results by varying the horizontal FoV ranges to obtain Tab. E.\nWe report the intrinsics performance without applying assumptions and by averaging over all datasets in each range.\n\nTable E: Intrinsics estimation results with FoV variation.\n\n|     FoV     | 10 - 20 | 20 - 30 | 30 - 40 | 40 - 50 | 50 - 60 | 60 - 70 | 70 - 80 | 80 - 90 |   All-Range  |\n|:-----------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:------:|\n|    Number   |   163   |   726   |   1927  |   2747  |   2554  |   404   |   151   |    21   |  8708  |\n| Percentile  |   1.9%  |   8.3%  |  22.1%  |  31.5%  |  29.3%  |   4.6%  |   1.7%  |   0.2%  | 100.0% |\n|    $e_f$    |  0.173  |  0.119  |  0.113  |  0.120  |  0.128  |  0.109  |  0.120  |  0.112  |  0.122 |\n\nTab. E suggests that our algorithm maintains robustness across various FoVs, with the exception of exceptionally small ones. This outcome is anticipated, as small FoVs inherently offer a highly limited camera view."}]

Author rebuttal:
**To all Reviewers:**
We value the reviewers' recognition of our method's novelty (R1, R3, R4, R5) and its strong performance (R1, R3, R4). We also thank reviewers R1 and R5 for recognizing the lucidity of our paper's explanations.

We present a method for calibrating $4$ DoF intrinsic of in-the-wild monocular images. Unlike other works, our method can be trained over non-perspective images. Beyond calibration, we showcase compelling downstream applications, including the detection of image cropping and uncalibrated two-view pose estimation.

**Convention:**
We refer to main and supplementary figures/tables/references as numbers (Fig. 1, Tab. 1, [1]). In contrast, we refer to rebuttal figures/tables/references as alphabets (Fig. A, Tab. A, [A]).

**One-Page PDF:**
We've appended extra figures and tables to the one-page PDF.

- Fig. $A$: 
We demonstrate an interesting downstream application by enhancing the in-the-wild monocular 3D object detection work [Omni3D, CVPR'23] using our inferred intrinsic parameters.

- Tab. $A$: 
	We additionally benchmark the baseline [26] after being trained with substantially more data. This checkpoint is released after the NeurIPS'23 submission deadline. Also, we provide the baseline results which employ the minimal solver outlined in Section $3.2$.

- Tab. $B$:
We provide interesting joint perspective-intrinsic calibration results. The method is elaborated in response to R3, Q $2\& 3$, point $2$.


Meta-review: This paper introduces an intriguing approach for monocular 4DoF intrinsic camera parameters. Reviewers favor the proposed method and the demonstrated results. The authors provided comprehensive feedback regarding specific questions and additional demonstration requests. The authors also provided a supplement pdf file that shows interesting downstream applications. There were concerns from reviewer wjnN, but AC notes that the authors adequately defend it. After the rebuttal phase, most reviewers voted for the positive scores. AC recommends paper acceptance. It is strongly advised to apply the valuable discussions and additional materials in the revision.
Decision: Accept (poster)