SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since the network inference executes on GPU device, we adopt a GPU-end RANSAC algorithm to recover the intrinsic  $\mathbf{K}$  from the incidence field  $\mathbf{V}$ . Unlike a CPU-based RANSAC, we perform fixed  $K_{r}$  iterations of RANSAC without termination. In RANSAC, we use the minimal solver to generate  $K_{c}$  candidates and select the optimal one that maximizes a scoring function (see Fig. 2).
RANSAC w.o Assumption. From Eq. (2), the incidence vector  $\mathbf{v}$  relates to the intrinsic  $\mathbf{K}$  as:
$$
\mathbf {v} = \mathbf {K} ^ {- 1} \mathbf {x} = \left[ \begin{array}{l l l} \frac {x - b _ {x}}{f _ {x}} & \frac {y - b _ {y}}{f _ {y}} & 1 \end{array} \right] ^ {\intercal}. \tag {15}
$$
From Eq. (15), a minimal solver for intrinsic is straightforward. In the incidence field, randomly sample two incidence vectors  $\mathbf{v}^1 = \left[v_x^1 v_y^1 1\right]^{\intercal}$  and  $\mathbf{v}^2 = \left[v_x^2 v_y^2 1\right]^{\intercal}$ . The intrinsic is:
$$
\left\{ \begin{array}{l} f _ {x} = \frac {x ^ {1} - x ^ {2}}{v _ {x} ^ {1} - v _ {x} ^ {2}} \\ b _ {x} = \frac {1}{2} \left(x ^ {1} - v _ {x} ^ {1} f _ {x} + x ^ {2} - v _ {x} ^ {2} f _ {x}\right) \end{array} , \quad \left\{ \begin{array}{l} f _ {y} = \frac {y ^ {1} - y ^ {2}}{v _ {y} ^ {1} - v _ {y} ^ {2}} \\ b _ {y} = \frac {1}{2} \left(x ^ {1} - v _ {y} ^ {1} f _ {y} + x ^ {2} - v _ {y} ^ {2} f _ {y}\right) \end{array} . \right. \right. \tag {16}
$$
Similarly, the scoring function is defined in  $x$ -axis and  $y$ -axis, respectively:
$$
\rho_ {x} \left(f _ {x}, b _ {x}, \{\mathbf {x} \}, \{\mathbf {v} \}\right) = \sum_ {i = 1} ^ {N _ {k}} \left(\| \frac {x ^ {i} - b _ {x}}{f _ {x}} - v _ {x} ^ {i} \| <   k _ {x}\right), \quad \rho_ {y} \left(f _ {y}, b _ {y}, \{\mathbf {x} \}, \{\mathbf {v} \}\right) = \sum_ {i = 1} ^ {N _ {k}} \left(\| \frac {y ^ {i} - b _ {y}}{f _ {y}} - v _ {y} ^ {i} \| <   k _ {y}\right), \tag {17}
$$
Table 2: In-the-Wild Monocular Camera Calibration. We benchmark in-the-wild monocular camera calibration. We use the first 9 datasets for training, and test on all 14 datasets. Except for MegaDepth, we synthesize novel intrinsic by cropping and resizing during training. Note the synthesized images violate the focal point and focal length assumption. [Key: ZS = Zero-Shot, Asm. = Assumptions, Syn. = Synthesized]
Dataset	Calibration	Scene	ZS	Syn.	Perspective [30]	Ours	Ours + Asm.
ef	eb	ef	eb	ef	eb
NuScenes [10]	Calibrated	Driving	X	✓	0.378	0.286	0.102	0.087	0.402	0.400
KITTI [22]	Calibrated	Driving	X	✓	0.631	0.279	0.111	0.078	0.383	0.368
Cityscapes [14]	Calibrated	Driving	X	✓	0.624	0.316	0.108	0.110	0.387	0.367
NYUv2 [53]	Calibrated	Indoor	X	✓	0.261	0.348	0.086	0.174	0.376	0.379
ARKitScenes [8]	Calibrated	Indoor	X	✓	0.325	0.367	0.140	0.243	0.400	0.377
SUN3D [65]	Calibrated	Indoor	X	✓	0.260	0.385	0.113	0.205	0.389	0.383
MVIImgNet [68]	SfM	Object	X	✓	0.838	0.272	0.101	0.081	0.108	0.072
Objectron [1]	SfM	Object	X	✓	0.601	0.311	0.078	0.070	0.088	0.079
MegaDepth [41]	SfM	Outdoor	X	X	0.319	0.000	0.137	0.046	0.109	0.000
Waymo [58]	Calibrated	Driving	✓	X	0.444	0.020	0.210	0.053	0.157	0.020
RGBD [55]	Pre-defined	Indoor	✓	X	0.166	0.000	0.097	0.039	0.067	0.000
ScanNet [16]	Calibrated	Indoor	✓	X	0.189	0.010	0.128	0.041	0.109	0.010
MVS [21]	Pre-defined	Indoor	✓	X	0.185	0.000	0.170	0.028	0.127	0.000
Scenes11 [11]	Pre-defined	Synthetic	✓	X	0.211	0.000	0.170	0.044	0.117	0.000
Table 3: Comparisons to Monocular Camera Calibration with Geometry on GSV dataset [4]. We follow the training and testing protocol of [38]. For a fair comparison, we convert the estimated intrinsic to camera FoV on the  $y$ -axis direction, following [30, 38], and report our results  $w/$  and  $w/o$  the assumptions.
FoV (°)	Upright [37] PAMI'13	Perceptual [28] CVPR'18	CTRL-C [38] ICCV'21	Perspective [30] CVPR'23	Ours
w/o Asm.	w/ Asm.
Mean	9.47	4.37	3.59	3.07	2.49	2.47
Median	4.42	3.58	2.72	2.33	1.96	1.92
where  $N_{k}$  and  $k_{x} / k_{y}$  are the number of sampled pixels and the threshold for axis-  $x / y$  directions.
RANSAC  $w$ / Assumption. If a simple camera model is assumed, i.e., intrinsic only has an unknown focal length, it only needs to estimate 1-DoF intrinsic. We enumerate the focal length candidates as:
$$
\{f \} = \left\{f _ {\min } + \frac {i}{N _ {f}} \left(f _ {\max } - f _ {\min }\right) \mid 0 \leq i \leq N _ {f} \right\}. \tag {18}
$$
The scoring function under the scenario is defined as the summation over  $x$ -axis and  $y$ -axis:
$$
\rho (f, \{\mathbf {x} \}, \{\mathbf {v} \}) = \rho_ {x} \left(f _ {x}, w / 2, \{\mathbf {x} \}, \{\mathbf {v} \}\right) + \rho_ {y} \left(f _ {y}, h / 2, \{\mathbf {x} \}, \{\mathbf {v} \}\right). \tag {19}
$$
3.5 Downstream Applications
Image Crop & Resize Detection and Restoration. Eq. (10) defines a crop and resize operation:
$$
\mathbf {x} ^ {\prime} = \Delta \mathbf {K} \mathbf {x}, \quad \mathbf {K} ^ {\prime} = \Delta \mathbf {K} \mathbf {K}, \quad \mathbf {I} ^ {\prime} (\mathbf {x} ^ {\prime}) = \mathbf {I} ^ {\prime} (\Delta \mathbf {K} \mathbf {x}) = \mathbf {I} (\mathbf {x}). \tag {20}
$$
When a modified image  $\mathbf{I}'$  is presented, our algorithm calibrates its intrinsic  $\mathbf{K}'$  and then:
Case 1: The original intrinsic  $\mathbf{K}$  is known. E.g., we obtain  $\mathbf{K}$  from the image-associated EXIF file [3]. Image manipulation is computed as  $\Delta \mathbf{K} = \mathbf{K}'\mathbf{K}^{-1}$ . A manipulation is detected if  $\Delta \mathbf{K}$  deviates from an identity matrix. The original image restores as  $\mathbf{I}(\mathbf{x}) = \mathbf{I}'(\Delta \mathbf{K}\mathbf{x})$ . Interestingly, the four corners of image  $\mathbf{I}'$  are mapped to a bounding box in original image  $\mathbf{I}$  under manipulation  $\Delta \mathbf{K}$ . We thus quantify the restoration by measuring the bounding box. See Fig. 4.
Case 2: The original intrinsic  $\mathbf{K}$  is unknown. We assume the genuine image possesses an identical focal length and central focal point. Any resizing and cropping are detected when matrix  $\mathbf{K}'$  breaks this assumption. Note, the rule cannot detect aspect ratio preserving resize or centered crop. We restore the original image by defining an inverse operation  $\Delta \mathbf{K}$  to an intrinsic fits the assumption.
3D Sensing Related Tasks. Intrinsic estimation can enable multiple applications. E.g., depthmap to point cloud, uncalibrated two-view pose estimation, etc.
Implementation Details Our network is trained using the Adam optimizer [32] with a batch size of 8. The learning rate is  $1e^{-5}$ , and the training process runs for 20,000 steps. Our RANSAC algorithm
Table 4: Comparisons to Monocular Camera Calibration with Object. We compare to the recent Face-Calib [29], which calibrates the camera using video containing human faces. We report our results  $w/$  and  $w/o$  assuming a simple camera model. We perform zero-shot testing using the same model benchmarked in Tab. 2.
Methods	BIWIRGBD-ID [48]	CAD-120 [59]
ef	efx	efy	eb	ebx	eby	ef	efx	efy	eb	ebx	eby
Louraki [45]	0.662	0.662	0.662	-	0.387	0.222	0.732	0.732	0.732	-	0.255	0.180
Fetzer [20] WACV'20	0.845	0.845	0.845	-	0.001	0.005	0.679	0.679	0.679	-	0.001	0.005
BPnP [12] CVPR'19	0.675	0.675	0.675	-	0.322	0.479	1.178	1.178	1.178	-	0.103	0.129
FaceCalib [29] FG'23	0.133	0.133	0.133	-	0.026	0.042	0.151	0.151	0.151	-	0.023	0.063
Ours	0.034	0.029	0.016	0.020	0.011	0.018	0.137	0.137	0.054	0.042	0.042	0.008
Ours + Assumptions	0.019	0.019	0.019	0.000	0.000	0.000	0.047	0.047	0.047	0.000	0.000	0.000
is executed on a GPU, performing  $K_{r} = 2,048$  iterations, with each iteration including a random sample of  $K_{c} = 20,000$  incidence vectors. During both training and testing, the image is resized to a resolution of  $480 \times 640$ . The threshold for determining RANSAC inliers is  $k_{x} = k_{y} = 0.02$ .
Evaluation Metrics We follow [29] in assessing camera intrinsic estimation using the metrics:
$$
e _ {f _ {x}} = \frac {\left\| f _ {x} ^ {\prime} - f _ {x} \right\|}{f _ {x}}, e _ {f _ {y}} = \frac {\left\| f _ {y} ^ {\prime} - f _ {y} \right\|}{f _ {y}}, e _ {b _ {x}} = 2 \cdot \frac {\left\| b _ {x} ^ {\prime} - b _ {x} \right\|}{w}, e _ {b _ {y}} = 2 \cdot \frac {\left\| b _ {y} ^ {\prime} - b _ {y} \right\|}{h}. \tag {21}
$$
We additionally consolidate the performance along the x-axis and y-axis into comprehensive measurements,  $e_{f} = \max (e_{f_{x}}, e_{f_{y}})$  and  $e_{b} = \max (e_{b_{x}}, e_{b_{y}})$ .
4.1 Monocular Camera Calibration In-The-Wild
Datasets. Our method is trained whenever a calibrated intrinsic is provided, making it applicable to a wide range of publicly available datasets. In Tab. 2, we incorporate datasets of different application scenarios, including indoor, outdoor scenes, driving, and object-centric scenes. Many of the datasets utilize only a single type of camera for data collection, resulting in a scarcity of intrinsic variations. Similar to [38], we employ random resizing and cropping to synthesize more intrinsic, marked in Tab. 2 column "Syn". In augmentation, we first resize all images to a resolution of  $480 \times 640$ . We then uniformly random resize up to two times its size and subsequently crop to a resolution of  $480 \times 640$ . As MegaDepth [41] collects images captured by various cameras from the Internet, we disable its augmentation. We document the intrinsic parameters of each dataset in Supp.
In Tab. 2 column "Calibration", we assess intrinsic quality into various levels. "Calibrated" suggests accurate calibration with a checkerboard. "Pre-defined" is less accurate, indicating the default intrinsic provided by the camera manufacturer without a calibration process. "SfM" signifies that the intrinsic is computed via an SfM method [52].
In-The-Wild Monocular Camera Calibration. We benchmark in-the-wild monocular calibration performance on Tab. 2. For trained datasets, except for MegaDepth, we test on synthetic data using random cropping and resizing. For the unseen test dataset, we refrain from applying any augmentation to better mimic real-world application scenarios.
We compare to the recent baseline [30], which regresses intrinsic via a deep network. Note, [30] can not train on arbitrary calibrated images as requiring panorama images in training. A fair comparison using the same training and testing images is in Tab. 4 and Sec. 4.2. [30] provides models with two variations: one assumes a central focal point, and another does not. We report with the former model whenever the input image fits the assumption. From Tab. 2, our method demonstrates superior generalization across multiple unseen datasets. Further, the result  $w/$  assumption outperforms  $w/o$  assumption whenever the input images fit the assumption. Tested on an RTX-2080 Ti GPU, the combined network inference and calibration algorithm runs on average in 87 ms per image.
4.2 Monocular Camera Calibration with Geometry
Methods in this line of research hold a Manhattan World assumption, positing that images consist of planes that are either parallel or perpendicular to each other. Stated in Tab. 1, baselines [28,30,38] relax the assumption to training data. Our method imposes no other assumption except for undistorted images in both training and testing.
This brings three benefits. First, the assumption restricts their training to panorama images. In contrast, our model is trainable with any calibrated images. This yields improved generalization,
(a) Input & Incidence
(b) Est. Restoration
(c) GT. Restoration
Figure 4: Image Crop & Resize Detection and Restoration. Image editing, including cropping and resizing changes intrinsic. As in Sec. 3.5, monocular calibration is applicable to detect and restore image manipulations. We visualize the zero-shot samples on ScanNet and Waymo. More examples are in Supp.
(d) Original Image
Table 5: Image Crop and Resize Restoration. Stated in Sec. 3.5, our method also encompasses the restoration of image manipulations. Use model reported in Tab. 2, we conduct evaluations on both seen and unseen datasets.
Methods	KITTI [22]	NYUv2 [53]	ARKitScenes [8]	Waymo [58]	RGBD [55]	ScanNet [16]	MVS [21]
mIOU	Acc	mIOU	Acc	mIOU	Acc	mIOU	Acc	mIOU	Acc	mIOU	Acc	mIOU	Acc
Baseline	0.686	0.795	0.621	0.710	0.586	0.519	0.581	0.721	0.636	0.681	0.597	0.811	0.595	0.667
Ours	0.842	0.852	0.779	0.856	0.691	0.837	0.681	0.796	0.693	0.781	0.709	0.887	0.638	0.795
as shown in Tab. 2. Second, it constrains the baselines to a simple camera parameterized by FoV. We consider the proposed incidence field a more generalizable and invariant parameterization of intrinsic.  $E.g.$ , while FoV remains invariant to image resizing, it still changes after cropping. However, the incidence field is unaffected in both cases. In Tab. 3, the substantial improvement we achieved  $(0.60 = 3.07 - 2.47)$  over the recent SoTA [30] empirically supports our argument. Third, our method calibrates the 4 DoF intrinsic with a non-learning RANSAC algorithm. Baselines instead regress the intrinsic. This renders our method more robust and interpretable. In Fig. 2 (b), the estimated intrinsic quality is visually discerned through the consistency achieved between the two incidence fields.
4.3 Monocular Camera Calibration with Objects
We compare to the recent object-based camera calibration method FaceCalib [29]. The baseline employs a face alignment model to calibrate the intrinsic over a video. Both [29] and ours perform zero-shot prediction. We report performance using Tab. 2 model. Compared to [29], our method is more general as it does not assume a human face present in the image. Meanwhile, [29] calibrates over a video, while ours is a monocular method. For a fair comparison, we report the video-based results as an averaged error over the videos. We report results  $w/$  and  $w/o$  assuming a simple camera model. Since the tested image has a central focal point, when the assumption applied, the error of the focal point diminished. In Tab. 4, we outperform SoTA substantially.
4.4 Downstream Applications
Image Crop & Resize Detection and Restoration. In Sec. 3.5, our method also addresses the detection and restoration of geometric manipulations in images. Using the model reported in Tab. 2, we benchmark its performance in Tab. 5. Random manipulations following Sec. 4.1 contribute to  $50\%$  of both train and test sets, and the other  $50\%$  are genuine images. In Tab. 5, we evaluate restoration with mIOU and report detection accuracy (i.e. binary classification of genuine vs edited images). From the table, our method generalizes to the unseen dataset, achieving an averaged mIOU of 0.680. Meanwhile, we substantially outperform the baseline, which is trained to directly regresses the intrinsic. The improvement suggests the benefit of the incidence field as an invariant intrinsic parameterization. Beyond performance, our algorithm is interpretable. In Fig. 4, the perceived image geometry is interpretable for humans.
Figure 5: In-the-Wild Camera Calibration Benefits In-the-Wild 3D Object Detection [9]. Our estimated intrinsic improves 2D projection of estimated 3D bounding boxes [9], highlighted with red arrows.
Restored  $\mathbf{I}_1$
Cropped  $\mathbf{I}_1$
Original Image I
Cropped  $\mathbf{I}_2$
Figure 6: Broader Impact. Our method detects geometric manipulation and helps press image authenticity. In the figure, our method restores cropping and the aspect ratio. Orange arrows mark estimated incidence field.
Restored  $\mathbf{I}_2$
Table 6: Uncalibrated Two-View Camera Pose Estimation. We use the model reported in Tab. 2 and assume distinct camera models for both frames. During calibration, we apply the simple camera assumption. The last two rows ablate the performance using GT intrinsic and our estimated intrinsic.
Methods	Calibrated	ScanNet [16]	MegaDepth [41]
@5°	@10°	@20°	@5°	@10°	@20°
SuperGlue [50] CVPR'19	✓	16.2	33.8	51.8	42.2	61.2	75.9
DRC-Net [44] ICASSP'22	✓	7.7	17.9	30.5	27.0	42.9	58.3
LoFTR [57] CVPR'21	✓	22.0	40.8	57.6	52.8	69.2	81.2
ASpanFormer [13] ECCV'22	✓	25.6	46.0	63.3	55.3	71.5	83.1
PMatch [78] CVPR'23	✓	29.4	50.1	67.4	61.4	75.7	85.7
PMatch [78] CVPR'23	X	11.4	29.8	49.4	16.8	30.6	47.4
Uncalibrated Two-View Camera Pose Estimation. With correspondence between two images, one can infer the fundamental matrix [25]. However, the pose between two uncalibrated images is determined by a projective ambiguity. Our method eliminates the ambiguity with monocular camera calibration. In Tab. 6, we benchmark the uncalibrated two-view pose estimation and compare it to recent baselines. The result is reported using the model benchmarked in Tab. 2 and assumes unique intrinsic for both images. We perform zero-shot in ScanNet. For MegaDepth, it includes images collected over the Internet with diverse intrinsics. Interestingly, in ScanNet, our uncalibrated method outperforms a calibrated one [44]. In Supp, we plot the curve between pose performance and intrinsic quality. The challenging setting suggests itself an ideal task to evaluate the intrinsic quality.
In-the-Wild Monocular 3D Sensing. In-the-wild 3D sensing, e.g., monocular 3D object detection [9], requires intrinsic to connect the 3D and 2D space. In Fig. 5, despite 3D bounding boxes being accurately estimated, incorrect intrinsic produces suboptimal 2D projection. Our calibration results suggest monocular incidence field estimation is another essential monocular 3D sensing task.
We calibrate monocular images through a novel monocular 3D prior referred as incidence field. The incidence field is a pixel-wise parameterization of intrinsic invariant to image resizing and cropping. A RANSAC algorithm is developed to recover intrinsic from the incidence field. We extensively benchmark our algorithm and demonstrate robust in-the-wild performance. Beyond calibration, we show multiple downstream applications that benefit from our method.
Limitation. In real application, whether to apply the assumption still requires human input.
Broader Impacts. Our method detects image spacial editing. If the Exif files are provided, it further localizes the modification region. Shown in Fig. 6, it helps improve press image authenticity.
Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020. 3, 7, 9
[59] J Sung, Hema Koppula, Bart Selman, and Ashutosh Saxena. Cornell activity datasets: CAD-60 & CAD-120. 8
[60] Zachary Teed and Jia Deng. DeepV2D: Video to depth with differentiable structure from motion. In ICLR, 2020. 3
[61] Rafael Von Gioi, Jeremie Jakubowicz, Jean-Michel Morel, and Gregory Randall. LSD: A fast line segment detector with a false detection control. TPAMI, 2008. 3
[62] Horst Wildenauer and Allan Hanbury. Robust camera self-calibration from monocular images of manhattan worlds. In CVPR, 2012. 1, 3
[63] Scott Workman, Connor Greenwell, Menghua Zhai, Ryan Baltenberger, and Nathan Jacobs. Deepfocal: A method for direct focal length estimation. In ICIP, 2015. 3
[64] Scott Workman, Menghua Zhai, and Nathan Jacobs. Horizon lines in the wil. In BMVC, 2016. 1, 3
[65] Jianxiong Xiao, Andrew Owens, and Antonio Torralba. Sun3D: A database of big spaces reconstructed using SFM and object labels. In ICCV, 2013. 7
[66] Yiliang Xu, Sangmin Oh, and Anthony Hoogs. A minimum error vanishing point detection approach for uncalibrated monocular images of man-made environments. In CVPR, 2013. 1, 3
[67] Qichao Ying, Hang Zhou, Zhenxing Qian, Sheng Li, and Xinpeng Zhang. Robust image protection countering cropping manipulation. arXiv preprint arXiv:2206.02405, 2022. 3
[68] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, Guanying Chen, Shuguang Cui, and Xiaoguang Han. MVImgNet: A large-scale dataset of multi-view images. In CVPR, 2023. 7
[69] Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. NeW CRFs: Neural window fully-connected CRFs for monocular depth estimation. In CVPR, 2022. 6
[70] Menghua Zhai, Scott Workman, and Nathan Jacobs. Detecting vanishing points using global image context in a non-manhattan world. In CVPR, 2016. 3
[71] Hui Zhang, Wong Kwan-yee, and Guoqiang Zhang. Camera calibration from images of spheres. TPAMI, 2007. 3
[72] Yueqiang Zhang, Langming Zhou, Haibo Liu, and Yang Shang. A flexible online camera calibration using line segments. Journal of Sensors. 3
[73] Zhengyou Zhang. A flexible new technique for camera calibration. TPAMI, 2000. 3
[74] Zhengyou Zhang. Camera calibration with one-dimensional objects. TPAMI, 2004. 3
[75] Wang Zhao, Shaohui Liu, Yezhi Shu, and Yong-Jin Liu. Towards better generalization: Joint depth-pose learning without posenet. In CVPR, 2020. 3
[76] Shengjie Zhu, Garrick Brazil, and Xiaoming Liu. The edge of depth: Explicit constraints between segmentation and depth. In CVPR, 2020. 3
[77] Shengjie Zhu and Xiaoming Liu. LightedDepth: Video depth estimation in light of limited inference view angles. In CVPR, 2023. 3
[78] Shengjie Zhu and Xiaoming Liu. PMatch: Paired masked image modeling for dense geometric matching. In CVPR, 2023. 10