SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shengjie Zhu, Abhinav Kumar, Masa Hu, and Xiaoming Liu
Department of Computer Science and Engineering,
Michigan State University, East Lansing, MI, 48824
{zhusheng, kumarab6, huynshen} @msu.edu, liuxm@cse.msu.edu
3D sensing for monocular in-the-wild images, e.g., depth estimation and 3D object detection, has become increasingly important. However, the unknown intrinsic parameter hinders their development and deployment. Previous methods for the monocular camera calibration rely on specific 3D objects or strong geometry prior, such as using a checkerboard or imposing a Manhattan World assumption. This work instead calibrates intrinsic via exploiting the monocular 3D prior. Given an undistorted image as input, our method calibrates the complete 4 Degree-of-Freedom (DoF) intrinsic parameters. First, we show intrinsic is determined by the two well-studied monocular priors: monocular depthmap and surface normal map. However, this solution necessitates a low-bias and low-variance depth estimation. Alternatively, we introduce the incidence field, defined as the incidence rays between points in 3D space and pixels in the 2D imaging plane. We show that: 1) The incidence field is a pixel-wise parametrization of the intrinsic invariant to image cropping and resizing. 2) The incidence field is a learnable monocular 3D prior, determined pixel-wisely by up-to-sacle monocular depthmap and surface normal. With the estimated incidence field, a robust RANSAC algorithm recovers intrinsic. We show the effectiveness of our method through superior performance on synthetic and zero-shot testing datasets. Beyond calibration, we demonstrate downstream applications in image manipulation detection & restoration, uncalibrated two-view pose estimation, and 3D sensing. Codes/models are held here.
Camera calibration is typically the first step in numerous vision and robotics applications [27, 46] that involve 3D sensing. Classic methods enable accurate calibration by imaging a specific 3D structure such as a checkerboard [49]. With the rapid growth of monocular 3D vision, there is an increasing focus on 3D sensing from in-the-wild images, such as monocular depth estimation, 3D object detection [35, 36], and 3D reconstruction [42, 43]. While 3D sensing techniques over in-the-wild images are developed, camera calibration for such in-the-wild images continues to pose significant challenges.
Classic methods for monocular calibration use strong geometry prior, such as using a checkerboard. However, such 3D structures are not always available in in-the-wild images. As a solution, alternative methods relax the assumptions. For example, [29] and [23] calibrate using common objects such as human faces and objects' 3D bounding boxes. Another significant line of research [7, 19, 34, 37, 51, 62, 66] is based on the Manhattan World assumption [15], which posits that all planes within a scene are either parallel or perpendicular to each other. This assumption is further relaxed [28, 30, 64] to estimate the lines that are either parallel or perpendicular to the direction of gravity. The intrinsic parameters are recovered by determining the intersected vanishing points of detected lines, assuming a central focal point and an identical focal length.
Figure 1: In (a), our work focuses on monocular camera calibration for in-the-wild images. We recover the intrinsic from monocular 3D-prior. In (c) - (e), an estimated depthmap is converted to surface normal using a groundtruth and noisy intrinsic individually. Noisy intrinsic distorts the point cloud, consequently leading to inaccurate surface normal. In (e), the normal presents a different color to (c). Motivated by the observation, we develop a solver that utilizes the consistency between the two to recover the intrinsic. However, this solution exhibits numerical instability. We then propose to learn the incidence field as an alternative 3D monocular prior. The incidence field is the collection of the pixel-wise incidence ray, which originates from a 3D point, targets at a 2D pixel, and crosses the camera origin, as shown in (b). Similar to depthmap and normal, a noisy intrinsic leads to a noisy incidence field, as in (e). By same motivation, we develop neural network to learn in-the-wild incidence field and develop a RANSAC algorithm to recover intrinsic from the estimated incidence field.
While the assumptions are relaxed, they may still not hold true for in-the-wild images. This creates a contradiction: although we develop robust models to estimate in-the-wild monocular depthmap, generating its 3D point cloud remains infeasible due to the missing intrinsic. A similar challenge arises in monocular 3D object detection, as we face limitations in projecting the detected 3D bounding boxes onto the 2D image. In AR/VR applications, the absence of intrinsic precludes placing multiple reconstructed 3D objects within a canonical 3D space. The absence of a reliable, assumption-free monocular intrinsic calibrator has become a bottleneck in deploying these 3D sensing applications.
Our method is motivated by the consistency between the monocular depthmap and surface normal map. In Fig. 1 (c) - (e), an incorrect intrinsic distorts the back-projected 3D point cloud from the depthmap, resulting in distorted surface normals. Based on this, intrinsic is optimal when the estimated monocular depthmap aligns consistently with the surface normal. We present a solution to recover the complete 4 DoF intrinsic by leveraging the consistency between the surface normal and depthmap. However, the algorithm is numerically ill-conditioned as its computation depends on the accurate gradient of depthmap. This requires depthmap estimation with low bias and variance.
To resolve it, we propose an alternative approach by introducing a novel 3D monocular prior in complementation to the depthmap and surface normal map. We refer to this as the incidence field, which depicts the incidence ray between the observed 3D point and the projected 2D pixel on the imaging plane, as shown in Fig. 1 (b). The combination of the incidence field and the monocular depthmap describes a 3D point cloud. Compared to the original solution, the incidence field is a direct pixel-wise parameterization of the camera intrinsic. This implies that a minimal solver based on the incidence field only needs to have low bias. We then utilize a deep neural network to perform the incidence field estimation. A non-learning RANSAC algorithm is developed to recover the intrinsic parameters from the estimated incidence field.
We consider the incidence field is also a monocular 3D prior. Similar to depthmap and surface normal, the incidence field is invariant to the image cropping or resizing. This encourages its generalization over in-the-wild images. Theoretically, we show the incidence field is pixel-wisely determined by depthmap and normal map. Empirically, to support our argument, we combine multiple public datasets into a comprehensive dataset with diverse indoor, outdoor, and object-centric images captured by different imaging devices. We further boost the variety of intrinsic by resizing and cropping the images in a similar manner as [23]. Finally, we include zero-shot testing samples to benchmark real-world monocular camera calibration performance.
We showcase downstream applications that benefit from monocular camera calibration. In addition to the aforementioned 3D sensing tasks, we present two intriguing additional applications. One is detecting and restoring image resizing and cropping. When an image is cropped or resized, it disrupts the assumption of a central focal point and identical focal length. Using the estimated
Table 1: Camera Calibration Methods from Strong to Relaxed Assumptions. Non-learning methods [23, 29, 34, 37, 38, 51, 62, 71-74] rely on strong assumptions. Learning-based methods [28, 30, 38] relax the assumptions to using gravity-aligned panorama images during training. Our method makes no assumptions except for undistorted images. This enables training with any calibrated images and calibrates 4 DoF intrinsic.
[23,29,38,71-74]	[34,37,51,62]	[28,38]	[30]	Ours
Degree of Freedom (DoF) Assumption	4	1	1	3	4
Specific-Objects	Manhattan	Train with Panorama Images	Undistorted Images
intrinsic parameters, we restore the edited image by adjusting its intrinsic to a regularized form. The other application involves two-view uncalibrated camera pose estimation. With established image correspondence, a fundamental matrix [25] is determined. However, there does not exist an injective mapping between the fundamental matrix and camera pose [27]. This raises a counter-intuitive fact: inferring the pose from two uncalibrated images is infeasible. But our method enables uncalibrated two-view pose estimation by applying monocular camera calibration.
$\diamond$  Our approach tackles monocular camera calibration from a novel perspective by relying on monocular 3D priors. Our method makes no assumption for the to-be-calibrated undistorted image.
$\diamond$  Our algorithm provides robust monocular intrinsic estimation for in-the-wild images, accompanied by extensive benchmarking and comparisons against other baselines.
We demonstrate its benefits on diverse and novel downstream applications.
Monocular Camera Calibration with Geometry. One line of work [7, 19, 34, 37, 51, 62, 66] assumes the Manhattan World assumption [15], where all planes in 3D space are either parallel or perpendicular. Under the assumption, line segments in the image converge at the vanishing points, from which the intrinsic is recovered. LSD [61] and EDLine [2] develop robust line estimators. Others jointly estimate the horizon line and the vanishing points [40, 54, 70]. In Tab. 1, recent learning-based methods [28, 30, 38, 63, 64] relax the assumption by using the gravity-aligned panorama images during training. Such data provides the groundtruth vanishing point and horizon lines which are originally computed with the Manhattan World assumption. Still, the assumption constrains [28, 38, 63, 64] in modeling intrinsic as 1 DoF camera. Recently, [30] relaxes the assumption to 3 DoF via regressing the focal point. In comparison, our method makes no assumption except for undistorted images. This enables us to calibrate 4 DoF intrinsic and train with any calibrated images.
Monocular Camera Calibration with Object. Zhang's method [73] based on a checkerboard pattern is widely regarded as the standard for camera calibration. Several works generalize this method to other geometric patterns such as 1D objects [74], line segments [72], and spheres [71]. Recent works [29] and [23] extend camera calibration to real-world objects such as human faces. Optimizers, including BnPnP [12] and PnP [56] are developed. However, the usage of specific objects restricts their applications. In contrast, our approach is applicable to any image.
Image Cropping and Resizing. Detecting photometric image manipulation [5, 17, 24] is extensively researched. But few studies detecting geometric manipulation, e.g., resizing and cropping. On resizing, [18] regresses the image aspect ratio with a deep model. On cropping, a proactive method [67] is developed. We demonstrate that calibration also detects geometric manipulation. Our method does not need to encrypt images, complementing photometric manipulation detection.
Uncalibrated Two-View Pose Estimation. With the fundamental matrix estimated, the two-view camera pose is determined up to a projective ambiguity if images are uncalibrated. Alternative solutions [31, 47, 58] exist by employing deep networks to regress the pose. However, regression hinders the usage of geometric constraints, which proves crucial in calibrated two-view pose estimation [60, 75, 77]. Other works [20, 26] use more than two uncalibrated images for pose estimation. Our work complements prior studies by enabling an uncalibrated two-view solution.
Learnable Monocular 3D Priors. Monocular depth [39, 76] and surface normals [6] are two established 3D priors. Numerous studies have shown a learnable mapping between monocular images and corresponding 3D priors. Recently, [30] introduces the perspective field as a monocular 3D
Figure 2: We illustrate the framework for the proposed monocular camera calibration algorithm. In (a), a deep network maps the input image  $\mathbf{I}$  to the incidence field  $\mathbf{V}$ . A RANSAC algorithm recovers intrinsic from  $\mathbf{V}$ . In (b), we visualize a single iteration of RANSAC. An intrinsic is computed with two incidence vectors randomly sampled at red pixel locations. From Eq. (2), an intrinsic determines the incidence vector at a given location. The optimal intrinsic maximizes the consistency with the network prediction (blue and orange). Subfigure (c) details the RANSAC algorithm. Different strategies are applied depending on if a simple camera is assumed. If not assumed, we independently compute  $(f_x, b_x)$  and  $(f_y, b_y)$ . If assumed, there is only 1 DoF of intrinsic. We proceed by enumerating the focal length within a predefined range to determine the optimal value.
prior for inferring gravity direction. In this work, we suggest the incidence field is also a learnable monocular 3D prior. A common characteristic of monocular 3D priors is their exclusive relationship with 3D structures, making them invariant to 2D manipulations, e.g., cropping, and resizing. In contrast, camera intrinsic changes on spacial manipulation. Unlike other monocular priors, computing the groundtruth incidence field is straightforward, relying solely on intrinsic and image coordinates.
Our work focuses on intrinsic estimation from in-the-wild monocular images captured by modern imaging devices. Hence we assume an undistorted image as input. In this section, we first show how to estimate intrinsic parameters by using monocular 3D priors, such as the surface normal map and monocular depthmap. We then introduce the incidence field and show it a learnable monocular 3D prior invariant to 2D spacial manipulation. Next, we describe the training strategy and the network used to learn the incidence field. After estimating the incidence field, we present a RANSAC algorithm to recover the 4 DoF intrinsic parameters. Lastly, we explore various downstream applications of the proposed algorithm. Fig. 2 shows the framework of the proposed algorithm.
3.1 Monocular Intrinsic Calibration
Our method aims to use generalizable monocular 3D priors without assuming the 3D scene geometry. Hence, we start with monocular depthmap  $\mathbf{D}$  and surface normal map  $\mathbf{N}$ . Assume there exists a learnable mapping from the input image  $\mathbf{I}$  to depthmap  $\mathbf{D}$  and normal map  $\mathbf{N}$ :  $\mathbf{D},\mathbf{N} = \mathbb{D}_{\theta}(\mathbf{I})$ , where  $\mathbb{D}_{\theta}$  is a learned network. We denote the intrinsic  $\mathbf{K}_{\mathrm{simple}}$ ,  $\mathbf{K}$ , and its inverse  $\mathbf{K}^{-1}$  as:
$$
\mathbf {K} _ {\text {s i m p l e}} = \left[ \begin{array}{c c c} f & 0 & w / 2 \\ 0 & f & h / 2 \\ 0 & 0 & 1 \end{array} \right], \quad \mathbf {K} = \left[ \begin{array}{c c c} f _ {x} & 0 & b _ {x} \\ 0 & f _ {y} & b _ {y} \\ 0 & 0 & 1 \end{array} \right], \quad \mathbf {K} ^ {- 1} = \left[ \begin{array}{c c c} 1 / f _ {x} & 0 & - b _ {x} / f _ {x} \\ 0 & 1 / f _ {y} & - b _ {y} / f _ {y} \\ 0 & 0 & 1 \end{array} \right]. \tag {1}
$$
The notation  $\mathbf{K}_{\mathrm{simple}}$  suggests a simple camera model with the identical focal length and central focal point assumption. Given a 2D homogeneous pixel location  $\mathbf{p} = [x y 1]^{\intercal}$  and its depth value  $d = \mathbf{D}(\mathbf{p})$ , the corresponding 3D point  $\mathbf{P} = [X Y Z]^{\intercal}$  is defined as:
$$
\mathbf {P} = \left[ \begin{array}{l} X \\ Y \\ Z \end{array} \right] = d \cdot \mathbf {K} ^ {- 1} \left[ \begin{array}{l} x \\ y \\ 1 \end{array} \right] = d \cdot \left[ \begin{array}{c} \frac {x - b _ {x}}{f _ {x}} \\ \frac {y - b _ {y}}{f _ {y}} \\ 1 \end{array} \right] = d \cdot \left[ \begin{array}{l} v _ {x} \\ v _ {y} \\ 1 \end{array} \right] = d \cdot \mathbf {v}, \tag {2}
$$
where the vector  $\mathbf{v}$  is an incidence ray, originating from the 3D point  $\mathbf{P}$ , directed towards the 2D pixel  $\mathbf{p}$ , and passing through the camera's origin. The incidence field  $\mathbf{V}$  is the collection of incidence rays  $\mathbf{v}$  associated with all pixels at location  $\mathbf{p}$ , where  $\mathbf{v} = \mathbf{V}(\mathbf{p})$ .
3.2 Monocular Intrinsic Calibration with Monocular Depthmap and Surface Normal
In this section, we explain how to determine the intrinsic matrix  $\mathbf{K}$  using the estimated surface normal map  $\mathbf{N}$  and depthmap  $\mathbf{D}$ . Given the estimated depth  $d = \mathbf{D}(\mathbf{p})$  and normal  $\mathbf{n} = \mathbf{N}(\mathbf{p})$  at 2D pixel
(c) RGB Image I
Figure 3: In (a) and (b), we highlight the ground truth depthmap of a smooth surface, such as a table's side. Even with the ground truth depthmap, the resulting surface normals exhibit noise patterns due to the inherent high variance. This makes the intrinsic solver based on the consistency of the depthmap and surface normals numerically unstable. Further, (a)-(d) demonstrate a scaling and cropping operation applied to each modality. In (c), the intrinsic changes per operation, leading to ambiguity if a network directly regresses the intrinsic values. Meanwhile, the FoV is undefined after cropping. In comparison, the incidence field remains invariant to image editing, the same as the surface normal and depthmap.
(a) GT Depthmap
(d) Incidence Field V
(b) GT Depthmap Converted Surface Normal
location  $\mathbf{p}$ , a local 3D plane is described as:
$$
\mathbf {n} ^ {\intercal} \cdot d \cdot \mathbf {v} + c = 0. \tag {3}
$$
By taking derivative in  $x$ -axis and  $y$ -axis directions, we have:
$$
\mathbf {n} ^ {\intercal} \nabla_ {x} (d \cdot \mathbf {v}) = 0, \quad \mathbf {n} ^ {\intercal} \nabla_ {y} (d \cdot \mathbf {v}) = 0. \tag {4}
$$
Note the bias  $c$  of the 3D local plane is independent of the camera projection process. Without loss of generality, we show the case of our method for  $x$ -direction. Expanding Eq. (4), we obtain:
$$
n _ {1} \nabla_ {x} (d \cdot \frac {x - b _ {x}}{f _ {x}}) + n _ {2} \frac {y - b _ {y}}{f _ {y}} \nabla_ {x} (d) + n _ {3} \nabla_ {x} (d) = 0, \tag {5}
$$
where  $\nabla_{x}(d)$  represents the gradient of the depthmap  $\mathbf{D}$  in the  $x$ -axis and can be computed, for example, using a Sobel filter [33]. Next, re-parametrize the unknowns in Eq. (5) to get:
$$
a _ {1} f _ {x} f _ {y} + a _ {2} f _ {x} b _ {y} + a _ {3} f _ {y} b _ {x} + a _ {4} f _ {y} + a _ {5} f _ {x} = 0. \tag {6}
$$
Divide both sides of the equation by  $f_{x}$  to get:
$$
a _ {1} f _ {y} + a _ {2} b _ {y} + a _ {3} r b _ {x} + a _ {4} r + a _ {5} = 0, \tag {7}
$$
where  $r = \frac{f_y}{f_x}$ . By stacking Eq. (7) with  $N \geq 4$  randomly sampled pixels, we acquire a linear system:
$$
\mathbf {A} _ {N \times 4} \mathbf {X} _ {4 \times 1} = \mathbf {B} _ {4 \times 1}, \tag {8}
$$
where the intrinsic parameter to be solved is stored in a vector  $\mathbf{X}_{4\times 1} = [f_y\quad b_y\quad rb_x\quad r]^{\intercal}$ . This solves the other intrinsic parameters as:
$$
f _ {y} = f _ {y}, b _ {y} = b _ {y}, f _ {x} = \frac {f _ {y}}{r}, b _ {x} = \frac {r b _ {x}}{r}. \tag {9}
$$
The known constants are stored in matrix  $\mathbf{A}_{N\times 4}$  and  $\mathbf{B}_{4\times 1}$ . If we choose  $N = 4$  in Eq. (8), we obtain a minimal solver where the solution  $\mathbf{X}$  is computed by performing Gauss-Jordan Elimination. Conversely, when  $N > 4$ , the linear system is over-determined, and  $\mathbf{X}$  is obtained using a least squares solver. The above suggests the intrinsic is recoverable from the monocular 3D prior.
3.3 Monocular Incidence Field as Monocular 3D Prior
Eq. (9) relies on the consistency between the surface normal and depthmap gradient, which may require a low-variance depthmap estimate. From Fig. 3, even groundtruth depthmap leads to spurious normal due to its inherent high variance. Thus, minimal solver in Eq. (9) can lead to a poor solution.
As a solution, we propose to directly learn the incidence field  $\mathbf{V}$  as a monocular 3D prior. In Eq. (2) and Fig. 1, the combination of the incidence field  $\mathbf{V}$  and the monocular depthmap  $\mathbf{D}$  creates a 3D point cloud. In Eq. (3), the incidence field  $\mathbf{V}$  can measure the observation angle between a 3D plane and the camera. Similar to depthmap  $\mathbf{D}$  and surface normal map  $\mathbf{N}$ , the incidence field  $\mathbf{V}$  is invariant to the image cropping and resizing. Consider an image cropping and resizing described as:
$$
\mathbf {x} ^ {\prime} = \Delta \mathbf {K} \mathbf {x}, \quad \Delta \mathbf {K} = \left[ \begin{array}{c c c} \Delta f _ {x} & 0 & \Delta c _ {x} \\ 0 & \Delta f _ {y} & \Delta c _ {y} \\ 0 & 0 & 1 \end{array} \right], \quad \mathbf {K} ^ {\prime} = \Delta \mathbf {K K}, \tag {10}
$$
where  $\mathbf{K}'$  is the intrinsic after transformation. The surface normal map  $\mathbf{N}$  and depthmap  $\mathbf{D}$  after transformation is defined as:
$$
\mathbf {N} ^ {\prime} \left(\mathbf {x} ^ {\prime}\right) = \mathbf {N} (\mathbf {x}) = \mathbf {N} \left(\Delta \mathbf {K} ^ {- 1} \mathbf {x} ^ {\prime}\right), \quad \mathbf {D} ^ {\prime} \left(\mathbf {x} ^ {\prime}\right) = \mathbf {D} (\mathbf {x}) = \mathbf {D} \left(\Delta \mathbf {K} ^ {- 1} \mathbf {x} ^ {\prime}\right). \tag {11}
$$
Similarly, the incidence field after transformation is:
$$
\mathbf {V} ^ {\prime} \left(\mathbf {x} ^ {\prime}\right) = \left(\mathbf {K} ^ {\prime}\right) ^ {- 1} \mathbf {x} ^ {\prime} = \mathbf {K} ^ {- 1} \left(\Delta \mathbf {K}\right) ^ {- 1} \mathbf {x} ^ {\prime} = \mathbf {K} ^ {- 1} \mathbf {x} = \mathbf {v} = \mathbf {V} (\mathbf {x}). \tag {12}
$$
Eq. (12) shows that the incidence field  $\mathbf{V}$  is a parameterization of the intrinsic matrix that is invariant to image resizing and image cropping. Other invariant parameterizations of the intrinsic matrix, such as the camera field of view (FoV), rely on the central focal point assumption and only cover a 2 DoF intrinsic matrix. An illustration is shown in Fig. 3.
Next, we prove the incidence field  $\mathbf{V}$  is a monocular 3D prior pixel-wisely determined by monocular depth  $\mathbf{D}$  and surface normal  $\mathbf{N}$ . Expanding Eq. (4) using the incidence vector  $\mathbf{v}$  in Eq. (2), we get:
$$
n _ {1} \nabla_ {x} (d) v _ {x} + n _ {1} \left(\frac {d}{f _ {x}}\right) + n _ {2} v _ {y} \nabla_ {x} (d) + n _ {3} \nabla_ {x} (d) = 0. \tag {13}
$$
Combined with the axis-  $y$  constraint, the 2 DoF incidence vector  $\mathbf{v} = [v_x v_y 1]^{\top}$  is uniquely solved. Eq. (13) gives identical solutions when depthmap  $\mathbf{D}$  is adjusted by a scalar. This implies that the incidence field  $\mathbf{V}$  is pixel-wise determined by the up-to-scale depth map and surface normal map, indicating a learnable mapping from the monocular image  $\mathbf{I}$  to the incidence field  $\mathbf{V}$  exists.
Given the strong connection between the monocular depth map  $\mathbf{D}$  and camera incidence field  $\mathbf{V}$ , we adopt NewCRFs [69], a neural network used in monocular depth estimation, for incidence field estimation. We change the last output head to output a three-dimensional normalized incidence field  $\widetilde{\mathbf{V}}$  with the same resolution as the input image  $\mathbf{I}$ . We adopt a cosine similarity loss defined as:
$$
\widetilde {\mathbf {V}} = \mathbb {D} _ {\theta} (\mathbf {I}), \quad L = \frac {1}{N} \sum_ {i = 1} ^ {N} \widetilde {\mathbf {V}} ^ {\intercal} \left(\mathbf {x} _ {i}\right) \widetilde {\mathbf {V}} _ {\mathrm {g t}} \left(\mathbf {x} _ {i}\right). \tag {14}
$$
We normalize the last dimension of the incidence field to one before feeding to the RANSAC algorithm. That is to say,  $\mathbf{V}(\mathbf{x}_i) = [\tilde{v}_1 / \tilde{v}_3\quad \tilde{v}_2 / \tilde{v}_3\quad 1]^{\mathsf{T}} = [v_1\quad v_2\quad 1]^{\mathsf{T}}$
3.4 Monocular Intrinsic Calibration with Incidence Field