# Off-Line Camera-Based Calibration for Optical See-Through Head-Mounted Displays
## 3 Methods
### Formulation of the Calibration Procedure  
As mentioned in the Introduction, calibration procedures of OST displays aim to achieve a perfect spatial alignment between a 3D real-world scene and the computer generated scene perceived through the display. To this end, we need to estimate the perspective projection matrix of the off-axis rendering  
Figure 1: 3D representation of the off-axis eye–display pinhole model comprising the eye as the projection center and the see-through virtual screen as the imaging plane. camera that yields a correct mapping of each 3D vertex \(\mathbf{v}^{\mathrm{w}}\) of the virtual object onto the imaging plane space \(\mathbf{i}^{\mathrm{E}}\). This 3D-2D mapping (i.e., process of rasterization) must be done in a way that all 3D vertexes along a ray that originates from the rendering camera center (i.e., the eye center E) is rendered at the same location on the virtual screen and perfectly aligned to their real counterpart. Our calibration procedure measures such a projection matrix by means of a camera that acts as a replacement of the user's eye. Hereafter, we use the term viewpoint camera to refer to such a camera. In our formulation, as well as in the implementation of the procedure, we exploit a vision based inside-out tracking technique as done in [14; 16; 22]. Nonetheless, the same methodological guidelines that we here describe would also apply with an outside-in tracking technique, provided that it measures the pose of the sensor reference system, attached to the HMD, S with respect to the world scene W.  
Figure 2 shows the spatial relationships between the coordinate systems associated with the tracking camera S, the internal viewpoint camera C (the user's eye replacement), and the rendering camera E.  
The first step in our calibration procedure is the estimation of the viewpoint camera projection matrix that maps world points \(\mathbf{v}^{\mathrm{w}}\) onto camera image points \(\mathbf{i}^{\mathrm{C}}\):  
\[ \begin{split}\lambda\mathbf{i}^{\mathrm{C}}&={}^{ \mathrm{C}}\mathrm{K}\left[{}^{\mathrm{C}}_{\mathrm{S}}\mathbf{R}\ \ ^{\mathrm{C}}_{\mathrm{S}}\mathbf{f}\right]\left[{}^{\mathrm{S}}_{ \mathrm{W}}\mathbf{R}\ ^{\mathrm{S}}_{\mathrm{W}}\mathbf{f}\right]\mathbf{v}^{\mathrm{w}}=\\ &={}^{\mathrm{C}}\mathrm{K}\left[{}^{\mathrm{C}}_{\mathrm{W}} \mathbf{R}\ ^{\mathrm{C}}_{\mathrm{W}}\mathbf{f}\right]\mathbf{v}^{\mathrm{w}}={}^{ \mathrm{C}}\mathbf{p}\mathbf{v}^{\mathrm{w}}\end{split}  \tag{5}\]  
We determine the intrinsic matrix of the viewpoint camera \({}^{\mathrm{C}}\mathrm{K}\) and the relative pose between the viewpoint camera and the tracking camera \(\left[{}^{\mathrm{C}}_{\mathrm{S}}\mathbf{R}\ ^{\mathrm{C}}_{\mathrm{S}}\mathbf{f}\right]\) with conventional calibration routines [23] described in more detail in Section 4.2, whereas the pose of the world scene W with respect to S is measured on-line by querying the tracking camera \(\left[{}^{\mathrm{S}}_{\mathrm{W}}\mathbf{R}\ ^{\mathrm{S}}_{\mathrm{W}}\mathbf{f}\right]\). We used an OpenCV checkerboard as the target object for the inside-out tracking (see Section 4.1). Next, the key step in the procedure is the computation of the planar homography \({}^{\mathrm{E}}_{\mathrm{C}}\mathrm{H}\) between the two pinhole cameras C and E that enables the image points of the viewpoint camera \(\mathbf{i}^{\mathrm{C}}\) to be mapped onto the imaging plane of the see-through display (i.e., the virtual screen) \(\mathbf{i}^{\mathrm{E}}\). \[\lambda\mathbf{i}^{\mathrm{E}}={}^{\mathrm{E}}_{\mathrm{C}}\mathrm{H}{}^{ \mathrm{C}} \tag{6}\]  
where the planar homography \({}^{\mathrm{E}}_{\mathrm{C}}\mathrm{H}\) can be broken down as follows [24]:  
\[{}^{\mathrm{E}}_{\mathrm{C}}\mathrm{H}={}^{\mathrm{\alpha-E}}\mathrm{K}\left({} ^{\mathrm{E}}_{\mathrm{C}}\mathrm{R}+\frac{{}^{\mathrm{E}}\mathbf{f}(n^{ \mathrm{C}})^{\mathrm{T}}}{d^{\mathrm{C-\alpha}}}\right){}^{\mathrm{C}} \mathrm{K}^{-1} \tag{7}\]  
Figure 2: Illustration of the setup for the off-line calibration of the see-through display.