## 3 Materials and Methods
### Problem Statement  
Firstly, we define the world coordinate system as a single coordinate system to which all of the sensors are calibrated. In the tests on the Argoverse dataset, this is the dataset's own world coordinate system; in our own experiments, this is the UTM coordinate system. We also define for each camera the camera coordinate system, which follows the OpenCV [41] convention. Next, the _Semantic Mesh Model_ is defined as follows: we define the vehicle coordinate system following ISO 8855. The origin is at the center of the bottom face of the 3D bounding box of the vehicle. The \(X\) axis points forward along the vehicle's  
Figure 2: This figure demonstrates the full process. The initial Reconstruction Phase is followed by an image-based 6 DOF pose estimation. First, the target vehicle is tracked both in the LiDAR point-cloud timeseries and the image for semantic keypoint extraction. Next, the LiDAR reference shape is constructed, and the keypoint timeseries is filtered for faulty detections. Finally, the _Semantic Mesh Model_ shape is optimized from the _undeformed Semantic Mesh Model_ by minimizing 3D loss and the re-projection loss. This model is reusable for highly accurate single- and multi-view pose estimation. longitudinal axis, the \(Y\) axis points left, laterally to the vehicle, and the \(Z\) axis points vertically up. The deformable and symmetric _Semantic Vertex Model_, \(\mathbf{M}^{semantic}\) is defined in our first work [15] as a structured set of 66 vertices representing semantically meaningful keypoints, which are associated with a distinct semantic label as defined in the ApolloCar3D dataset [14]. We add a brief introduction here for clarity. Formally,  
\[\mathbf{M}^{semantic}=[\mathbf{m}_{1},\mathbf{m}_{2},\dots,\mathbf{m}_{66}], \quad\mathbf{m}_{j}=[x,y,z] \tag{1}\]  
where \(\mathbf{m}_{j}\) are the 3D semantic vertices and \(\mathbf{M}_{semantic}\in\mathbb{R}^{66\times 3}\). The vertices are arranged in such a way that for \(j=1,2,\dots,33\)\(m_{j}\) constitutes the left side of the vehicle. These vertices will be deformed during the optimization as follows:  
\[\mathbf{m}_{j}^{def}=\mathbf{m}_{j}+\mathbf{d}_{j}, \tag{2}\]  
where \(d_{j}\) is the deformation vector for the \(j\)-th index, and \(\mathbf{D}\in\mathbb{R}^{33\times 3}\) is the set of deformation vectors used. For \(j=34,35,\dots,66\)\(m_{j}\) is the right side, which due to the assumption of symmetry can be parametrized as follows:  
\[\mathbf{m}_{j}=\mathbf{T}_{xz}\mathbf{m}_{(j-33)}, \tag{3}\]  
where  
\[\mathbf{T}_{XZ}= \begin{bmatrix}1&0&0\\ 0&-1&0\\ 0&0&1\end{bmatrix} . \tag{4}\]  
From here on, we refer to this deformation process as the deformation function  
\[\mathbf{M}^{deformed}=f^{deformation}(\mathbf{M}^{semantic},D). \tag{5}\]  
In this work, we extend this vertex model to a mesh model by defining faces between the vertices that create a watertight mesh from the semantic keypoints. \[\mathcal{S}^{\text{semantic}}=(\mathbf{M}^{\text{semantic}},\mathcal{F}), \tag{6}\]  
where the following apply:  
* \(\mathbf{M}^{\text{semantic}}\) is the aforementioned vertex set. * \(\mathcal{F}=\{\mathbf{f}_{k}\}_{k=1}^{N_{f}}\) is the set of triangular faces defined by vertex indices, creating a watertight manifold mesh representing the vehicle's surface: \[\mathbf{f}_{k}=[i,j,l],\quad\text{with}\;i,j,l\in\{1,\dots,66\},\quad k\in\{ 1,\dots,N_{f}\}.\] (7)  
The triangular faces were selected in such a way that they form a watertight mesh. The connectivity of the keypoints was defined manually so that the individual faces belong to semantically meaningful parts of the vehicle. For example, the headlight has four corners in the _Semantic Vertex Model_, so in order to define triangular faces, the plane of the headlight was divided into two faces (see Figure 3). All in all, we define 128 mesh faces, for exact face definitions refer to Appendix A.