## 3 Materials and Methods
### Semantic Mesh Shape Optimization
#### 3D LiDAR Shape Loss  
The _Semantic Mesh Model_ is also aligned to the LiDAR reference shape, \(P_{B}^{ref}=\{p_{0},\dots p_{n}\}\) by also minimizing a 3D shape loss. This helps enforce the scale of the model and reduce unreasonable warping of the shape in cases of noisy 2D keypoint detections. The function combines point-to-plane, point-to-edge and Laplacian smoothing losses, all of which are readily implemented in the Pytorch3D library. Mathematically, the LiDAR shape loss is shown below:  
\[L_{\text{LiDAR}}=\lambda_{t}L_{\text{face}}+\lambda_{\text{e}}L_{\text{edge} }+\lambda_{1}L_{\text{laplace}}, \tag{22}\]  
with \((\lambda_{t},\lambda_{\text{e}},\lambda_{1})=(10,1,1)\) in all experiments.  
The _point-to-plane_ loss measures how far reference LiDAR point \(\mathbf{p}_{n}\) lies from the closest mesh face \(f_{k}\) along the face normal, where \(\mathcal{F}=\left\{f_{k}\right\}_{k=1}^{N_{f}}\) are the faces of the _Semantic Vertex Model_. The pull of this loss is therefore in the normal direction of the mesh faces, helping the shape expand or contract to accurately capture the scale of the target. Additionally, since our _Semantic Mesh Model_ has far fewer vertices compared to Autoshape [39], a point-to-point loss would pull the sparse vertices toward dense clusters in the inhomogeneous LiDAR reference shape. For face \(f_{k}\) with unit normal \(\mathbf{n}_{k}\) and a point on the face \(\mathbf{p}_{k}^{f}\), the squared distance is  
\[d^{2}\big{(}\mathbf{p}_{n},f_{k}\big{)}=\big{(}\mathbf{n}_{k}^{\top}(\mathbf{p} _{n}-\mathbf{p}_{k}^{f})\big{)}^{2}. \tag{23}\]  
Averaging over all \(N\) points and choosing for each point the minimal squared distance to any of the \(N_{f}\) faces yields the total face loss  
\[L_{\text{face}}=\frac{1}{N}\sum_{n=1}^{N}\min_{k=1,\ldots,N_{f}}d^{2}\big{(} \mathbf{p}_{n},f_{k}\big{)}, \tag{24}\]  
where each face \(f_{k}\) has a unit normal \(\mathbf{n}_{k}\) and centroid \(\mathbf{p}_{k}^{f}\).  
The _point-to-edge_ loss acts in a similar fashion while also enforcing that the faces cannot expand perpendicularly to their normal direction. Thus, for the edges \(\mathcal{E}=\{e_{\ell}\}_{\ell=1}^{N_{\ell}}\), we calculate  
\[L_{\text{edge}}=\frac{1}{N}\sum_{n=1}^{N}\min_{\ell=1,\ldots,N_{e}}\big{\|} \mathbf{p}_{n}-\Pi_{e_{\ell}}(\mathbf{p}_{n})\big{\|}_{2}^{2}. \tag{25}\]  
where \(\Pi_{e_{\ell}}(\mathbf{p})\) denotes the orthogonal projection of point \(\mathbf{p}_{n}\) onto edge \(e_{\ell}\).  
In order to enforce smoothness of the resulting mesh, we calculate the _Laplacian loss_. Let \(\mathbf{L}\) be the cotangent Laplacian of the undeformed mesh.  
\[L_{\text{lap}}=\big{\|}\mathbf{L}\big{(}\mathbf{M}_{0}+\delta\big{)}\big{\|}_ {F}^{2}. \tag{26}\]  
The _Point to Plane_, _Point to Edge_, _and Laplacian_ loss functions are already part of the Pytorch3D library; please refer to its source code for more details at [43]. The novelty lies in the formulation of the _Semantic Mesh Model_ and the _Reference Lidar Shape_ between which the loss functions perform a comparison.  
Thus, the total objective is the combination of the re-projection and LiDAR losses.  
\[L(\Theta)=L_{\text{2D}}+L_{\text{LiDAR}}. \tag{27}\]  
We minimize (27) using the PyTorch (version 2.0.1 with CUDA 11.7) implementation of the Adam optimizer. The above LiDAR loss functions are already part of the Pytorch3D library (corresponding to Pytorch and CUDA versions). The learning rate is controlled by a scheduler that reduces the learning rate by a factor of 0.5 if the loss value does not decrease for 10 consecutive iterations. All deformation parameters \(d_{i}\) are initialized to zero, while the camera poses \((\mathbf{R}_{t},\mathbf{t}_{t})\) are seeded using the LiDAR-based estimates. The optimization runs for a maximum of 1000 iterations with early stopping triggered if the learning rate drops below \(l_{r}=10^{-6}\). The loss weights \(\lambda_{\text{fr}},\lambda_{\text{e}},\lambda_{1}\) used in Equation (22) are fixed throughout the optimization for all scenarios.