## 3 Materials and Methods
### Keypoint Extraction and Fault Detection  
Provided that there are cameras (one or more) that are calibrated and synchronized to the LiDAR point cloud, we record images \(l_{t}\) corresponding to the point clouds at each timeframe \(t\). We run a semantic keypoint detector, such as OpenPiffPaf [33] or Yolov8-Pose [34], which were trained on the Apollocar3D dataset [14]. For each image, this yields a set of \(\mathbf{P}_{t,c}^{semantic}\in\mathbb{R}^{66\times 3}\) keypoint detections for each \(c\)-th of the \(F\) cars in the image at time step \(t\). There are 66 semantic keypoints within a detection, each defined as pixel coordinates, and an associated confidence value as \(s_{j}=[u,v,confidence]_{j}\). On the corresponding LiDAR frame, we detect \(l\) cars using a 3D object detector.  
Figure 3: This figure depicts the semantic keypoint definitions from [14] (**left**), the original _Semantic Vertex Model_[15], and the new _Semantic Mesh Model_ (**right**).  
In order to find which car's keypoint detection corresponds to the LiDAR detection, we use the camera's calibration information and the pose of the vehicles as detected by the LiDAR. We take the deformable _Semantic Mesh Model_ and move it to the target's pose as detected by the LiDAR sensor. This is achieved as follows. For every vertex \(\mathbf{m_{i}}\) of the _Semantic Mesh Model_\(\mathbf{M}_{i}/^{\lceil\frac{\delta}{\delta}\cdot(\cup\cup)\rceil}\), we apply the transform  
\[\mathbf{m^{\prime}_{i}}=\mathbf{m_{i}}\mathbf{R^{T}_{i}}+p_{t}, \tag{11}\]  
yielding the transformed model \(\mathbf{M^{\prime}_{i}}^{f\lceil\frac{\delta}{\delta}\cdot(\cup\cup)\rceil}\) for the \(l\)-th car. Then, we project the vertices into the image:  
\[\mathbf{m^{\prime\prime}_{i}}=\mathbf{K}\mathbf{m^{\prime T}_{i}}, \tag{12}\]  
yielding the projected model \(\mathbf{M^{\prime\prime}_{i}}^{f\lceil\frac{\delta}{\delta}\cdot(\cup\cup)\rceil}\).  
We select the keypoint detection for the given 3D bounding box by defining an error metric \(Q_{e}\) between the projected _Semantic Vertex Model_ and the detected semantic keypoints. \(Q_{e}\) is defined as follows: First, the keypoints are normalized to the object detection size:  
\[\mathbf{k^{det}_{norm}}=\frac{\mathbf{k^{det}_{c}}-\mathrm{mean}(\mathbf{k^{ det}_{c}})}{\mathrm{diag}(\mathbf{k^{proj}_{l}})},\quad\mathbf{k^{proj}_{norm}}= \frac{\mathbf{k^{proj}_{l}}-\mathrm{mean}(\mathbf{k^{proj}_{l}})}{\mathrm{ diag}(\mathbf{k^{proj}_{l}})} \tag{13}\]  
Then, the keypoint error is determined:  
\[e_{\mathrm{kp}}(c,l,t)=\frac{1}{N}\sum_{j=1}^{N}\|\mathbf{k^{det}_{\mathrm{ norm},j}}-\mathbf{k^{proj}_{\mathrm{norm},j}}\|_{2} \tag{14}\]  
We also use an instance segmentation model Yolov8-mask [34] to detect the semantic mask of the targets. We compute the convex hull of the projected _Semantic Vertex Model_ and find their IoU:  
\[e_{\mathrm{hull}}(c,l,t)=\mathrm{max}(0,\tau_{\mathrm{hull}}-\mathrm{IoU}( \mathrm{Hull}(\mathbf{k^{proj}_{l}}),\,\mathcal{C}_{t,c}))\times f_{\mathrm{ penalty}} \tag{15}\]  
Finally, the complete metric is found:  
Figure 4: The process of LiDAR reference shape extraction. A LiDAR detector tracks the object and segments its point cloud within the bounding box. These points are transformed to the target’s local coordinate system and merged. Next, filtering, ground removal and mirroring take place. On the right, the completed reference LiDAR model can be observed.  
\[Q_{e}=e_{\text{kp}}(c,l,t)+e_{\text{null}}(c,l,t) \tag{16}\]  
and from the set of \(\mathbf{P}_{t,i}^{semantic}\) detections, we calculate the following metric for each \(\mathbf{P}_{t,i}^{semantic}\) in the set. We keep only the timeframes \(t_{confirmed}\) where the semantic keypoints are available and correct.