SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Results Table 2 shows the MPJPE results on mouse pose estimation. The baseline directly triangulates the 3D estimates via SVD based on the 2D detection results of pretrained HRnet. RANSAC is implemented based on the Anipose toolkit [36]. Both methods above are based on the results of a pretrained 2D detector without other fine-tuning, so the performance is relatively poor. DeeplabCut [15] is the most popular animal pose estimation method. GeneralTriang [10] is a stochastic framework for generalizable pose estimation, but it relies on the predicted 2D results, so it can not update the 2D detector in an end-to-end fashion. The  $\mathrm{Sup3d^{*}}$  model [9] is finetuned with MSE loss between the 3D estimate and 3D ground truth. TR model even achieves better results on some tasks than  $\mathrm{Sup3d^{*}}$  even without using the 3D ground truth. TR+DD model adds the domain discrimination loss to reduce domain discrepancy in cross-dataset tasks, slightly improving performance. Training the model with both TR loss and 3D supervision loss  $(\mathrm{TR} + \mathrm{Sup3d^{*}})$  can lead every view ray to focus on the 3D ground truth, so it achieves the best MPJPE results for all tasks.
Table 2: MPJPE results on mouse pose estimation. * means the method used 3D ground truth during training. The unit is mm.
Dannce	THM	Dannce-THM	THM-Dannce	Calms2-Dannce	Calms2-THM
Baseline	3.25	5.67	10.02	9.48	20.65	22.50
RANSAC[36]	2.65	4.96	9.66	8.01	18.39	17.50
DeeplabCut[15]	4.20	4.40	11.52	11.43	21.43	17.78
GeneralTriang[10]	2.56	4.65	8.56	8.06	18.55	16.50
Sup3d* [9]	2.34	3.45	7.33	7.15	15.50	14.46
TR	2.22	3.49	5.87	7.15	15.16	15.11
TR+DD	-	-	5.78	5.83	14.84	15.07
TR+Sup3d*	2.00	3.37	5.76	5.57	14.84	14.38
To demonstrate the "plug-and-play" ability of TR loss, we implement the experiments on four tasks by plugging the triangulation head and the confidence head into different 2D detectors (including PVT[37], SCNet[38], and MobileNet[39]). As shown in Table 3, all 2D detectors get lower MPJPE errors with TR loss. Especially for the cross-domain tasks (Dannce-THM and THM-Dannce), TR loss reduces the MPJPE by an average of  $3.85\mathrm{mm}$ $(36\%)$ .
We also present the MPJPE results of each joint on the Dannc dataset in Appendix A.5. The results show that the nose and ears have lower errors than those of other points, while limb positions are difficult to estimate because of the self-occlusion, and body joint error is relatively large due to the lack of texture features.
Table 3: MPJPE results of different 2D detectors with and without TR
PVT	SCNet	MobileNet
w/o TR	w/ TR	error	w/o TR	w/ TR	error	w/o TR	w/ TR	error
Dannce	3.12	2.74	-0.38	3.27	2.48	-0.79	3.26	2.81	-0.45
THM	4.57	3.81	-0.76	4.47	3.49	-0.98	4.94	4.04	-0.90
Dannce-THM	12.87	7.96	-4.91	11.08	6.44	-4.64	11.15	5.57	-5.58
THM-Dannce	11.09	7.74	-3.35	8.49	5.48	-3.01	9.47	7.85	-1.62
4.4 Further analysis
Data efficiency analysis We compare the MEJPE of models trained with different data sizes and present the results in Figure 3 (a) - (b). Naive triangulation means triangulating directly based on the 2D estimates of the backbone. Supervised triangulation means training the total model with the ground-truth 3D labels. Our model is trained with TR loss. Figure 3 (a) shows that the MPJPE of all models decreases as the training data size increases. Our model performs better than other models in most cases especially when the training data size is small ( $5\% -10\%$ ). The cross-subject results in Figure 3 (b) show that our self-supervised method can significantly improve the generalization ability of different individuals, but supervised triangulation fails.
Sensitivity testing on the score threshold To avoid the trivial solution, we set a threshold range to the confidence score. Figure 3 (c) shows the MPJPE results curves with different threshold ranges. We set the threshold range to be symmetric about 0.5 as shown on the x-axis, where "No" means model without score head and (0.0, 1.0) means model without threshold. Models without a score head get degraded performance since they can not adjust the weight of each view in the triangulation process, while models without a score threshold get significantly large errors since TR loss to assign most views with a weight close to 0, in this way, these views have no contributions to triangulation. Except for these two extreme settings, our method obtains similar results because TR loss and confidence can be adaptively balanced during the training process.
Ablation studies Figure 3 (d) - (e) present the ablation experiment results. The baseline model and 3D supervised model share the same model structure as the "TR+confidence" model having the confidence head, while the "TR" model does not have the confidence head. The results show that TR
(a) Subject 1 - Subject 1
(d) Subject 1 - Subject 11
Figure 3: (a) - (b) Data efficiency analysis on Human3.6M dataset. (c) Sensitivity testing on the score threshold. (d) - (e) Ablation studies. The results show that TR loss is the most critical component and DD loss is optional. (f) Results on different numbers of cameras.
(b) Subject 1 - Subject 11
(e) CalMS21 - THM
(c) Different thresholds
(f) Different cameras
loss is the most critical component, yielding a  $77.56\%$  improvement on Subject 1-Subject 11 and a  $46.41\%$  improvement on the CalMS21-THM experiment compared to the baseline. The confidence head also provides a slight improvement due to its learnable weights selecting high-confidence views. The DD loss contributes the least to the performance.
Results on different numbers of cameras Figure 3 (f) shows the results on different numbers of cameras on the Dannce dataset. The error increases slightly as the number of cameras decreases from 6 to 3, and increases sharply when the number of cameras decreases to 2. However, the TR loss achieves consistent improvement for all camera numbers.
Running time analysis In the training process, our method is the same as typical deep learning models that compute the loss functions (including TR loss) only once per batch to update the weights. The TR loss does not require additional iterative optimization within a batch. Therefore, TR loss does not excessively increase the training time. During testing, our method also does not require iterative optimization. The experiments show that the average per-batch running time of the same model with and without TR loss on the Dannce dataset is 0.39s and 0.46s, respectively.
4.5 Qualitative results
Figure 4 shows qualitative examples of the Human3.6M dataset. The first four columns present the 2D estimates in different views generated by the 2D backbone, while the last column shows the 3D estimates of different methods. The 2D detector is pretrained in the same way as the "Subject 1 - Subject 11" experiment. As shown, the 2D backbone can not provide steady 2D estimates because of the domain discrepancy and limited training data. The naive triangulation and supervised method are incapable of correcting these errors (such as the right arm and right hip in view 2). However, our method corrects the failed 2D estimates in each view via global geometry consistency self-supervision, providing accurate 3D results.
Figure 4: Qualitative results on Human3.6M dataset. Our method provides geometrically consistent 2D estimates in each view and an accurate 3D estimate.
The examples of mouse pose estimation in Figure 5 also show that our method obtains better 3D estimates. Due to space limitations, we only show three views and the 3D results here. All models in Figure 5 are pretrained on the THM training set. As shown in column 3 of Subfigure (a), only our method provides geometrically plausible 2D estimates for limbs and body. Moreover, only our method distinguishes the left paw and right paw in Subfigure (b). More qualitative results and failure examples can be found in Appendix A.6.
(a) THM dataset example
(b) Dannc dataset example
Figure 5: Qualitative results of mouse pose estimation. Our method obtains accurate results for the hard joint (body) and unseen joints (paws) in View 3.
In this paper, we present TR loss for multiview 3D human/animal pose estimation, which aims to minimize the sum of distances between 3D estimation and view rays. Its advantages contain (1) Self-supervised. It achieves global geometric consistency without 3D supervision and any reprojection processes. (2) Simple. It only minimizes the smallest singular value of the triangulation matrix and can be easily applied to various 2D backbones; (3) Effective and data efficient. It achieves state-of-the-art MPJPE performance for single-frame 3D pose estimation on the Human3.6M dataset and obtains competitive results with only  $5\%$  training data. Future work includes combining TR loss with end-to-end learnable association methods to generalize to multi-object pose estimation, or using TR loss to jointly optimize the camera poses and object poses for moving cameras.
We thank the reviewers for their constructive comments. This work was supported in part by Guoqiang Institute of Tsinghua University (No.2021GQG0001), NSFC (No.62171255), China Postdoctoral Science Foundation under Grant BX20220186, and Shimu Tsinghua Scholar Program.
[1] Qiuxia Lin, Linlin Yang, and Angela Yao. Cross-domain 3d hand pose estimation with dual modalities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[2] Kerui Gu, Linlin Yang, Michael Bi Mi, and Angela Yao. Bias-compensated integral regression for human pose estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-16, 2023.
[3] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325-1339, jul 2014.
[4] Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Scott Godisart, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social interaction capture. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[5] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation New benchmark and state of the art analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.