Triangulation Residual Loss for Data-efficient 3D Pose Estimation
Authors: Jiachen Zhao
Tao Yu
Liang An
Yipeng Huang
Fang Deng
Qionghai Dai
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D pose estimation
triangulation
animal pose estimation

Abstract: This paper presents Triangulation Residual loss (TR loss) for multiview 3D pose estimation in a data-efficient manner. Existing 3D supervised models usually require large-scale 3D annotated datasets, but the amount of existing data is still insufficient to train supervised models to achieve ideal performance, especially for animal pose estimation. To employ unlabeled multiview data for training, previous epipolar-based consistency provides a self-supervised loss that considers only the local consistency in pairwise views, resulting in limited performance and heavy calculations. In contrast, TR loss enables self-supervision with global multiview geometric consistency. Starting from initial 2D keypoint estimates, the TR loss can fine-tune the corresponding 2D detector without 3D supervision by simply minimizing the smallest singular value of the triangulation matrix in an end-to-end fashion. Our method achieves the state-of-the-art 25.8mm MPJPE and competitive 28.7mm MPJPE with only 5\% 2D labeled training data on the Human3.6M dataset. Experiments on animals such as mice demonstrate our TR loss's data-efficient training ability.

Triangulation Residual Loss for Data-efficient 3D Pose Estimation
Jiachen Zhao Tsinghua University Beijing, China 100084 zhao__ waw.hhchheen...... com
Tao Yu' Tsinghua University Beijing. China 100084 ytrock@126.com
Liang An
Tsinghua University
Beijing. China 100084
al17@mails ...17mmailsttsinnnnuee twwnghua..u...a edu. cn
Yipeng Huang
Fang Deng
Beijing Institute of Technology Beijing, China 100081 huangyipeng@bit. edu. cn
Beijing Institute of Technology
Beijing, China 100081 dengf oaggaan@bit..o edu com
Qionghai Dait Tsinghua University Beijing, China 100084
qhdai @mails. www.ils.stiiguuuaa.ommmm edu.
Abstract
This paper presents Triangulation Residual loss (TR loss) for multiview 3D pose estimation in a data-efficient manner. Existing 3D supervised models usually require large-scale 3D annotated datasets, but the amount of existing data is still insufficient to train supervised models to achieve ideal performance, especially for animal pose estimation. To employ unlabeled multiview data for training, previous hipolar-basee consistency provides a self-supervised loss that considers only the local consistency pairwise views, resulting limited performance and heavy calculations. In contrast, TR loss enables self-supervision with global multiview geometric consistency. Starting from initial 2D) keypoint estimates, the TR loss can fine-tune the corresponding 2D detector without 3D supervision by simply minimizing the smallest singular value of the triangulation matrix in an end-to-end fashion. Our method achieves the state-of-the-art 25.8mm MPJPE and competitive 28. 7m MPJPE with only 5% 2D labeled training data on the Human3.6k dataset. Experiments on animals such as mice demonstrate our TR loss's data-efficient training ability.
Introduction
Multiview 3D human/animal pose estimation is a crucial and challenging task with wide application in mixed reality telepresence, animation, and neuroscience. Most existing animal pose estimation
(APE) methods are transferred from the highly developed human pose estimation (HPE) methods 2 The success of HPE depends large-scale datasets 35 and high-quality 3D modelS (6]. However, these conditions are not met in APE tasks. On the one hand, labeling large-scale datasets for each species requires significant user effort but cannot obtain the same commercial benefits as HPF. On the other hand, obtaining 3D ground truth via sticky marks not feasible since the animal's body could be tiny and animals tend to destroy the markers.
Triangulation is crucial approach for multiview pose estimation due to its efficiency and effectiveness. However, traditional triangulation involves non-differenniable argmax and RANSAC process
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

disabling the end-to-end training. The learnable triangulation methods [7-10 have been proposed to perform triangulation an end-to-end fashion. Its typical pipeline is (1) generating the 2D joint heatmaps for each view, (2) using soft-argmax to get 2D joint coordinates from the heatmaps, and (3) performing triangulation to estimate 3D joint coordinates. Finally, the MAE loss between the ground truth and the predicted 3D positions is used to train the model as shown in Figure (a). Several works introduce learnable weights [9], and stochastic frameworks [00] to improve performance and generalization. However, 3D supervision needs large-scale labeled data which is usually unavailable for APE.
introduce epipolar geometry supervisory signal to eliminate 3D ground truth. Its key idea is shown Figure (b). The epipolar geometry restricts that the 2D keypoint in view 2 must lie in the corresponding epipolar line 1 from view 1. To enable the epipolar constraint to be trained end-to-end, Yao al. [22 propose the epipolar divergence which measures the distribution distance between epipolar lines. These epipolar-based methods pursue local consistency in pairwise views, so most of them involve trivial pairing and reprojection processes. They may also suffer from heavy computation when the view number is large.
Some papers [1] [4]
3D estimate
.3D ground truth
4
epipolar line
2D estimate
H2
9
(a) Supervised triangulation
(b) Painwise epipolar consistency
3D ground truth (not used)
3D estimate
C2
C2
G2
Iteration
Iteration
Iteration
(c) Global geometric consistency via TR loss
Figure 1: (a) Supervised triangulation trained with MSE loss between 3D estimation and 3D ground truth. (b) Unsupervised triangulation with pairwise epipolar consistency. (c) Self-supervised triangu- lation with TR loss. The TR loss realigns the predicted heatmap locations to the accurate point using multi-view consistency. Geometrically, TR loss can be viewed the sum of distances between 3D estimate and view rays as shown the figure. Algebraically, TR loss can be simply formulated as smallest singular value of the triangulation matrix.
The paper tackles these challenges by leveraging global multiview geometric consistency. The of our methods is simple but powerful loss function, called Triangulation Residual Loss (TR loss), which is defined as the sum distances between the 3D estimate and view rays shown in Figure (c). Supervised methods aim to minimize the absolute error between the 3D estimate and 3D ground truth. Differently, our goal is minimize the residual between the 3D estimate and 2D observations, where 2D observation is represented by the view ray from the camera center to the 2D keypoint in the image. In every single iteration in Figure([[c) we estimate the optimal 3D estimate by triangulation, but there must be residuals, i.e. the distances between 3D estimate and view rays, since the 3D estimate doesn't lie in all view rays. Iterative minimizing the residual will lead the 3D estimate to converge to a geometrically consistent position. Moreover, we use algebraic errors to approximate geometric errors, TR loss is simplified the smallest singular value of the triangulation matrix without any reprojection processes.

---

The contributions can be summarized as follows: (1) A simple yet effective loss function, TR loss, for multiview 3D pose estimation. It achieves data-efficient training by using the multiview geometrical consistency without 3D supervision and any reprojection processes. It can be sim- ply formulated as minimizing the smallest singular value of the triangulation matrix. (2) A new benchmark for laboratory mouse 3D pose estimation based on existing datasets (CalMS21 and Dannce) and THmouse data collected by ourselves. (3) Our method achieves state-of-the-ar MPJPE performance for single-frame 3D pose estimation on the Human3,6 3.6M dataset and obtains com- petitive performance with only 5% 2D labeled training data. Code has been made available at http:///ttuuboom///jjjiiaahhaeee www.cchhenn999/eaagglllllaaaaa Resiiia--oss.ss.LLosso.....
2
Related works
Learnable triangulation Triangulation is popular in 3D multiview animal pose estimation [1
[6]. To overcome the non-differentiabiiiy of the vanilla triangulation (RANSAC), the soft and differentiable versions of RANSAC have been proposed 7 IO]. The core of these methods is to
select high-confidence 2D estimates via the probabilistic learning scheme. Karim et al. [9] utilize the learnable weights to softly select the 2D estimates. Kristijan et al. [TO] fix the pretrained 2D detector and use the subsequent hypothesis selection process to finetune the 3D pose. Therefore, this method heavily relies on the performance of the pretrained 2D detector. Some volumetric approaches [7 [8] also achieve great performance on 3D pose estimation. However, all the above methods require ground-truth 3D labels during training, so they are not suitable for APE. Some methods leverage the epipolar geometry to learn feature consistency [1] geometry consistency [3 [4] so that
the 3D supervision is no longer required. However, all these methods involve complex reprojection and alignment in pairwise views. Therefore, they only consider the local pairwise consistency and
become inapplicable when the number of cameras is large.
Animal pose estimation The physical behavior of animals is the most intuitive external manifestation of neural activity. However, observing animal behavior is one of the most time-consuming operations in biology or neuroscience research. Experimenters need to spend several hours observing and recording the behavior of mice only to complete a trial. A research topic often requires hundreds of trials. Fortunately, vision-based keypoint detection methods provide an efficient tool to measure the behaviors of lab animals. However, labeling data under different experimental paradigms is still labor-intensive. Therefore, Nomain-adaptatie
[2] methods have been introduced to APE. Li et al. [19] and Cao et al. (22) design pseudo-label generation approaches to expand labeled data during training. Jiang al. [20] and Mu et al. [2] utilize synthetic animal images the source domain and reduce the domain shift to predict animal pose in real images. RegDA (23) extends the disparity discrepancy theory from classification to kcypoint detection and improves the cross-domain keypoint detection performance. The above methods are limited to 2D pose estimation, which loses key information in the analysis of experimental animal behavior. Further, this paper focuses on multiview 3D APE with only a few 2D labels.
Different from existing learnable triangulation methods, our method achieves global 3D geometry consistency without 3D supervision and reprojection operations. Moreover, the loss can be simply formulated as minimizing the smallest eigenvalue of the triangulation matrix.
3 Methods
3.1 Problem formulation
We assume few 2D labeled pose data are given and aim to estimate the 3D pose from multiview images. The 2D labeled data Dj can be an existing large-scale dataset or a few annotations under the same dataset, which contains input images 1; IR ff xXXx3 x3 and the corresponding ground-truth 2D keypoint coordinates J?d E IR J for all keypoints, where is the number of keypoints. We can train a 2D keypoint detector on the labeled data # x3 + R 1g/x3 The target multiview images D, are simultaneously captured from C' cameras but unlabeled, where Le, is the image captured by the eth camera at the ith scene. Our goal is to train a 3D keypoint detector XC xXW x3 + /x3
to predict the global 3D positions for D,.. Throughout this paper, we use Fay to indicate the 2D coordinate of jth joint in eth view and use X, to indicate the 3D coordinate of jth joint.
3

---

3.2 Model structure
Figure presents the framework of our model including a backbone network Φ, heatmap head h, confidence head f. differentiable triangulation head and an optional domain discriminator 2/.
Φ
Supp2 (Ds)
Labeled 2D datasets
Shared weights
U
Lad (Ds, Dy)
Shared weights
h
Φ
Triangulate
Ly (DT)
view; Unlabeled 3D datasets
Figure The framework our method containing backbone network Φ, a heatmap head h, confidence head f. differentiable triangulation head and optional domain discriminator 1p.
The input data contains several 2D labeled images and unlabeled target multíview images. All input images fed into CNN-based backbone & to generate feature maps. Following the backbone, a heatmap head transforms the intermediate feature maps to keypoint heatmaps R x x3 - H x w' xJx2 Then the keypoint heatmaps are transformed into 2D coordinates via a differentiable process including spatial softmax in Eq. I and soft-argmax Eq. 2
H' W E h
Softmax exp(Hee))
exp(He,, (
(1)
H' Be. ΣV
W
1h'
w/) + He, (h',(h"ww)) w()
Here, He, is the heatmap for jth joint in cth view. H' and W are the heatmap height and width. // and are the spatial indexes across heatmap height and width. means the element-wi wise product. We further denote the combination of Eq. and Eq. (2) as K., the 2D keypoint detector can be formulated as
Soj (n o h (I.)))
Parallel with the heatmap head, confidence head f produces the triangulation confidence for each keypoint each view H J
We,j (f ({L)))
(4)
Following and f. triangulation head estimates the 3D position X; based on the multiview 2D estimates by minimizing the weighted sum of reprojection errors in all views:
X; arg min X,
We.j || P. 12
where || is the L2 norm, P. is the projection matrix of cth view and We,j is the learnable weight to adjust the contribution of each view in the triangulation process so that the erroneous 2D estimates will have little effect. However, direct solving (5) is not differentiable, so we use the linear algebraic method, which is solve the overdetermined system of equations on homogeneous X,:
Wy + A, X, =0
(6)

---

where Aj is the matrix composed of P. and &e.g. We further annotate A; = w @ Aj as the triangulation matrix in this paper. The details about how to formulate A, can be found in Appendix A. The overdetermined equations Eq. (6) can be solved via SVD as A; -U-V. The optimal estimate X is the last column of V. So far, supervised triangulation methods train the model
with MSE loss between the 3D estimate and 3D ground truth: Lrp || x3'Il" Differently,
we eliminate the 3D supervision by using the TR loss presented in Section 3.3
3.3 Triangulation residual loss
Figure (c) presents the intuition of TR loss from geometric perspective. In the first iteration, can estimate a preliminary 3D estimate X* by solving Eq. (6). Since the 2D detector is pretrained on limited data, there must be estimation errors in the 2D position and there may be some large deviations, such as H3. As a result, X is inaccurate and does not fall on view rays. In this paper, we minimize the sum of distances from X to each view ray so that X X will converge a reliable position. During iterative minimization, the learnable weight W.,j is also updated in such a way that 2D estimates with less consistency (such as Hy) are given smaller weights, and 2D estimates with more consistency (such as M1 and H2) are given larger weights. To sum up, the TR loss realigns the predicted heatmap locations the accurate point using multi-view consistency.
From an algebraic perspective, the overdetermined Eq. (6 can not be fully established. There must be residual as Á X o//. Minimizing the sum of distances from X* to each view ray can be reduced to minimize the
Cres.j = A xx:|'
Since X; is the right singular vector corresponding the smallest eigenvalue of Aj. we omit the subscript j, Lrss can be reformulated as
A 1/2 = 1 I | o vall = ||o ||0411
(8)
In Eq. (8) Á with rank 4 can be decomposed by SVD Ā= ouu where 01 02
03 2 04 are the unit singular values of X", uss and r,s are the left and right singular vectors of A. According the property of SVD. both u,s and uys unit vectors and orthogonal to each other, so Eq. (8) is established. Here, we directly use PyTorch's 's built-in auto-diffcrential SVD function, while robust differentiable SVD 24 is also alternative. Finally, Liss can be simply formulated as minimizing Aj smallest eigenvalue o4:
Crea = 1|a,1
(9)
Eliminate trivial solutions A straightforward way to obtain the learnable weight in A, is through the sigmoid function. If so, the TR loss tends to assign high confidence (nearly 1) to only two views and assign low confidence (nearly 0) other views, since the fewer views involved in triangulation, the smaller the error. The TR loss can no longer improve 2D estimates in low-confidence views. Moreover, all weights equal is trivial solution for Eq. (7). So we set threshold range for the confidence score to shrink it. More details about the threshold are discussed in Section 4.4
3.4 Total loss functions
Since the TR loss only pursues geometric consistency, this may cause the 2D estimates to be non- semantic points instead of the correct joints. Therefore, we still need 2D supervision loss Laupa on the labeled 2D data D, Same as in [25] 26), we utilize the pixel-wise MSE loss between the predicted
heatmap His and the ground-truth heatmap Higi
Camp2d
Σ
|| H1.
(10)
5

---

If the labeled images Dj and the target multiview images Du are from different domains, we can also add domain discriminator 2 trained with domain discrimination loss (2, P.)) reduce domain
discrepancy. Same in [22 27 28), the domain discrimination loss is defined as cross-entropy loss
and set be adversarial Lup2a and Lee::
N log(() + +(1 24) log(1 2,))
La
Σ
(11)
where 2; and 2 are the ground-trute -runh and predicted domain labels, respectively. The total loss function
== Lres + Ochouupd + BLad
(12)
where a and are tradeoff hyperparameters. The ablation studies Section 4.4 show that our proposed TR loss is the most critical component but the DD loss is dispensable.
4 Experiments
To evaluate the effectiveness of our proposed method, we conduct experiments on lab mouse pose estimation and human pose estimation. It should be noted that we only use 2D labels (not 3D) to train our model.
4.1 Implement details
We implement our model based on the mmpose library [29]. We apply the HRnet as the backbone, build the heatmap head with convolution layer with 3x3 kernel size, and build the confidence head with two convolution layers followed by three linear layers [512. 256, J] with a sigmoid activation function. The domain discriminator is an average pooling layer followed by three linear layers [512, 256, J] with a sigmoid activation function. The models are trained on NVIDIA 3090 GPU with the Adam optimizer and an imitial learning rate of 1e -5 The loss tradeoff parameters are so all losses have the same effect.
4.2 Human Pose Estimation
Datasets We conduct human pose estimation experiments the Hluman3.6M dataset. The Hu- man3.6M dataset 3 contains 3.6 million frames of single-person activity captured by 4 synchronized cameras. Its ground-truth 3D joint position is obtained by marker-based motion capture system with 10 IR sensors. We assume that only 2D labels in the training dataset are available to train a 2D detector. Then, we train the total model with TR loss on the 1% multiview unlabeled data and heatmap loss on the labeled training images. Therefore, our method does not require any additional annotations.
Results Table 1 presents the MPJPE met- rics of our method and the comparison methods. All methods in Table D are based on single-frame, multiview im- ages of the Human3.6N, dataset, exclud- ing some methods using additional la- beled training data [9 31] and single- view methods 32
Table 1: MPJPE results on the Human3.6e 3.6M dataset fol- lowing Protocol
Method
Supervision
Average
Baseline
2D
69.9
Cross-view fusion
3D
31.2
Table [ shows that our method achieves 1. mm MPJPE than the existing best method. Moreover, our method achieves 28.7 MPJPE with only 5% training data to train the 2D de- tector. The MPJPE of naive triangulation 69. mm, indicating that the erroneous 2D estimates at certain views heavily af- fect the final 3D estimates. Our method achieves significant improvements and even performs better These results demonstrate that the TR loss is capable of
33
Remelli al. 1v/
3D
30.2
Epipolar transformers
2D
26.9
Kristijan al. 10
3D
29.1
MIFT-Transforigg 30
3D
27.5
TR loss (100%)
2D
25.8
TR loss (5%)
2D
28.7
than 3D supervised methods
[13 17 10 30).

---

reducing the negative effect of erroneous views and guiding the view rays converging to the ground truth (as the schematic (c) shows).
4.3 Laboratory mouse pose estimation
Datasets Laboratory mouse pose estimation is key task in neuroscience. However, there is still no high-quality large-scale multiview dataset, demonstrating the need for data-cfficient approach. We conduct experiments
three datasets to evaluate our method: CalMS21, Dannce, and THmouse (THM, a dataset we collected). CalMS21 [ is a large-scale 2D mouse dataset containing 30,000 frames of two interacting mice captured by two cameras from top-view and front-view. Nine joints (top-view) and thirteen joints (front) of each mouse are annotated in each frame. Dannce dataset 35 consists of 1032 labeled frames recorded by six cameras around a single mouse. Our THmouse dataset has 1200 frames from 6 synchronized cameras (1 top-view and 5 around-views). We annotate the 2D ground truth and triangulate the 3D ground truth based 2DD labels. We conduct six pose estimation tasks: CalMS21-DDnnnc, Dannce, and THmousee In intra-dataset experiments, split the training/test train models on the training set. In cross-dataset experiments, we also split the target dataset into training data and test data by 60%/40%. Then we use the 2D ground truth on the source dataset to compute heatmap loss and use the target training data to compute the TR loss. We select the shared joint across datasets to evaluate methods and use MPJPE as the evaluation metric. Details about the joint definition can be found in Appendix A.4.
Dannce-THmouss, T'Hmouse Dannce,
by 60%//0%% and
Results shows shows the MPJPE results mouse pose estimation. The baseline directly triangulates the 3D estimates via SVD based on the 2D detection results of pretrained HRnet. RANSAC is implemented based the Anipose toolkit [36. Both methods above are based the results of pretrained 2D detector without other fine-tuning, so the performance is relatively poor. DeeplabCut [1] the most popular animal pose estimation method. General/rring (10] is stochastic framework for generalizable pose estimation, but relies on the predicted 2D results, so it can not update the 2D detector in end-to-end fashion. The Sup3d" model [9] is finetuned with MSE loss between the 3D estimate and 3D ground truth. TR model even achieves better results on some tasks than Sup3d" even without using the 3D ground truth. TR+DD model adds the domain discrimination loss to reduce domain discrepancy in cross datssedt tasks, slightly improving performance. Training the model with both TR loss and 3D supervision loss (TR+Sup3d") can lead every view ray focus on the 3D ground truth, so it achieves the best MPJPE results for all tasks.
Table 2: MPJPE results on mouse pose estimation. means the method used 3D ground truth during training. The unit is mm.
Dannee
THM Dannce-THM THM-Dannce Calms2- Dannce
Calms2-THM
Baseline
3,25
5.67
10.02
9.48
20.65
22.50
RANSAC
36
2.65
4.96
9.66
8.01
18.39
17.50
DeeplabCut 15
4.20
4.40
11.52
11.43
21.43
17.78
GeneralTriang
2.56
4.65
8,56
8.06
18.55
16.50
Sup3d' 9
2.34
3.45
7.33
7.15
15.50
14.46
TR
2.22
3.49
5.87
7.15
15.16
15.11
TR+DD TR+Sup3d
5.78
5.83
14.84
15.07
2.00
3.37
5.76
5.57
14.84
14.38
To demonstrate the "plug-and-playy ability TR loss, we implement the experiments four tasks by plugging the triangulation head and the confidence head into different 2D detectors (including PVT SCNet(38) and (33). As shown in Table 3 all 2D detectors get lower MPJPE errors with TR loss. Especially for the cross-domain tasks Deannce-THM and THM-Dannce), TR loss reduces the MPJPE by average of 3.85mm (36%).
We also present the MPJPE results of each joint on the Dannce dataset Appendix A.5. The results show that the nose and ears have lower errors than those of other points, while limb positions are difficult to estimate because of the self-occlusion, and body joint error relatively large due to the lack of texture features.
7

---

Table 3: MPJPE results of different 2D detectors with and without TR
PVT
SCNet
MobileNet
w/o TR
w/ TR
emror
w/o TR
w/ TR
ertor
w/o TR
w/ TR
errrr
Dannce
3.12
2.74
-0.38
3.27
2,48
-0.79
3.26
2.81
-0..45
THM
4.57
3.81
-0.76
4.47
3.49
-0.98
4.94
4.04
-0.90
Mannce-THM
7.96
-4.91
11.08
6,44
-4.64
11.15
5.57
-5.58
THM-Danncc
11.09
7.74
-3.35
8.49
5.48
-3.01
9.47
7.85
-1.62
4.4 Further analysis
Data efficiency analysis We compare the MEJPE of models trained with different data sizes and present the results Figure (b). Naive triangulation means triangulating directly based on the 2D estimates of the backbone. Supervised triangulation means training the total model with the ground-truth 3D labels. Our model is trained with TR loss. Figure (a) shows that the MPJPE of all models decreases the training data size increases. Our model performs better than other models in most cases especially when the training data size is small (5%-10%). The cross-subject results in Figure (b) show that our self-supervised method can significantly improve the generalization ability of different individuals, but supervised triangulation fails.
Sensitivity testing the score threshold To avoid the trivial solution, set threshold range the confidence score. Figure (c) shows the MPJPE results curves with different threshold ranges. We set the threshold range be symmetric about 0.5 as shown the x-axis, where "No" means model without score head and (0.0, 1.0) means model without threshold. Models without a score head get degraded performance since they not adjust the weight of each view in the triangulation process, while models without score threshold get significantly large errors since TR loss to assign most views with weight close to 0, in this way, these views have contributions triangulation. Except for these two extreme settings, our method obtains similar results because TR loss and confidence can be adaptively balanced during the training process.
Ablation studies Figure (d) (e) present the ablation experiment results. The baseline model and 3D supervised model share the same model structure the "TR+confidence" model having the confidence head, while the "TR" model does not have the confidence head. The results show that TR
Bxeeène
30
(mm))
HOPE (mm))
MOPE (hmm)
super
100
30
100
15
130
no 10000
(.009)220 6.200 (. yo threshold
training data size (N)
training data size (%5
(a) Subject Subject
(b) Subject 1 Subject 11
(c) Different thresholds
203
372.56
36
27.54
150
TReconf-ercc
MOPPE (mim)
12
TRecuntsence
MPjPE (mmm
223
TR+90
TR+00
15.00
303
#confidence
17.81
14.65
eereidence
15
3.34
3:48
2.22
number cameras
(d) Subject Subject
(c) CalMS21 THM
(f) Different cameras
Figure 3: (a) (b) Data efficiency analysis on Human3.06 6M dataset. (c) Sensitivity testing on the score threshold. (d) (e) Ablation studies. The results show that loss is the most critical component and DD loss is optional. (f) Results on different numbers of cameras.

---

loss is the most critical component, yielding a 77.56% improvement on Subject 1 1-Subject 11 and 46.41 % improvement on the CaIMS21-THM experiment compared the baseline. The confidence head also provides slight improvement due to its learnable weights selecting high-confidence views. The DD loss contributes the least to the performance.
Results on different numbers of cameras Figure (f) shows the results on different numbers of cameras on the Dannce dataset. The error increases slightly as the number of cameras decreases from 6 to 3, and increases sharply when the number of cameras decreases to 2. However, the TR loss achieves consistent improvement for all camera numbers.
Running time analysis In the training process, our method is the same as typical deep learning models that compute the loss functions (including TR loss) only once per batch update the weights. The TR loss does not require additional iterative optimization within a batch. Therefore, TR loss does not excessively increase the training time. During testing, our method also does not require iterative optimization. The experiments show that the average per-hatch running time of the same model with and without TR loss on the Dannce dataset 0.39s and 0.46s, respectively.
4.5 Qualitative results
Figure 4 shows qualitative examples of the Human3.6N M. dataset. The first four columns present the 2D estimates in different views generated by the 2D backbone, while the last column shows the 3D estimates of different methods. The 2D detector is pretrained in the same way as the "Subject 1 Subject 11" experiment. As shown, the 2D backhone can not provide steady 2D estimates because of the domain discrepancy and limited training data. The naive triangulation and supervised method incapable of correcting these errors (such as the right arm and right hip in view 2). However, our method corrects the failed 2D estimates in each view via global geometry consistency self-supervision, providing accurate 3D results.
baseline
supervised
ours
Figure 4: Qualitative results on Human3.61 dataset. Our method provides geometrically consistent 2D estimates in each view and an accurate 3D estimate.
The examples of mouse pose estimation Figure show that our method obtains better 3D estimates. Due to space limitations, we only show three views and the 3D results here. All models in Figure 5 are pretrained on the THM training set. As shown column 3 of Subfigure (a), only our method provides geometrically plausible 2D estimates for limbs and body. Moreover, only our method distinguishes the left paw and right paw in Subfigure (b). More qualitative results and failure examples can be found in Appendix A.6.

---

baselinn
79
Ft
(a) THM dataset example
(b) Dannce dataset example
Figure 5: Qualitative results of mouse pose estimation. Our method obtains accurate results for the hard joint (body) and unseen joints (paws) in View 3.
Conclusion
In this paper, we present TR loss for multiview 3D human/animal pose estimation, which aims to minimize the of distances between 3D estimation and view rays. Its advantages contain (1) Self- supervised. It achieves global geometric consistency without 3D supervision and any reprojection processes. (2) Simple. It only minimizes the smallest singular value of the triangulation matrix and can be easily applied to various 2D backbones; (3) Effective and data efficient. It achieves state-of-the-an MPJPE performance for single-frane 3D pose estimation on the Human3.6M dataset and obtains competitive results with only 5% training data. Future work includes combining TRR loss with end-to-end learnable association methods to generalize multi-object pose estimation, or using TR loss to jointly optimize the camera poses and object poses for moving cameras.
Acknowledge
We thank the reviewers for their constructive comments. This work supported in part by Guoqiang Institute Tsinghua University (No.2021GQU) 202120000,,,)), NSFC (No.61171255), China Postdoctoral Science Foundation under Grant BX20220186, and Shuimu Tsinghua Scholar Program.
References
[1] Qiuxia Lin, Linlin Yang, and Angela Yao. Cross-domain 3d hand pose estimation with dual modalities. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[2] Kerui Gu, Linlin Yang, Michael Mi, and Angela Yao. Bous-comee integral regression for human pose estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-16. 2023.
[3] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3, 3.6:: Large scale datasets and predictive methods for human sensing in natural environments. IEEE Transactions Pattern Analysis and Machine Intelligence, 36(7): 11-11122113333 jul 2014.
[4] Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timo- thy Scott Godisart, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social interaction capture. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[5] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation New benchmark and state of the art analysis. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2014.
10

---

[6] Matthew Loper. Naureen Mahmood, Javier Romero, Gerard Pons-Moll and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1 1-:48::1220 248:166, 16, October 2015.
[7] Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother. Dsac differentiable ransac for camera localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages -400,2500, 2017.
[8] Eric Brachmann and Carsten Rother. Neural-guides ransac: Learning where to sample model hypotheses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4322-4331, 2019.
[9] Karim Iskakov, Egor Burkov, Victor Lempitsky, and Yury Malkov. Learnable triangulation of human pose. pages 7717-7726, 2019.
[10] Kristijan Bartol, David Bojanic, and Tomislav Pribanic. Generalizable Human Pose Triangula- tion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022-June: 1018-11027 2022.
[11] Muhammed Kocabas, Salih Karagoz, and Emre Akbas. Self-supervised learning of 3D human pose using multi-view geometry. P Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019:1077
2019.
[12] Yuan Yao, Yasamin Jafarian, and Hyun Soo Park. Monet: Multiview semi-supervised keypoint detection via epipolar divergence. In Proceedings of the IEEE International Conference on Computer Vision, pages 753-762, 2019.
[13] Haibo Qiu, Chunyu Wang. Jingdong Wang, Naiyan Wang, and Wenjun Zeng. Cross view fusion for 3d human pose estimation. In Proceedings of the IEEE International Conference on Computer Vision, pages 4342-4351, 2019.
[14] Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu. Epipolar transformers. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7779-7788, 2020.
[15] Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe, Venkatesh N. Murthy, Mackenzie Weygandt Mathis, and Matthias Bethge. DecpLabCut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 21(9):121--1289 2018.
[16] Kang Huang, Yaning Han, Ke Chen, Hongli Pan, Gaoyang Zhao, Wenling Yi, Xiaoxi Li, Siyuan Liu, Pengfei Wei, and Liping Wang. A hierarchical 3D-motion learning framework for animal spontancous behavior mapping. Nature Communications,, 12(1), 2021.
[17] Edoardo Remelli, Shangchen Han, Sina Honari, Pascal Fua, and Robert Wang. Lightweight Multi- 3D Pose Estimation through Casttae-Dieettangled Representation. Proceedingy of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1:6039-6048, 2020.
[18) Hanyue Tu, Chunyu Wang, and Wenjun Zeng. Voxelpose: Towards multi-camera 3d human pose estimation in wild environment. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 197-212. Springer, 2020.
[19] Chen Li and Gim Hee Lec. ScarceNet: Animal Pose Estimation with Scarce Annotations. 2023.
[20]
Chen Li and Gim Hee Lee. From synthetic to real: Unsupervised domain adaptation for animal pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1482-1491, 2021.
[21] Jiteng Mu, Weichao Qiu, Gregory D Hager, and Alan L Yuille. Learning from synthetic animals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12386-12395 2020.
[22] Jinkun Cao, Hongyang Tang, Hao-Shu Fang, Xiaoyong Shen, Cewu Lu, and Yu-Wing Tai. Cross- domain adaptation for animal pose estimation. In Proceedings of the IEEF/CVF International Conference on Computer Vision. pages 9498-9507 2019.
[23] Junguang Jiang, Yifei Ji, Ximei Wang, Yufeng Liu, Jianmin Wang, and Mingsheng Long. Regressive Domain Adaptation for Unsupervised Keypoint Detection. Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 6776-6785,
2021.
11

---

[24] Wei Wang. Zheng Dang, Yinlin Hu, Pascal Fua, and Mathieu Salzmann. Robust differentiable svd. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[25] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In ECCV, pages 483-499, 2016.
[26] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In European Conference on Computer Vision (ECCV), 2018.
[27] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adverssrial training of neural networks. The journal of machine learning research, 17(1):099-20330 2016.
[28] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2962-2971, Los Alamitos, CA. USA. jul 2017. IEEE Computer Society.
[29] MMPose Contributors. Openmmlab pose estimation toolbox and benchmark. https://
github. httpb/coooooooo.ccoooooooooooooo open mm hamlal/mmppsee 2020.
[30] Hui Shuai, Lele Wu, and Qingshan Liu. Adaptive multi- and temporal fusing transformer for 3d human pose estimation, IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(4):4122-4133 2023. doi: 10.1 10.1109/PPMMM2000 28822.122221111
[31] N Dinesh Reddy, Laurent Guigues, Leonid Pishchulin, Jayan Eledath, and Srinivasa G. Narasimhan. Tessetrack: End-to-end learnable multi-person articulated 3d pose tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15185-15195, 2021. doi: 10. 109/1VVR44437.222100000
[32] Mohsen Gholami, Bastian Wandt, Helge Rhodin, Rabab Ward, and Z. Jane Wang. AdaptPose: Cross-Dataset Adaptation for 3D Human Pose Estimation by Leamable Motion Generation.
pages 13065-13075, June 2022.
[33] Wentao Zhu, Xiaoxuan Ma. Zhaoyang Liu, Libin Liu, Wayne Wu, and Yizhou Wang. Learning human motion representations: A unified perspective. arXiv preprint arXiv:2210.06551, 2022.
[34] Jennifer J. Sun, Tomomi Karigo, Dipam Chakraborty, Sharada Mohanty, Benjamin Wild, Quan Sun, Chen Chen, David J. Anderson, Pietro Perona, Yisong Yue, and Ann Kennedy. The multi-agent behavior dataset: Mouse dyadic social interactions. 35th Conference on Neural Information Processing Systems (NeurIPS 202 2021.
[35] Timothy W. Dunn, Jesse D. Marshall, Kyle S. Severson, Diego E. Aldarondo, David G.C. Hildebrand, Selmaan N. Chettih, William L. Wang, Amanda J. Gellis, David E. Carlson, Dmitriy Aronov, Winrich A. Freiwald, Fan Wang, and Bence P. Ölveczky. Geometric deep learning enables 3D kinematic profiling across species and environments. Nature Methods, 18 (5):564-573, 2021.
[36] Pierre Karashchuk, Katie L. Rupp, Evyn S. Dickinson, Sarah Walling-Bell BBel, Elischa Sanders, Eiman Azim, Bingni W. Brunton, and John C. Tuthill. Anipose: toolkit for robust markerless 3d pose estimation. Cell Reports, 36(13): (1113)119730, 2021. ISSN 221 12477
[37] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 668-578. 2021.
[38] Jiang-Jiang Liu, Qibin Hou, Ming-Ming Cheng, Changhu Wang, and Jiashi Feng. Improving convolutional networks with self-calibrated convolutions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10096-10105 2020.
[39] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 45 4510-45 2018
12


Reviews:
[{"review_id": "1iaMbqROAR", "reviewer": "Reviewer_4Fs8", "summary": "The paper addresses the problem of predicting human or animal joint positions in 3d from a set of posed images (with calibrated cameras). In order to do so the authors propose to train a neural network that leverages the multi-view constraints between images and predicts the 3d positions in and end-to-end fashion.\n\nThe contribution of the paper is an objective formulation that doesn't rely on explicit 3d ground truth, but can be supervised from 2d ground truth only and is regularized via a loss on the triangulation residual (in image space, not in 3d). The authors demonstrate that the latter can be achieved by minimizing the smallest singular value of the triangulation matrix; that is, the 3d points never need to be computed explicitly for the loss computation and no 3d grund truth is required for training.\n\nExperimental results demonstrate the effectiveness of the approach for human and mice pose estimation.", "questions": "**Q1** Does the number of 3d points / joints to estimate need to be known upfront? Do you aim to predict all joints in each view and how do you handle occlusions?\n\n**Q2** The residual loss in Eq. (9) is formulated for a single 3d point j. How is the loss over all joints formulated? If the optimization is separate per joint, what prevents that two headmap / confidence heads converge onto the same joint?\n\n**Q3** Fig. 3 lists performance numbers for the ablation study and shows that the confidence predictions only have a small influence on the overall prediction performance. Do you have an estimate about the frequency of occlusions per joint in the dataset? I'd imagine that with more occlusions the prediction confidence would become more important in order to be able to triangulate successfully.\n\n*Comment*: In the paragraph starting at line 153 it should read *trivial*, not trial.", "limitations": "The paper is missing a discussion about limitations of the approach. I appears that the proposed triangulation loss is of interest not only for the estimation of human / animal joint positions, but also for structure estimation, e.g. jointly reconstructing a point cloud and learning a local interest point detector. However, it remains unclear if the presented approach is applicable to this use-case and what constraints / limitations need to be considered.\n\nInteresting questions to address would be:  \na) Is there an upper limit on the number of 3d points that can be handled?  \nb) How does the method perform if each 3d point is only observed by a (small) subset of images? Can the confidence estimation handle those cases and can the triangulation loss still be formulated over all cameras (Eq. (5-6))? Line 153 ff. mentions that the learnable weights are forced to be > 0 in order to avoid a trivial solution. Is this still applicable in case of only partial observations?", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**S1** 3D ground truth is expensive and hard to obtain compared to 2d annotations. Therefore the proposed framework and losses (requiring 2d ground truth only) address an important and relevant topic for the greated (3d) vision community. \n\n**S2** Experimental results demonstrate that the proposed method performs better than state-of-the-art and also methods that are trained with 3d ground truth. This is a strong contribution and underlines the importance of the unsupervised multi-view constraint in learning.\n\n**S3** The formulation of the the residual minimization is elegant as it does not require to actually triangulate and backproject points for residual computation. ", "weaknesses": "**W1** Eq. (9) suggests to minimize the smallest singular value of the projection matrix. In order to do so its computation via SVD needs to be differentiable. A discussion on this topic is missing, but is required for the paper to be self consistent. References to related work are also missing, e.g. \"Robust Differentiable SVD\", Wei Wang, Zheng Dang, Yinlin Hu, Pascal Fua, Mathieu Salzmann, TPAMI 2022. \n\n**W2** A discussion about the relation of the minimized algebraic error (the smallest singular value) and the desired geometric error (the residuals in image space) is missing. Though, it would be helpful for the reader to understand the relation between the minimized error of Eq. (5) and Eq. (9).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful and constructive review. To address your questions and concerns:\n\n> Q1:  Eq. (9) suggests to ..., References to related work are also missing, ...\n\nR1: Thanks to the reviewer for recommending \"robust differentiable SVD\". We will add the reference and discuss this topic in our final paper. Robust differentiable SVD addressed the instability problem of eigendecomposition during deep network training. We will try to use it in the future instead of the current Pytorch-AutoDiff-based SVD.\n\n> Q2: A discussion about the relation of the minimized algebraic error and the desired geometric error is missing...\n\nR2: **The algebraic error Eq. (9) is an approximate geometric error Eq. (5).** The derivation from Eq. (6) to Eq. (9) is the standard procedure of the SVD decomposition solution of the least square optimization problem, which has been strictly proved, so we only need to clarify the relation between Eq. (5) to Eq. (6). By using homogeneous coordinates, for an observed point $x(u,v,t)$ and an estimated point $x’(u’,v’,t’)$ projected from estimated 3D point $X’$ using projection matrix $P_c$ (i.e. $x’=P_c \\times X’$), the geometric error penalizes the 2-norm of the geometric error vector and writes $d_{geo}(x,x’) = ||(u/t-u’/t’)^2 + (v/t-v’/t’)||\\_2$, where $t’$ is the estimated depth of point $X’$ from the $c$th view, which is usually different from each view. Therefore, directly minimizing $d_{geo}$ usually involves heavy iterative optimization. The algebraic error penalizes the 2-norm of the algebraic error vector and writes $d_{alg}(x,x’) = ||(vt’-tv’)^2 + (u’t-ut’)||\\_2$, so $d_{alg} = d_{geo} * t * t’$.  We always set $t=1$. By accumulating the errors from all the points from different views, $d_{alg}$ is not proportional to $d_{geo}$ because $t'$ from different views are inequal. Therefore minizing $d_{alg}$ yields a slightly different results. However, $d_{alg}$ can be solved linearly and is more suitable for end-to-end training.\n\n> Q3: Does the number of 3d points/joints to estimate need to be known upfront? Do you aim to predict all joints in each view and how do you handle occlusions?\n\n\nR3: The number of joints needs to be known for the 2D detector since the number of heatmap head layers should be equal to the joint number. However, TR loss itself does not require the number of joints to be known in advance.\n\nOur final goal is to predict the 3D position of each joint. We first simply filter out the obvious occluded 2D keypoints in each view according to the confidence value output of the 2D detector. For those keypoints with high confidence but still under occlusion, our TR loss can enforces them to converge to a position that is cross-view spatially consistent in 3D. \n\n> Q4:  The residual loss ... How is the loss over all joints formulated? ..., what prevents that two headmap / confidence heads converge onto the same joint?\n\nR4: The TR loss over all joints is formulated as the sum of losses for all joints. The predefined one-to-one relationship between the heatmap channel and its target joint prevents two heatmap heads from converging to the same joint. The same 2D heatmap channel estimates the same joint in different views, while two different 2D heatmap channels are defined to predict different joints in the same view. Therefore, there is no need for additional cross-view matching(association).\n\n> Q5: Fig. 3 ... shows that the confidence predictions only have a small influence ...\n\nR5: We agree with the reviewer that prediction confidence would become more important for cases with more occlusions. Inspired by the comments, we checked the occlusion rate in our data. The average occlusion rate of all joints in the THM mouse dataset is about 27\\%, but the occlusion rate of different joints varies a lot (more than 60\\% for feet but less than 5\\% for ears). The Human3.6M is labeled by 3D ground truth reprojection so that even the occluded joint has a 2D label. We will design a more systematic and detailed ablation study on mouse experiments in the revision. \n\n> Q6: The paper is missing a discussion about limitations of the approach...\n\nR6: The limitation of our method is that it requires a 2D detector in giving a relatively accurate initial estimate. This is also a common limitation of 3D pose estimation without 3D supervision.\nWe agree with the reviewer's comment that TR loss may be able to improve some structure estimation methods since it provides better 3D spatial consistency of local points in different views. A potential challenge is that the scene in structure estimation/SFM is much more complex than pose detection, so a better detection model and system-level optimization may be needed to achieve this goal. We will discuss in the future work section.  \n\n> Q7: Is there an upper limit on the number of 3d points that can be handled?\n\nTheoretically, there is no upper limit on the number of 3d points that can be handled because our method calculates each point individually. \n\n> Q8: How does the method perform if each 3d point is only observed by a (small) subset of images? ...\n\nR8: We set a threshold of 0.2 on the confidence of the 2D detector output to filter out invisible points (referring to \"OpenMMLab Pose Estimation Toolbox and Benchmark\"). These points will not be involved in the triangulation process. For other points, we set the learnable triangulation weights to adapt their contributions to triangulation. This weight is not allowed to be 0, because it may lead to a trivial solution, as discussed in L153 of the manuscript.\n\n\n\n\n\n\n\n"}, {"review_id": "rVmARUQqak", "reviewer": "Reviewer_ZQc8", "summary": "The paper proposes an triangular residual loss to optimize for the optimization of the 3D locations of the pose estimation. the iterative optimization framework address the problem of the erroneous predictions of the keypoints using learning based methods. The results are shown on multile pose estimation datasets with different objects like humans and mouses. ", "questions": "what is the training time of the current formulation?\n\nWhat happens if the estimated keypoint from one of the views is erroneous?\n\n", "limitations": "Limitations and societal impact have not been discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper has addressed an fundamental problem in triangulation i.e. is uncertain estimate of the keypoints in 2D and how to optimize for the 3D location given that the 2D locations have uncertainty.\n\nThe iterative triangulation residual is a good formulation to realign the predicted heatmap locations to the accurate point using multi-view consistency. \n\nThe method shows substantial result improvement on multiple datasets. Further they show reasonable improvements using much fewer images. \n\nSuch formulation is easily generalizable and analysis has been provided for the same.", "weaknesses": "Compared to the previous methods using a single loss over the 3D triangulation, in the current framework using iterative formulation causes additional compute and time to optimize. Analyzing the tradeoff in time vs accuracy compared to baselines will be helpful. \n\nExperiments on additional datasets and with different objects should be explored .i.e. using panoptic studio or more datasets with different animals like monkey might show more benefits of the current formulation. \n\nAlthough the method works with 5-6 camera views. analyzing what happens with fewer than that like 2-4 should be analyzed. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful and constructive review. To address your questions and concerns:\n> Q1: Compared to the previous methods using a single loss over the 3D triangulation, in the current framework using iterative formulation causes additional compute and time to optimize. Analyzing the tradeoff in time vs accuracy compared to baselines will be helpful.\n\nR1: In the training process, our method is the same as typical deep learning models that compute the loss functions (including TR loss) only once per batch to update the weights. The TR loss does not require additional iterative optimization within a batch. So TR loss does not excessively increase the training time. During testing, our method also does not require iterative optimization.\nRegarding the training efficiency, we empirically compare the training time of our model with the baseline model. Both models are running under the same conditions (same batch size, same optimizer) on the same device (an NVIDIA 3090 GPU and Intel  i7-11700 CPU). The average per-batch running time of the baseline model and our model on the Dannce dataset are 0.389s and 0.463s. Therefore, the training time efficiency of our method is (0.463-0.389)/0.389=19\\% lower than baseline, but the performance significantly improved by (5.86-3.61)/3.61=62\\%. We will add a more systematic discussion and analysis of training time in the revised paper.\n\n> Q2: Experiments on additional datasets and with different objects should be explored. i.e. using panoptic studio or more datasets with different animals like monkey might show more benefits of the current formulation.\n\nR2: We thank the reviewers for the constructive comments. We apologize for not being able to provide the experimental results within the short deadline of the rebuttal. We will add more quantitative and qualitative experiments on different subjects in the revision.\n\n> Q3: Although the method works with 5-6 camera views. analyzing what happens with fewer than that like 2-4 should be analyzed.\n\nR3: We evaluate the results of different numbers of camera views on the Dannce dataset. As shown in the following table, the accuracy drops significantly when the number of viewpoints is less than 4. However, our TR loss achieves consistent improvement in all cases.\n\n|       num of cams      |   6  |   5  |   4  |   3  |   2   |\n|:---------------:|:----:|:----:|:----:|:----:|:-----:|\n| without TR loss | 5.86 | 6.52 | 6.72 | 9.79 | 24.70 |\n| with TR loss    | 3.61 | 4.41 | 4.49 | 6.78 | 18.04 |\n\n> Q4: what is the training time of the current formulation?\n\nR4: The training time for the laboratory mouse pose estimation experiments is around 30-40 minutes. The training time for human pose estimation experiments is around 3-4 hours.\n\n> Q5: What happens if the estimated keypoint from one of the views is erroneous?\n\nR5: If the 2D estimates on one view are erroneous, the TR loss will enforces it to converge to the point that has 3D consistency with other views. Its triangulation confidence will also be lower so that its negative effect is reduced. Therefore, the proposed method achieves relatively robust performance against erroneous 2D estimates. \n\n> Q6: Limitations and social impact have not been discussed.\n\nR6: The limitation of our method is that it requires a 2D detector in giving a relatively accurate initial estimate. This is also a common limitation of 3D pose estimation without 3D supervision. We present some failure cases in the appendix. We will discuss the limitation of our method in the revised paper. \nRegarding the social impact, our method is a novel solution for non-3d supervision pose estimation for both humans and animals, so it is meaningful for both human application and animal behavior analysis. Our method and results have no negative impact on animal conservation and welfare, human health and safety, and social ethics.\n\n\n\n\n\n"}, {"review_id": "yBjOx6TJFa", "reviewer": "Reviewer_3C6c", "summary": "This paper proposes to perform 3D pose estimation from multi-view RGB images. The key contribution is to develop a new loss function to enable effective training with only 2D pose supervision. This loss function is intended to iteratively optimize the geometric consistency between multi-view rays of each keypoints. The goal is to minimize the distance between the initial 3D estimate and multi-view rays to converge to a stable 3D position. The performance has been evaluated on one 3D human pose dataset, Human3.6M, and multiple mouse pose datasets.", "questions": "Is there any way to ensure the reproducibility of the results in the paper? \nHow does the proposed method perform in the wild?", "limitations": "There are some obvious limitations, which have not been discussed in the paper. \nFor instance, the performance is heavily relying on the initial 2D pose estimation. While the in-the-wild performance of this part has not been validated.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Impressive quantitative results are achieved on multiple benchmarks. Extensive ablation studies testify the effectiveness of key designs. \n2. The idea of developing a iterative loss to alleviate the in-consistency between 2D pose supervision is awesome. \n3. The paper is well written and easy to follow.", "weaknesses": "1. Reproducibility. The results obtained in this paper is very attractive.  The code may not be released. The 5-lines implementation details at Sec. 4.1 clearly are not enough for reproducing the results. There is no guarantee for the reproducibility. \n2. Limited qualitative results are provided to show the performance. More visual results, such as a video, would be very helpful. \n3. Limitations are not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' careful review and constructive comments. To address your questions and concerns:\n\n> Q1: Reproducibility. The results obtained in this paper is very attractive. The code may not be released. The 5-lines implementation details at Sec. 4.1 clearly are not enough for reproducing the results. There is no guarantee for the reproducibility.\n\nR1: We will open-source our code and dataset if our paper can be accepted.** We will also improve the implementation details. Model details:  We apply HRnet as the backbone, build the heatmap head with a convolution layer with a 3x3 kernel size, and build the confidence head with two convolution layers followed by three linear layers [512, 256, num_joint] with a sigmoid activation function. The domain discriminator is an average pooling layer followed by three linear layers [512, 256, num_joint] with a sigmoid activation function. Training details: All models are trained on 1 NVIDIA 3090 GPU and an Intel  i7-11700 CPU with the Adam optimizer and an initial learning rate of 1e−5. We will also add more specific implementation details for each dataset in the appendix. \n\n> Q2: Limited qualitative results are provided to show the performance. More visual results, such as a video, would be very helpful.\n\nR2: We add a video comparing qualitative results w/o TR loss with w/ TR loss on Dannce dataset. TR loss largely improves the visual quality by removing obvious errors such as floating legs/paws or falsely overlaped front paws. \n\n> Q3: Limitations are not discussed.\n\nR3: Our method's limitation is that it relies on a pretrained 2D detector, which is a common drawback in 3D pose estimation without 3D supervision, especially for triangulation-based methods. We have shown some failure cases in the appendix. \n\n> Q4: Is there any way to ensure the reproducibility of the results in the paper? How does the proposed method perform in the wild?\n\nR4: We will release our code and datasets on GitHub as long as our paper is accepted.  As can be expected, our method can be applied in the wild. The 2D keypoint detector and 2D dataset are well-established, especially the human keypoint detector. As the 2D results shown in existing papers and the MMPose package, the 2D detector trained on the COCO dataset shows great in-the-wild performance, which can provide enough initial values. \n\n> Q5: There are some obvious limitations, which have not been discussed in the paper. For instance, the performance is heavily relying on the initial 2D pose estimation. While the in-the-wild performance of this part has not been validated.\n\nR5: Similar to common triangulation-based methods, our method also relies on the initial 2D pose estimation. However, most of the existing triangulation-based methods can not update inaccurate 2D estimates but only reduce their negative impact. On the contrary, our method can optimize the 2D estimates unsupervised. Thanks to the reviewer for the suggestions, we will make this more clear and evaluate our method on the in-the-wild datasets. \n"}, {"review_id": "Dm6OTYqOxM", "reviewer": "Reviewer_y7FT", "summary": "The authors present an approach for 3D pose estimation that leverages multi-view stereo cues. A weak 2D keypoint detector is refined using an unsupervised triangulation consistency from posed cameras (TR loss) which iteratively minimizes triangulation discrepencies. The method is experimentally evaluated on an animal dataset of mice and Human3.6M.", "questions": "* Q1. Given the iterative nature of the MVS triangulation consistency, it is unclear to me how an outlier measurment would affect the procedure. Where is the robustness to single outliers in the process?\n* Q2. What is the reason for the choice of solely self-made baselines? Why are no MVS methods compared? How would other methods work if the TR loss is used (e.g. additionally)?\n* Q3. What would be the influence of camera pose error on the result?\n* Q4. Are all other baselines (H3.6M) trained with the same data amount (100%)? Are they most recent SotA?\n", "limitations": "Failure cases are discussed in appendix.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* S1. Multi-view consistency as refiner. The idea to use multi-view consistency to refine a (potentially weak) 2D keypoint detector with a cross-view loss is interesting.\n* S2. Simple idea with potential. The use of unsupervised consistency loss can lift simpler 2D labels into 3D where annotation is more cumbersome.", "weaknesses": "* W1. Missing Baselines / Backup Experiments. In details:\n   * L16 claims a \"plug-and-play module that enables data-efficient training of all 2D keypoint detectors\". However, the 2D keypoint detectors are not changed in experiments.\n   * The related work talks about [11,12,13,14] as previous MVS setups - and L91 speaks about \"local pariwise consistency\", however, this can be globally optimized. Would it be possible to include a comparison against commonly used reconstruction methods / optimizations such as the one from COLMAP? No competitor baselines are used for comparison such as [11,12,13,14, ...], but only self-made baselines. Why?\n   * The experiments on H3.6M are compared against 3D supervision. The idea of the paper, however, is conceptual. How would these methods/pipelines improve with the TR loss?\n* W2. Quesitonable robustness. Given the iterative nature of the MVS triangulation consistency, it is unclear to me how an outlier measurment would affect the procedure. Where is the robustness to single outliers in the process?\n* W3. Missing mathematical rigorosity. There are a couple of unclarities / mathematical details that should be specified such as:\n   * Eqn (5): It is unclear whihc norm is used? L2? Depending on this, the equivalence of (5) and (6) migh not be given. Would it help to add a discussion to the paper about the minimization of a geometric vs. an algebraic error?\n   * Considerations for non-degenerate / degenerate constraints for eqn (6) are missing. E.g. rank considerations\n   * Around eqn (8). \\sigma_i, u_i, v_i should be defined\n   * L149f sigma not necessary in strict order >= should be used\n   * L152 / eqn (9): What happens if the smallest eigenvalue is not unique?\n   * Eqn (11), L168ff z with hat is missing a definition\n* W4. Minor grammatical errors etc. / typos\n   * L53: colloquial: \"doesn't\"\n   * L87: captial letters \"All\"\n   * L95: \\sigma_4 not defined\n   * L102/L104/L112/L121 notation arrow is typically used to define the mapping (not the sets)\n   * L114: missing space \"(1)and\"\n   * Template: NeurIPS22\n   * Eqn.(1): Consistency on the naming (e.g. not italic)\n   * L117: \"indices\"\n   * L123: \"objective\"?\n   * L131: norm not defined (L1 or L2 used?)\n   * L135: \"Eq.(6)\" missing space\n   * L145: norm definition is missing\n   * Eqn (7): norm not defined (also in eqn (8,9,10))\n   * Colloquial (L209 / L211): doens't, can't\n   * L261: doen't\n   * L270: can't\n\nMost critical points have been addressed during the rebuttal!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors thank the reviewer for constructive and helpful feedback. To address your questions and concerns:\n\n> Q1: L16 claims a \"plug-and-play module ... However, the 2D keypoint detectors are not changed in experiments.\n\nR1: To demonstrate the \"plug-and-play\" ability, we complemented the experiments on the Dannce dataset by plugging our triangulation head and confidence head on different 2D detectors (including PVT [r1], SCNet [r2], and MobileNetV2 [r3]). In the following table, \"baseline\" triangulates the 3D estimates based on the 2D results of each 2D detector, \"baseline+TR\" is finetuned with our TR loss. The MPJPE errors show that our modules and loss functions provide consistent improvements on all 2D detectors. \n\n|    Dannce     | PVT  | SCNet | MobileNet | \n| :-----------: | :--: | :---: | :-------: | \n|   Baseline    | 4.11 | 3.85  |   5.78    | \n| Baseline + TR | 2.94 | 2.86  |   4.33    |  \n\n| THM-Dannce | PVT  | SCNet | MobileNet |  \n| :--------: | :--: | :---: | :-------: |\n|  Baseline  | 9.44 | 9.29  |   11.13   |\n|   Baseline +  TR     | 6.25 | 6.32  |   7.22    |  \n\n[r1] Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions, ICCV, 2021\n\n[r2] Improving Convolutional Networks With Self-Calibrated Convolutions, CVPR, 2020\n\n[r3] MobileNetV2: Inverted Residuals and Linear Bottlenecks, CVPR, 2018. \n\n\n\n> Q2: The related work ..., such as the one from COLMAP? No competitor baselines are used for comparison such as [11,12,13,14, ...], but only self-made baselines. Why?\n\nR2: We add the DeepLabCut[r4] and GeneralTriang[10] as the SotA comparison methods. We chose these methods because DeepLabCut is the most popular animal pose estimation method, and GeneralTriang is the latest SotA pose estimation method proposed in CVPR 2022. GeneralTriang is also based on triangulation. The results are shown in the tables in R1. \n\n[r4] Using DeepLabCut for 3D markerless pose estimation across species and behaviors, Nature Protocols, 2019.\n\n|            | DeepLabCut | GeneralTriang | Ours |\n|:----------:|:----------:|:-------------:|:----:|\n|   Dannce   |    4.20    |      4.15     | 3.54 |\n| THM-Dannce |    11.43   |      7.29     |  5.18    |\n\nWe have not directly compared with COLMAP since its typical implementation of the triangulation is nearly the same as the \"baseline\" and \"RANSAC\" in Table.1 and Table. 2. Specifically, i) ``TriangulateMultiViewPoint`` is the core function for COLMAP triangulation and it uses DLT (the same as our baseline). ii)  According to its document, \"a 3D similarity transformation will be estimated with a RANSAC estimator to be robust to potential outliers in the data.\", and we have compared with RANSAC in Table.1. We will clarify this in the revision.  \n\n> Q3: The experiments on H3.6M ..., How would these methods/pipelines improve with the TR loss?\n\nR3: Validating the effectiveness of the TR Loss under different setups is important. Thus, we have validated the effectiveness of the TR Loss even with 3D supervision in Table.1 on the mouse dataset. Moreover, the validity of the method is also demonstrated when 3D GT labels are unavailable (referring to R1). We will add additional validations on human datasets in the revision to make it more thorough. \n\n\n> Q4:Quesitonable robustness.\n\nR4: In the classical MVS triangulation,  2D outliers are excluded based on the reprojection errors during iterative optimization. However, 2D estimates in each view are not corrected throughout the process (including the outlier). Differently, our method updates the 2D estimates in an end-to-end training framework, and the TR loss forces the 2D outliers in each view to gradually converge to the positions with 3D consistency. (As mentioned by Reviewer ZQc8: \"The iterative triangulation residual is a good formulation to realign the predicted heatmap locations to the accurate point using multi-view consistency\"). We will release our code for reproducibility. \n\n> Q5: Missing mathematical rigorosity.\n\nR5: [1] Eqn(5) uses L2 norm. In fact, Eqn(5) and Eqn(6) are not fully equivalent, and we will tune the claim. Please refer to our answer \"R2\" to reviewer 4Fs8 for detailed discussion. \n\n[2]  To avoid degenerate cases where A is not a full-rank matrix, we simply ignore the TR loss where the condition number of A is too large. \n\n[3] The definition of \\sigma_i, u_i, v_i are presented in L149-L150 actually. We find that it does not follow Eqn (8) closely, which is confusing. We will clarify it.\n\n[4] In L.149, we will use >=. \n\n[5] We simply use the last eigenvalue of SVD. In the used datasets, cameras are arranged reasonably and equal samllest eigenvalue is hard to observe. \n\n[6] We will clarify z hat.  \n\n> Q6: Minor grammatical errors etc. / typos\n\nR6: We will go through the paper carefully with a native speaker.\n\n> Q7: Given the iterative ... Where is the robustness to single outliers in the process?\n\nR7: Please refer to R4.\n\n> Q8: What is the reason for the choice of solely self-made baselines? ...\n\nR8: Please refer to R1 and R2. \n\n> Q9: What would be the influence of camera pose error on the result?\n\nR9: Currently, we assume the camera poses are correct as other related methods. However, the differentiable nature of TR loss may have the potential to correct erroneous camera poses. \n\n> Q10: Are all other baselines (H3.6M) trained with the same data amount (100%)? Are they most recent SotA?\n\nR10: Yes. All the other baselines in Table 3 are trained with 100% data. They are the SotA under the setting of single-frame, multi-view 3D human pose estimation on the Human3.6M dataset without using additional training data. Notice that some papers achieved better results in different settings, for example, STCFormer (CVPR2023) and DiffPose (CVPR2023) used the ground-truth 2D pose as input, Token-Pruned Pose Transformer (ECCV2022), AdaFuse(IJCV2021), TesseTrack(CVPR2021) used the additional training data. We exclude these methods for fair comparision. \n\n\n\n\n\n\n\n\n\n\n\n\n\n"}, {"review_id": "TkplOOAtCD", "reviewer": "Reviewer_iymk", "summary": "This paper focuses on the task of multi-view 3D human/animal pose estimation, and proposes Triangulation Residual Loss, termed as TR Loss, to constraint multi-view geometry consistency in a self-supervised manner. TR Loss is used to minimize the distances between the predicted 3D point to the view rays, and simply implemented as minimizing the smallest singular value of the triangulation matrix. Experiments verify the effectiveness of TR Loss on both laboratory mouse pose estimation and human pose estimation.", "questions": "- In Line158, the weights are scaled to (0.4, 0.6). Does it limit the effect of the predicted confidence? Why the superparameters (0.4 and 0.6) are set? Ablation studies are needed.\n\n- In Table 2, the best results are not bolded.\n\n- In Table 3, it seems that the result of MTFT-Transformer is not consistent with that in their paper. And there are some methods listed in their paper and not compared in this paper, e.g, [1] and [2], which achieve better results. For different ratios of training data (5% and 100%), what are the ratios of unlabeled data?\n\n- In Fig. 3(b), why TR Loss has stronger generalization ability than 3D supervised loss?\n\nReference:\n\n[1] AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild.\n\n[2] Fusing Wearable IMUs with Multi-View Images for Human Pose Estimation: A Geometric Approach.", "limitations": "The authors do not discuss the limitations and potential negative societal impacts of their work in the paper.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- TR Loss is simple yet effective, which self-supervises the multi-view geometry consistency by minimizing the smallest singular value of the triangulation matrix without 3D annotations or heavy computation of reprojection.\n\n- A new THmouse dataset is constructed to help build a benchmark for laboratory mouse pose estimation.\n\n- SOTA MPJPE results on Human3.6M dataset are achieved, which verify the effectiveness of TR Loss. \n", "weaknesses": "- In Fig. 1, there are three groups of methods discussed; however, the quantitative comparisons with those of the second group (cf. Fig. 1b), especially in terms of accuracy and efficiency, are not presented in the experiments.\n\n- For the newly built benchmark of laboratory mouse 3D pose estimation, experiments in Sec. 4.2 seem to be ablation studies. No SOTA methods are directly applied on this benchmark. It’s suggested to provide the results of some recent methods.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: In Fig. 1, there are three groups of methods discussed; however, the quantitative comparisons with those of the second group (cf. Fig. 1b), especially in terms of accuracy and efficiency, are not presented in the experiments. \n\nR1: We chose GeneralTriang[10] as a comparison method here. As the results in the result table in   R2 shown, the GeneralTriang performs worse than our method. It is also less efficient than our method. The average per-batch running time of the baseline model, our method, and GeneralTriang are 0.389s, 0.463s, and 2.05s. GeneralTriang takes a longer time since it needs to select view subsets and update the 3D hypotheses iteratively in each batch. Our method just calculates one more loss function than baseline so the computational time complexity is nearly the same.\n\n\n\n> Q2: For the newly built benchmark of laboratory mouse 3D pose estimation, experiments in Sec. 4.2 seem to be ablation studies. No SOTA methods are directly applied on this benchmark. It’s suggested to provide the results of some recent methods.\n\nR2: We add the DeepLabCut and GeneralTriang[10] as the SotA comparison methods. We chose these methods because DeepLabCut is the most popular animal pose estimation method, and GeneralTriang is the latest SotA pose estimation method proposed in CVPR 2022. The results are shown in the following table. \n\n|            | DeepLabCut | GeneralTriang | Ours |\n|:----------:|:----------:|:-------------:|:----:|\n|   Dannce   |    4.20    |      4.15     | 3.54 |\n| THM-Dannce |    11.43   |      7.29     | 5.18 |\n\n\nIn addition, We also plug our TR loss onto different 2D detectors. including PVT, SCNet, and MobileNetV2 (referring R1 to reviwer y7FT). The MPJPE errors in the following table show that our modules and loss functions provide consistent and improved results on all 2D detectors.\n\n\n\n> Q3: In Line158, the weights are scaled to (0.4, 0.6). Does it limit the effect of the predicted confidence? Why the superparameters (0.4 and 0.6) are set? Ablation studies are needed.\n\nR3: Learnable confidence mitigates the negative effects of inaccurate 2D estimates to triangulation. But in turn, it also affects the updating value of inaccurate 2D estimates. As discussed in L153, if the weights of a view are too small, then its 2D estimate is hardly updated. Theoretically, as long as the weights of each view are not all equal or differ by orders of magnitude (one with 1e-4, one with 9.999), TR loss can obtain a balance between mitigating and updating the inaccurate 2D estimates. We supplemented the sensitivity experiments with confidence thresholds, which showed that setting the parameters at (0.4, 0.6)-(0.1, 0.9) gives approximate results that are significantly better than without confidence and (0,1). \n\n|                | without confidence | (0.4, 0.6) | (0.3, 0.7) | (0.2, 0.8) | (0.1, 0.9) | (0,1) |\n| :------------: | :----------------: | :--------: | :--------: | :--------: | :--------: | :---: |\n| Dannce dataset |        4.54        |    3.61    |    3.74    |    3.60    |    3.90    | 5.89  |\n|   THM-Dannce   |        8.11        |    6.21    |    6.52    |    6.72    |    6.62    | 29.42 |\n\n> Q4: In Table 2, the best results are not bolded.\n\nR4: We thank the reviewers for the careful review and we will update Table 2 in the revised paper.\n\n> Q5: In Table 3, it seems that the result of MTFT-Transformer is not consistent with that in their paper. And there are some methods listed in their paper and not compared in this paper, e.g, [1] and [2], which achieve better results. For different ratios of training data (5% and 100%), what are the ratios of unlabeled data? \n\nR5: Sorry for the mistake, we corrected the performance of the MTFT-Transformer to 27.46mm according to TABLE 9 of the MTFT-Transformer paper. We also checked the results of other compared methods.\n\nReferences [1] and [2] are both outstanding works in human pose estimation, AdaFuse [1] fuses the features of two corresponding points in two different views. ORPSM [2] integrates the IMU signal and multiview images to improve both 2D and 3D pose estimation. We will discuss the difference between them and our manuscript. However, AdaFuse used the MPII dataset as additional data, while ORPSM used the IMU data.\n\nFor the 100\\% training data, we compute the TR loss on 100\\% training data with 2D label and 1\\% test data with only image (without 2D/3D supervision). For the 5\\% training data, we compute the TR loss on 5\\% training data with 2D label and other 95\\% training data with only image (without 2D/3D supervision). We used such a setting because we try to demonstrate TR loss can be used on unlabeled data.  We will add the experiments with only training data.\n\n> Q6: In Fig. 3(b), why TR Loss has stronger generalization ability than 3D supervised loss?\n \nR6: The 3D supervised loss fits the 3D ground truth mainly by weighting the different views, so the 3D supervised loss may be small but the 2D estimates are erroneous. The  3D supervised loss does not have a direct cue to improve the 2D detector. Instead, the TR loss successfully enforces the geometrical consistency by minimizing the sum of distances between 3D estimation and view rays. Therefore, it is a reasonable result that TR loss demonstrates better generalization ability in cross-dataset experiments.\n\n> Q7: Limitations and societal impacts.\n\nR7: The limitation of our method is that it requires a 2D detector in giving a relatively accurate initial estimate. This is also a common limitation of 3D pose estimation without 3D supervision. We present some failure cases in the appendix.\nIn this paper, we propose a solution for non-3d supervision pose estimation for both humans and animals, which has potential applications for both human and animal behavior analysis. As mentioned by other reviewers, our method and results have no negative impact on animal conservation and welfare, human health and safety, and social ethics.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"}]

Author rebuttal:
We thank the reviews for their insightful and valuable comments.
In summary, reviewers are positive about the novelty, formulation, impact, performance, and potential of our method as mentioned:  
"address an important topic for the greated (3d) vision community, ... the formulation is elegant. " (**R-4Fs8-S1&S3**),  "achieve substantial improvement on multiple datasets." (**R-ZQc8-S3**), "conduct extensive ablation studies testify the effectiveness of key designs." (**R-3C6c-S1**), "introduce Simple idea with potential." (**R-y7FT-S2**) and "contribute Simple yet effective idea and a new THmouse dataset for laboratory mouse pose estimation." (**R-iymk-S1&S2**).  

After carefully analyzing all the reviews, major concerns can be summarized:  
1. More rigorous explanation of the mathematical formulation. (**R4Fs8-W1&W2 and Ry7FT-W3**)
2. More in-depth analysis of the robustness. (**R-4Fs8-Q3, R-ZQc8-Q2, R-y7FT-W2&Q1 and R-iymk-Q4**)
3. More thorough comparisons and demonstrations. (**R-3C6c-W2, R-y7FT-W1, R-iymk-W1&W2 and R-ZQc8-W2**)
4. Missing detailed discussion on the limitation. (**R-4Fs8, R-ZQc8, R-3C6c and R-iymk**)
 
We simply summarize the response to each major concern at here.
For more detailed illustrations and experimental results, please refer to the reviewer-specific response letters.
1. For the description of the mathematical formulation, the algebraic error is indeed an approximation of the geometric error in the standard SVD algorithm. We will clarify the description and add a more detailed explanation in the revision. Moreover, although we use Pytorch auto diff in our implementation of the TR Loss, we regard the mentioned "robust differentiable SVD" method as an interesting direction to further improve the performance of the implementation, and we will discuss this method in both the related work and the future work section.
2. For a more in-depth analysis of the robustness, we analyze in detail the robustness of our method under different situations like occlusions, outliers, and partial observations. Please refer to the detailed response to the corresponding questions. Note that the overall improvement of our method mainly comes from the key design: optimizing the 2D keypoint detector with a more "global 3D aware" TR loss, which enhances multi-view 3D consistency in an unsupervised manner.  We also ablate the performance under different view number setups (from 2 to 6), which also demonstrate the effectiveness of our method when compared with other SOTA methods.
3. For more thorough comparisons and demonstrations, we provide an additional video result (will provide more in the revision), ablate the TR Loss with different 2D detectors and also compare with other SOTA mouse 3D pose estimation methods. Due to the time limit, we are not able to add experiments on additional datasets, and with different objects, we will add more experiments in the revision.
4. Missing detailed discussion on the limitation: we have discussed in detail the limitation of our method in the reviewer-specific response letter, and we will clarify this in the revision according to the review, the response, and the failure cases we provide in the appendix.

Meta-review: After discussion between the authors and the reviewers, all reviewers are inclined to accept the paper. The AC agrees. Given the good reviews and discussion, the AC strongly encourages the authors to incorporate all changes from the discussion into the final version of the paper.
Decision: Accept (poster)