# Animal Pose Estimation Based on 3D Priors
## 4 Proposed Method
### Pose Refinement  
#### 4.3.1 Optimization-Based Pose Refinement  
In this paper, we simplify the relationship between 3D and 2D poses as follows:  
\[p\approx\Pi P \tag{3}\]  
where \(p\in\mathbb{R}^{2\times\mathbb{J}}\) and \(P\in\mathbb{R}^{3\times\mathbb{J}}\) denote 2D and 3D poses, respectively. \(\mathbb{J}\) is the number of joints or key points. \(\Pi\) is usually defined based on the weak perspective camera model as  
\[\Pi= \begin{bmatrix}a&0&0\\ 0&a&0\end{bmatrix}  \tag{4}\]  
where \(a\) is a scalar depending on the focal length and the distance to the object [38]. According to Equation (3), 2D pose \(p\) can be obtained if the latent 3D pose \(P\) is accessible. However, estimating a 3D pose from a 2D pose is an ill-posed problem. Fortunately, this pursuit is actually not in vain, as it seems, and attempts in the field of 3D human pose estimation have justified that lifting 2D poses to 3D poses is very fruitful [26; 38]. We assume that the latent 3D pose has a combined representation:  
\[P=\sum_{k=1}^{K}c_{k}R_{k}B_{k} \tag{5}\]  
where \(B_{k}\in\mathbb{R}^{3\times\mathbb{J}}\) for \(k\in[1,K]\) represents a base pose in the learned dictionary. While \(c_{k}\) denotes the weight of each base pose, \(R_{k}\) is a rotation matrix. Then, we substitute Equation (5) into Equation (3) to obtain  
\[p \approx\Pi P=\Pi\sum_{k=1}^{K}c_{k}R_{k}B_{k}=\sum_{k=1}^{K}M_{k}B _{k}\] \[s.t. M_{k}M_{k}^{\mathbb{T}}=c_{k}^{2}\mathbb{I}_{2},\quad k\in[1,K] \tag{6}\]  
where \(M_{k}=c_{k}\bar{R}_{k}\), \(\bar{R}_{k}\) is the first two row vectors of \(R_{k}\), and \(\mathbb{I}_{2}\) is the unit matrix of size \(2\times 2\). Further, we use the following objective function to estimate the latent 3D pose:  
\[\min_{M_{k},M_{k}}\frac{1}{2}\left\|p-\sum_{k=1}^{K}M_{k}B_{k} \right\|_{F}^{2}+\alpha\sum_{k=1}^{K}\left\|M_{k}\right\|_{2}\] \[s.t. M_{k}M_{k}^{\mathbb{T}}=c_{k}^{2}\mathbb{I}_{2},\quad k\in[1,K] \tag{7}\]  
where \(\alpha\) is a predefined coefficient of the regularization. An auxiliary variable \(Z\) is introduced, and Equation (7) is rewritten as  
\[\min_{\widetilde{M},Z}\frac{1}{2}\left\|p-Z\widetilde{B}\right\|_{F}^{2}+ \alpha\sum_{k=1}^{K}\left\|M_{k}\right\|_{2}\]  
\[s.t.\widetilde{M}=Z,\widetilde{M}= \begin{bmatrix}M_{1}\cdots M_{K}\end{bmatrix} ,\widetilde{B}= \begin{bmatrix}B_{1}\\ \vdots\\ B_{K}\end{bmatrix}  \tag{8}\]  
The steps are summarized in Algorithm 2. With \(\{M_{k}\}_{1}^{K}\) we can finally obtain our refined 2D pose \(p^{\prime}\), which is the projection of the estimated latent 3D pose \(P\), i.e.,  
\[p^{\prime}=\sum_{k=1}^{K}M_{k}B_{k} \tag{9}\]```
0:\(p,\alpha\)
0:\(M_{1},\cdots,M_{K}\)
1: initialize \(Z=Y=0,\mu>0\);
2:while not converged do
3:for\(k=1\) to \(K\)do
4:\(Q_{k}^{t}=Z_{k}^{t}-\dfrac{1}{\mu}Y_{k}^{t}\)
5:\(M_{k}^{t}=\mathcal{D}_{\frac{\mu}{\mu}}(Q_{k}^{t})\)
6:\(Z^{t+1}=(p\widetilde{B}^{T}+\mu\widetilde{M}^{t+1}+Y^{t})(\widetilde{B} \widetilde{B}^{T}+\mu I)^{-1}\)
7:\(Y^{t+1}=Y^{t}+\mu(\widetilde{M}^{t+1}-Z^{t+1})\)
8: where \(\mathcal{D}_{\frac{\mu}{\mu}}\) is proximal operator [38]. ```

**Algorithm 2** Pose Refinement

#### 4.3.2 Deep-Learning-Based Pose Refinement

We provide a deep-learning-based implementation, in which both the coefficients \(c_{k}\) and the rotation matrix \(R_{k}\) in Equation (6) are estimated using a neural network. Figure 5 shows the pipelines. The proposed network consists of the coefficient network and camera network. Given the initial pose \(p\) as network input, the coefficient network and camera network estimate coefficients \(c_{k}\) and rotation matrix \(R_{k}\) in Equation (6), respectively. The coefficients \(c_{k}\) and the rotation matrix \(R_{k}\) estimated via the network are combined with the learned 3D pose dictionary \(B\) to generate the 3D pose \(P\) corresponding to the initial pose \(p\). Finally, the 3D pose \(P\) is reprojected to obtain the refined pose \(p^{\prime}\). Network Architecture.