Towards Labeling-free Fine-grained Animal Pose Estimation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 17:25:23

Abstract: In this paper, we are interested in identifying denser and finer animals joints. The lack of standardized joint definitions across various APE datasets, e.g., AnimalPose with 20 joints, AP-10k with 17 joints, and TigDog with 19 joints, presents a significant challenge yet offers an opportunity to fully utilize annotation data. This paper challenges this new non-standardized annotation problem, aiming to learn fine-grained (e.g., 24 or more joints) pose estimators in datasets that lack complete annotations. To combat the unannotated joints, we propose FreeNet, comprising a base network and an adaptation network connected through a circuit feedback learning paradigm. FreeNet enhances the adaptation network's tolerance to unannotated joints via body part-aware learning, optimizing the sampling frequency of joints based on joint detection difficulty, and improves the base network's predictions for unannotated joints using feedback learning. This leverages the cognitive differences of the adaptation network between non-standardized labeled and large-scale unlabeled data. Experimental results on three non-standard datasets demonstrate the effectiveness of our method for fine-grained APE.

Main paper: 
 related work 2.1 datasets for animal pose estimation: 
 Deep architectures have made remarkable progress in human pose estimation thanks to the availability of large-scale, high-quality annotated datasets like MPII [1] and COCO [15] . When it comes to animal pose estimation, early works have established datasets for specific animals such as horses [4] , dogs [3] , and tigers [19] . However, models trained on these datasets have poor generalization ability. To reduce the need for extensive human labor, some researchers have explored using CAD models to generate synthetic animal images for pose estimation [12, 19, 23] . However, these synthetic images have limited variations in environmental conditions and viewpoints, which leads to a significant performance gap when adapting to real-world animal images. To bridge this gap, several large-scale datasets including AP-10k [30] , APT-36k [29] , Animal-Pose [4] , and Animal Kingdom [20] have been introduced recently.The AP-10k and APT-36k datasets were established by the same research group, with APT-36k focusing on video-based animal pose tracking. The Animal Kingdom dataset provides multiple annotated tasks to facilitate understanding animal behavior. However, there is no consensus on the annotation standards, and the definition of animal joints differs across datasets, as shown in Table 1 . For instance, the AP-10k dataset includes 54 animal species with 17 joints, while the AnimalPose dataset covers only 5 species but with 20 joints. The lack of standardization in defining animal poses brings a challenge yet opportunity in fully exploiting annotation data. Pose estimation can be viewed as a regression of heatmaps, and ideas such as bottom-up and top-down approaches for human pose estimation can also be adapted for animals [10] . However, due to the significant differences between humans and animals, the development of APE methods heavily relies on the availability of APE datasets. When only specific animal pose data is available, the focus is on transferring specific domain knowledge to a more generalized animal context [12, 19, 23] . However, a considerable gap still exists, especially when dealing with unseen animal species [14] . To address cross-species generation for APE, D-Gen [14] enhances animal pose estimation by breaking the inconsistent relations among joints while preserving the consistent ones. Synthetic animal images generated from CAD are then used to facilitate APE by reducing the domain gap. CC-SSL [19] employs three consistency criteria to constrain both spatial and temporal to general pseudo labels. To correct noisy pseudo labels, MDAM [12] gradually updates the pseudo labels to prevent network overfitting. However, synthesizing realistic images is limited due to environmental and viewpoint variations. To reduce human labor, ScarceNet [13] learns APE with scarce but full annotations. In contrast, Our FreeNet focuses on learning from several non-standard datasets to achieve fine-grained animal pose estimation with zero full annotation data.Annotating animal images can be labor-intensive, error-prone, and hard to maintain semantic consistency for joints. To address this problem, semi-supervised learning with unlabeled data can be feasible. General image classification methods such as PL [11] and noisy student [27] generate pseudo labels for unlabeled data from model predictions. UDA [24] improves semi-supervised learning by incorporating data augmentation [5] to limit the invariance of model predictions to input noise. FixMatch [24] simplifies the learning process by training the model with high-confidence pseudo labels.MPL [22] enables the teacher network to adjust based on students' performance feedback on labeled data, which improves pseudo labels. GPGML [28] proposes inexact supervised meta-learning to use coarse-grained labels of training samples to reduce the need for labeled data. To mitigate the negative effect of unannotated joints while facilitating the learning of shared joints, we combine PL with body part-aware sampling and feedback learning. D 𝑙 = {D 𝑙 1 ∪ D 𝑙 2 ... ∪ D 𝑙 𝑀 }. The labeled data D 𝑙 forms a joint sample space X 𝑙 × Y *𝑙 , where X 𝑙 is the image space and Y * 𝑙 is the corresponding label space with partial annotations (i.e., some joints are missing annotations). Specifically, the 𝑖-thlabeled dataset D 𝑙 𝑖 = 𝑥 𝑙 𝑖 , 𝑦 * 𝑙 𝑖 comprises 𝑁 𝑖 instance samples.Here, 𝑁 J 𝑖 denotes the total number of joints for all samples, and J 𝑖 refers to the definition of joints. By combining these 𝑀 nonstandard datasets, we obtain animal poses characterized by a richer set of semantic joints, denoted by J 𝑙 = {J 1 ∪ J 2 ...∪ J 𝑀 }, where J 𝑠 𝑖 represents the shared joints, and J 𝑒 𝑖 denotes the exclusive joints unique to the labeled dataset, e.g., chin for TigDog and wither for AnimalPose. Therefore, the unannotated joints for D 𝑙 𝑖 can be represented by J 𝑙 \ (J 𝑠 𝑖 ∪ J 𝑒 𝑖 ). We propose the FreeNet framework, which leverages several non-standard datasets and unlabeled data to address the unannotated joints. 
our framework. freenet consists of an adaptation network (a): 
 and a base network (B), both utilizing the same network architecture (e.g., HRNet [25] ) but with independent weights. This bi-model training helps combat unannotated joints by leveraging the different abilities of the two models. We learn FreeNet by training the adaptation network and the base network sequentially in each generation. The training objective for the adaptation network is denoted byL A = L 𝑢 ,where L 𝑢 guides to learn body part-aware features from pseudolabels. Meanwhile, the overall training objective for the base network is:L B = L 𝑠 + L 𝑓 ,where L 𝑠 supervises learning of prior features from non-standard datasets, and the feedback loss L 𝑓 refines the base network through feedback learning. Figure 3 illustrates the FreeNet pipeline, with non-standard labeled and selected unlabeled data serve as inputs on the left. We avoid merging non-standard and pseudo labels in a single training pool to prevent poor performance and convergence failure due to their label distribution disparity (see Figure 4 ). At each generation, the base network generates pseudo labels for the unlabeled data D𝑢 = {(𝑥 𝑢 , ŷ𝑢 )}. The adaptation network then uses a selected pseudo-labeled subset Û ⊂ D𝑢 (achieved by applying body part-aware sampling) for training. The base network learns prior knowledge from several non-standard labeled data. With feedback learning, the base network uses another selected pseudolabeled subset Ŝ𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 ⊂ D𝑢 (i.e., using a threshold 𝛼 𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 ) ... to generate better pseudo-joints, especially for unannotated joints, therefore improving the performance of the adaptation network. Ultimately, the adaptation network outperforms the base network in the circuit feedback paradigm. During the inference stage, only the adaptation network is used to estimate animal poses with denser joints. The following parts will detail the different learning modules and loss computation.Joints in different animal bodies often exhibit varying learning difficulties and can adversely hurt performance if used directly for training. To address this, FreeNet dynamically selects pseudo-joints from unlabeled data to facilitate body part-aware learning in the adaptation network. We first decompose the joints into three animal body parts: the head, frontal body, and back body. Based on their confidence scores determined by the base network, we rank the joints for each part and select the most reliable for learning. We intentionally avoid using joints with small losses for training, as relying on a "small loss" criterion [9] can lead to a skewed distribution of joints, which hinders learning a good adaptation network.Our body part-aware sampling strategy determines the sampling frequency of joints based on joint detection difficulty. It prioritizes hard-to-detect joints by sampling these more frequently than those easy-to-detect joints, as shown in Figure 3(b) .The proportion of joints to be selected for each body part is determined by thresholds 𝛾 ℎ𝑒𝑎𝑑 , 𝛾 𝑓 𝑟𝑜𝑛𝑡𝑎𝑙 , and 𝛾 𝑏𝑎𝑐𝑘 . For instance, a head joint will be selected if its corresponding confidence score falls within the 𝛾 ℎ𝑒𝑎𝑑 proportion of all sorted confidence scores for head joints. The selected pseudo-joints corresponding to the head, frontal body and back body are included in Û to update the adaptation network using L 𝑢 :L 𝑢 = 𝑁 𝐽 𝑘=1 Ĥ k u ∈ Û 1 L 𝑘 𝑢 𝑁 𝐽 𝑘=1 Ĥ k u ∈ Û 1 ,whereEQUATIONĤ k u represents the pseudo heatmap for the 𝑘-th joint, which can be derived from soft prediction B (x u ; 𝜃 B ) in two steps: 1) extracting joints from soft prediction with highest confidence; 2) applying 2D Gaussian centered on each joint location with a standard deviation of 1 pixel. L 𝑘 𝑢 is the corresponding pseudo label based loss for the 𝑘-th joint. {condition} 1 is a condition function, which outputs 1 when the condition is true and 0 otherwise.The base network is trained with non-standard labeled datasets. We apply a joint combination to transform partially annotated labeled spaces, Y * 𝑙 , into fully annotated ones, Ỹ𝑙 . In this transformation, we use 0 to represent joints that lack annotations, resulting in a comprehensive labeled dataset D𝑙 . Specifically, for each joint set 𝐽 𝑖 , 𝑖 ∪ 𝐽 𝑒 𝑖 ) ∥. We use ground truth heatmaps, 𝐻 𝑙 , derived from Ỹ𝑙 , as targets for training the base network. The training process aims to minimize the mean square error (MSE) between the predicted heatmaps and these target heatmaps, which can be expressed as:EQUATION)where 𝑀 is the number of non-standard datasets and 𝑀 ≥ 2.To combat the unannotated joints, we propose feedback learning that aims to improve the base network's ability to predict pseudojoints, especially those that are unannotated. The intuition behind the base network update is the relationship between the "new" adaptation network on non-standard labeled data and the "old" adaptation network on unlabeled data. Essentially, the adaptation network gauges the cognitive differences between labeled and unlabeled data to update the base network as feedback. If the gradients of two networks have the same direction, the base network is updated in the current direction. In contrast, if the gradients of two networks have different directions, the base network is punished using the adverse direction of current gradients. As the base network is improved to produce accurate predictions for unannotated joints, this strategy can help narrow the gap between non-standard labeled and large-scale unlabeled data. Using more accurate pseudo labels will further improve the performance of the adaptation network for APE.The base network is trained on non-standard labeled data that includes unannotated joints, whereas the adaptation network is trained using unlabeled data with fully pseudo-labels. In view of this, we use the confidence score rankings of pseudo-joints from both networks as the selection criterion. The threshold 𝛼 𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 = (𝑎%, 𝑏%) determines the proportion of pseudo-joints selected for Ŝ𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 . The criteria are as follows,(1) Pseudo-joints ranked between 𝑎% and 𝑏% by either the base network or the adaptation network are included in Ŝ𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 . (2) Pseudo-joints with conflicting ranking between the two networks are also selected in Ŝ𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 . Specifically, we focus on those with ranking falling smaller than 𝑎% or greater than 𝑏% in either network. This selection process is rational because pseudo-joints confirmed as high-confidence by two networks do not require further updates. Conversely, pseudo-joints identified as low-confidence by both networks are excluded from feedback learning to avoid performance degradation. Once Ŝ𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 is established, the feedback loss is calculated as the dot product of two terms:L 𝑓 =𝑓 • 𝑁 𝐽 𝑘=1 Ĥ k u ∈ Ŝ𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 1 L 𝑘 𝑓 𝑁 𝐽 𝑘=1 Ĥ k u ∈ Ŝ𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 1 ,whereL 𝑘 𝑓 = Ĥ k u -(B (I u ; 𝜃 B )) 𝑘 2 .(EQUATIONThe first term 𝑓 is the feedback coefficient that determines the direction and strength of the update; the second term is the loss of the base network on selected unlabeled data Ŝ𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 . L 𝑘 𝑓 is the corresponding loss for the 𝑘-th joint. Specifically, the feedback coefficient 𝑓 is defined as:EQUATIONwhere 𝑓 is calculated as a dot product of two terms: the gradients of the "new" adaptation network on non-standard labeled data and the gradients of the "old" adaptation network on large-scale unlabeled data. The sign of 𝑓 will determine the direction of the update, while the absolute value of 𝑓 will determine its strength. The adaptation network uses selected pseudo-joints data to update the parameters to A (𝑡 +1) . In particular, we approximate it with the parameters obtained from A (𝑡 ) by updating the base network parameters on (x u , Ĥu ), i.e., 𝜃(𝑡 +1) A = 𝜃 (𝑡 ) A -𝜂 A ∇ 𝜃 A MSE( Ĥu , A (x u ; 𝜃 A )).We listed detailed step-by-step pseudo-code for FreeNet in Algorithm 1. FreeNet learns fine-grained pose estimators by extracting rich APE knowledge from non-standard labeled and large-scale unlabeled data. This is achieved by body part-aware learning and a circuit feedback paradigm. Each generation of FreeNet involves the following steps: 1) Select unlabeled images from Û based on whether the pseudo-joint confidence score ranking of a specific animal body part meets the sampling thresholds: (𝛾 ℎ𝑒𝑎𝑑 ,𝛾 𝑓 𝑟𝑜𝑛𝑡𝑎𝑙 ,𝛾 𝑏𝑎𝑐𝑘 ). 2) The adaptation network is first updated in line 10 by minimizing the unsupervised loss L 𝑢 , on selected unlabeled data Û. This facilitates knowledge transfer from the base network B to the adaptation network A through the generation of pseudo ground truths, conditioned on body part-aware sampling. 3) The base network is then updated in line 22 with two losses: the supervised loss L 𝑠 (line 13) and the feedback loss L 𝑓 (line 18). The two losses guide the learning process of the base network, as illustrated in line 15 and line 20, respectively. Specifically, an unlabeled image is selected from Ŝ𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 to trigger the feedback loss L 𝑓 when a pseudo-joint is moderately ranked based on the confidence score of entire joints in the batch. This selection process improves the base network's predictions for unannotated joints. By this design, FreeNet enables the base network and adaptation network to continuously enhance and complement each other. 
experiments 4.1 datasets and implementation details: 
 4.1.1 Datasets. Our experiments are conducted on three widely used benchmark datasets for fine-grained APE. AP-10k [30] is a large-scale benchmark that contains 10,015 labeled images across 54 species, each annotated with 17 joints. These images are divided into train, validation, and test sets with a ratio of 7:1:2 for each species. AnimalPose [4] contains 4,666 labeled images from 5 animal species: cat, dog, sheep, cow, and horse, annotated with 20 joints. TigDog includes 6,523 tiger images and 8,380 horse images, each annotated with 19 joints. We provide a detailed joint definition for each dataset in Table 1 . 𝑥 𝑙 , 𝐻 𝑙 ← SampleMiniBatch( D𝑙 ) 5:x u ← SampleMiniBatch(D 𝑢 ) 6: Ĥu ← Forward(x u , 𝜃 (𝑡 ) B ) 7:Obtain Û by using thresholds (𝛾 ℎ𝑒𝑎𝑑 ,𝛾 𝑓 𝑟𝑜𝑛𝑡𝑎𝑙 ,𝛾 𝑏𝑎𝑐𝑘 ) 8:if 𝑥 𝑢 ∈ Û then Obtain Ŝ𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 by using threshold 𝛼 𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 4.1.2 Evaluation metrics. We adopt the Object Keypoint Similarity (OKS) as the evaluation metric and represent the mean Average Precision (mAP) across OKS=0.50,0.55,...0.90,0.95 following [25] . Additionally, we include the Percentage of Correct Keypoints (PCK) metric [19] , which quantifies the percentage of joints accurately predicted within a normalized distance from the ground truths.To evaluate the precision of additional learned joints in fine-grained animal poses, we consider two settings: 1) using synthetic datasets derived from AP-10k and 2) employing a combination of real datasets. The synthetic datasets are generated by segmenting AP-10k into subsets, each containing an equal number of images but varying joint definitions to mimic the non-standard annotation problem in real-world scenarios. During testing, we utilize AP-10k's ground-truth joints to independently evaluate the accuracy of the unannotated and shared joints, as well as their accuracy gap. In the case of the real dataset combination, we report results for 10% of the two combined datasets (AP-10k and AnimalPose) and 10% of the three combined datasets (AP-10k, Ani-malPose, and TigDog) to demonstrate the efficacy of our FreeNet in terms of body part-aware learning and feedback learning. The combination of real datasets is used as the experimental protocol in the ablation study.All images are augmented using random scaling, rotation, horizontal flip, and a half-body mask and then resized to 256×256 pixels. More details are provided in the supplementary material. The size of output heatmaps is 64×64 pixels. We use HRNet-w32, trained on non-standard labeled data for 210 epochs, as the default backbone for both the base and adaptation networks. The learning rates are first initialized (i.e., 1e-5 for the base network and 1e-3 for the adaptation network) and further decayed with a cosine annealing strategy. Our batch size is 64, comprising 8 labeled and 56 unlabeled samples. Unless otherwise stated, the default thresholds (𝛾 ℎ𝑒𝑎𝑑 , 𝛾 𝑓 𝑟𝑜𝑛𝑡𝑎𝑙 , 𝛾 𝑏𝑎𝑐𝑘 ) for body part-aware sampling are set at (0.9, 0.75, 0.6). The default feedback learning threshold 𝛼 𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 is (20%, 80%). We set different training steps for different settings to fully train the model: 30,000 for AP-10k, 60,000 for the 10% combination of two datasets, and 60,000 for the 10% combination of three datasets. It is trained end-to-end with two NVIDIA V100 GPUs.We compare our method with four state-of-the-art semi-supervised learning methods, including UDA [24] , FixMatch [24] , MPL [22] and ScarceNet [13] . We re-implemented UDA, FixMatch, and MPL using open-source repositories originally developed for classification tasks. These methods are selected as they align well with our aim of leveraging unlabeled data to improve performance when labeled data is limited. Additionally, we include ScarceNet, a SOTA method for APE with scarce annotations, using its available source code for training and evaluation.Given the recent success of ScarceNet in APE with limited data, we evaluate our method in the ScarceNet setting on the synthetic dataset of AP-10k. The 𝛾 ℎ𝑒𝑎𝑑 , 𝛾 𝑓 𝑟𝑜𝑛𝑡𝑎𝑙 , and 𝛾 𝑏𝑎𝑐𝑘 thresholds for the body part-aware sampling are 0.5, 0.6, and 0.65, respectively. The feedback learning threshold 𝛼 𝑓 𝑒𝑒𝑑𝑏𝑎𝑐𝑘 is (20%, 80%). Table 2 shows the comparison results with state-of-art methods. We focus on estimating animal poses from datasets with scarce, non-standardized annotations, specifically under the "25 images per species" setting (including 1250 images and 1710 instances), where ScarceNet achieves the best accuracy. However, this setting remains unfair to other comparison methods and our approach, since we do not assume scarce labels in the problem set, and these images are tailored to favor the performance of ScarceNet. Our three synthesized non-standard datasets contain the head, frontal body, and back body as exclusive joints, respectively, with "neck" and "root of tail" as shared joints. The total number of joints is equals to that in the "5 images per species" setting used by ScarceNet. All compared methods have the same inference time of 0.023 seconds per image as they have the same backbone model with 28.5M parameters. In Table 2 , The results present two findings: 1) Non-standard datasets pose more significant challenges than scarce data with complete annotations, as most methods underperform the "5 images per species" setting. 2) Our method outperforms all other methods, demonstrating the effectiveness of FreeNet in handling both scarce and non-standardized annotations.To verify the effectiveness of our FreeNet for non-standardized annotations, we report mAP and PCK@0.05 accuracy on two synthetic datasets from AP-10k in Table 3 . Specifically, we use the accuracy as a function of the size of the shared joints, ranging from 5, 9 to 17. The two synthetic datasets, AP-10k-subA and AP-10k-subB, have different exclusive joints. AP-10k-subA has exclusive joints from the frontal body of animals, while AP-10k-subB has exclusive joints from the back body of animals. 1) "5" shared joints include the left/right eye, nose, neck, and root of the tail, which coarsely reflect the animal's length. 2) "9" shared joints expand this by adding the left/right frontal paw and left/right back paw, which coarsely constrains the bounding box of the animal. 3) "17" shared joints represent the full annotation of AP-10k for supervised learning, setting an upper bound performance. The results demonstrate that FreeNet consistently outperforms SL in mAP and PCK@0.05 metrics. Moreover, FreeNet achieves more balanced results between shared and unannotated joints, leading to a smaller accuracy gap. Specifically, in the "5" shared joint setting, FreeNet shows a smaller accuracy gap of 8.7 compared to the SL method's 10.4. Similarly, in the "9" shared joint setting, FreeNet maintains a smaller accuracy gap of 7.5 compared to the SL method's 8.46. body parts exhibit different learning difficulties. Moreover, using the traditional "small loss" criterion in our method is also inappropriate as it tends to exaggerate the imbalanced number of joints, resulting in decreased mAP and PCK@0.05 accuracy. For instance, settings of 𝛾 ℎ𝑒𝑎𝑑 = 90%, 𝛾 𝑓 𝑟𝑜𝑛𝑡𝑎𝑙 = 75%, and 𝛾 𝑏𝑎𝑐𝑘 = 60% underperform compared to the setting using all pseudo-joints. In contrast, our body part-aware learning module effectively balances the learning difficulty of joints in different parts. Table 5 shows that we achieve more balanced accuracy (6% std in PCK@0.05) across different animal body parts and also yield higher APE accuracy by sampling 50% of head joints, 60% of frontal body joints, and 65% of back body joints. It's worth noting that further tuning these empirical sampling thresholds could lead to even higher accuracy.Proper learning of unannotated joints is essential to improve fine-grained APE. Table 6 shows the impact of using different proportions of pseudo-joints in feedback learning. Notably, employing pseudo-joints with confidence scores ranking from 20% to 80% yields the highest accuracy. Another interesting finding is that excluding joints with the lowest confidence scores does not affect the performance, as (0,100%) and (0,80%) thresholds produce similar results. However, discarding joints with the highest confidence scores, from (0,80%) to (20%,80%), can enhance network learning and increase performance. This is because these excluded joints are already well-detected and do not need further feedback learning, as illustrated in Figure 7 (a). In contrast, our feedback learning at (20%,80%) significantly improves the prediction confidence for unannotated joints by training more steps as illustrated in Figure 7 (b). These results are consistent with the quantitative results in Table 6 and demonstrate our feedback learning under (20%,80%) can better learn unannotated joints.We use our FreeNet to identity denser and finer joints in real-world applications where there is no consensus on APE annotation standards. Specifically, we expanded two non-standard datasets by adding a 10% TigDog dataset for training. As illustrated in We believe that FreeNet provides a general framework that can leverage arbitrary nonstandard annotation data for other pose estimation related tasks (e.g., human/hand pose estimation) where consistent high-quality data are lacking. 
limitations & conclusion: 
 Our approach achieves impressive performance in identifying denser joints even when only a few non-standard annotations are available. However, there are still several challenges remaining for finegrained APE. The network sometimes has difficulty generalizing well to some unannotated joints for certain species (e.g., tigers) that only appear in one dataset, since animal species vary greatly across datasets (see Figure 6 (b)). We leave this for further research.In conclusion, this paper tackles the emerging and significant challenge of non-standardized annotations in APE. Our solution, FreeNet, aims to fully utilize these annotations from multiple nonstandard datasets to learn denser joints at no additional labeling cost. Specifically, we address the dilemma of unannotated joints and facilitate the learning of joints shared across datasets. Experimental results on synthetic and combined real datasets demonstrate the effectiveness of FreeNet, which is useful in real-world applications. 
: 
 handle species that do not share certain similarities with trained animals. Some methods [12, 19] utilize CAD models to generate synthetic animal images and labeled joints for APE. However, these models often face a significant domain gap due to the limited variation in environmental conditions and viewpoints.Existing studies [20, 29] circumvent this domain gap by utilizing a general APE dataset with massively increased scale and animal diversity (in Figure 1 (a)). For example, AnimalPose [4] contains 4,666 images, AP-10k [30] contains 10,015 images, and TigDog [6] contains 14,093 images. However, collecting large-scale datasets with precise full annotation costs a large effort (e.g., 20 seconds for one joint even using AI-assisted annotation [16] ). Another feasible way is leveraging large-scale unlabeled data to alleviate the need for fully annotated labeled data. For example, ScarceNet [13] employs strategies such as reliable pseudo-label selection and reusable sample re-labeling to combat the noisy pseudo labels for APE. Although the annotation effort is getting smaller (in Figure 1(b) ), full data annotation is still mandatory. This raises a question: for achieving fine-grained animal pose (denser joints), do we need to re-annotate existing datasets with full annotations? To answer this question, we focus on how to learn fine-grained animal poses from non-standard labeled data with zero full annotation as illustrated in Figure 1(c) .This non-standardized annotation problem is a new challenge in fine-grained APE. Nevertheless, it is significant and valuable in practical applications as there is no consensus on annotation standards across existing APE datasets. Specifically, AnimalPose [4] with 20 joints, AP-10k [30] with 17 joints, and TigDog [6] with 19 joints, presents a unique challenge and opportunity in fully utilizing annotation data. However, existing works rely on full annotated data and neglect to utilize multiple non-standard annotation data for APE. In this paper, we aim to address the non-standardized annotation problem. As illustrated in two samples of Figure 2 , the shared joints are grouped into three body parts, namely the head, frontal body, and back body, and the exclusive joints of each dataset (marked in green) are grouped as "unshared joints". We mark the unannotated joints in red. Figure 2 (a) demonstrates the joint relationship across different datasets, which provides two key insights. First, we observe shared joints in different animal body parts exhibit different learning difficulties due to dataset gaps and considerable species variation. This accuracy difference between head joints and back body joints is close to 10%. Second, due to lacking full annotated data, the unshared joints across different datasets have the smallest average number and the worst accuracy, aggravating the skewed joint distribution in the combined training data.Intuitively, different classifiers can produce different decision boundaries and have different learning abilities. To combat the unannotated joints, we chose bi-model training to leverage the different abilities of the two models in complementary ways. Specifically, we propose FreeNet, which consists of a base network and an adaptation network, aiming at fine-grained APE with free full annotation labels. FreeNet enhances the adaptation network's tolerance to unannotated joints via body part-aware learning and improves the base network's predictions for unannotated joints via feedback learning. Body part-aware learning is proposed to mitigate different learning difficulties of animal bodies, and the core is part-aware sampling, ensuring that easy-to-detect joints (i.e., the head joints) are sampled less and hard-to-detect joints (i.e., the back body joints) are sampled more. Our circuit feedback learning paradigm connects the base network and adaptation network and informs the base network how good the pseudo labels of unannotated joints are by using the cognitive difference of the adaptation network between non-standard labeled data (i.e., with unannotated data) and large-scale unlabeled data. As a result, the training error caused by those joints is adaptively corrected.Two training pools, non-standard labeled and unlabeled data, are utilized to train the two networks. We refer to each fully-trained base and adaptation mode as a generation. At each generation, 1) FreeNet dynamically selects pseudo-joints from unlabeled data to facilitate body part-aware learning of the adaptation network. The joints are divided into three parts based on different animal bodies: the head, frontal body, and back body. For each part, we rank the joints based on the confidence scores produced by the base network and select the most confident joints for learning. Our body partaware sampling determines the percentage of joints selected for each part. 2) FreeNet adopts a circuit feedback mechanism to refine the base network's predictions for unannotated joints, leveraging the cognitive differences of the adaptation network. If the cognitive difference between non-standard labeled and selected unlabeled data is significant, the base network is penalized using the adverse direction of current gradients. The selection of unlabeled data is guided by the confidence score rankings of pseudo-joints from both networks to ensure unannotated joints are further improved during feedback learning.To sum up, this paper makes the following contributions:• We address the non-standardized annotation problem, a new and significant challenge in fine-grained animal pose estimation, which is useful in real-world applications due to the lack of consensus on APE annotation standards. • We propose FreeNet, a general framework to combat the unannotated joints via meta-optimization. FreeNet enhances the adaptation network's tolerance to unannotated joints through body part-aware learning and uses feedback learning to improve the base network's predictions of these joints. • We achieve state-of-the-performance for fine-grained APE on non-standard datasets. Furthermore, we can learn finegrained pose estimators without requiring full annotations. 
 
 Appendix:  


Decision: This paper initially received a mix of reviews. However, after the rebuttal, reviewers upgraded their review and agreed to accept the paper. The AC thoroughly considered the paper, reviews, rebuttal, and subsequent discussions, ultimately deciding to accept it. The authors are encouraged to incorporate the reviewers' suggestions to enhance the final version.

Reviews: review: paper_summary: paper_summary: The main idea of this paper is to address the challenge of non-standardized annotations in fine-grained animal pose estimation (APE). The authors propose a novel framework called FreeNet to tackle the problem of estimating denser and finer animal joints, especially when complete annotations are lacking.
strength_weakness: strength_weakness: 1. Non-Standardized Annotation Problem:
   - The paper identifies and tackles the challenge of non-standardized joint definitions across APE datasets, which is significant for real-world applications where consistent annotations are lacking.
2. FreeNet Framework:
   - Structure: Composed of a base network and an adaptation network linked through a circuit feedback learning paradigm.
   - Body Part-Aware Learning:This technique optimizes sampling frequency based on joint detection difficulty, improving the adaptation network’s tolerance to unannotated joints.
   - Feedback Learning: Enhances the base network's predictions for unannotated joints by leveraging cognitive differences between labeled and unlabeled data.
3. Meta-Optimization for Unannotated Joints:
   - FreeNet utilizes meta-optimization to select pseudo-joints from unlabeled data dynamically.
   - The method ensures effective learning by sampling joints based on their detection difficulty and confidence scores.
limitations: limitations: 1.	The paper lacks fluency in expression and clarity in logic, which need further improvement.
2.	The paper contains few formulae, and the explanations of the existing ones are insufficient.
3.	The paper frequently mentions pseudo-joints and pseudo-label, but their specific forms are not clearly defined. Could you provide further clarification?
4.	In Figure 3, the framework for FreeNet is overly simplistic. It would be beneficial to provide a more detailed diagram of the specific network.
5.	There are issues with the experimental design. In section 4.1.3, what are the selection criteria for the non-standard labeled data? Are there targeted experiments or risks of overfitting? What would be the effects of using different datasets or combinations?
paper_summary: paper_summary: In this paper, the authors address the challenge of non-standardized joint definitions in various Animal Pose Estimation (APE) datasets. They propose FreeNet, a method that aims to learn fine-grained pose estimators (with 24 or more joints) in datasets lacking complete annotations.
strength_weakness: strength_weakness: - This paper addresses the challenge of non-standardized annotation in fine-grained animal pose estimation, crucial for real-world applications due to the lack of consensus on APE annotation standards.
- This paper introduce FreeNet, a framework that enhances the adaptation network's tolerance to unannotated joints and improves the base network's predictions through meta-optimization and feedback learning.
limitations: limitations: - Will different categories of keypoints cause a long-tail effect?
- Define the missing keypoints as 0, how is the gradient for this part propagated?
- The network can predict denser joints, but do they all possess the properties of keypoints? For example, the top-most and bottom-most keypoints on the wolf's neck in Figure 6(a) – what do these points represent?
paper_summary: paper_summary: This work presents a new framework called FreeNet that addresses the challenges of animal pose estimation (APE) using datasets with non-standardized annotations and without requiring full annotation. FreeNet utilizes a base network and an adaptation network, enhancing tolerance to unannotated joints through body part-aware learning and improving prediction accuracy with feedback learning. This approach is tested across multiple non-standard datasets, demonstrating its ability to efficiently learn fine-grained animal poses.
strength_weakness: strength_weakness: - FreeNet introduces a unique strategy combining body part-aware learning and feedback learning to handle unannotated and non-standardized data effectively. 
-  By leveraging datasets with partial annotations and enhancing learning through its dual-network architecture, FreeNet reduces reliance on fully annotated datasets. 
-  The method has been shown to improve the accuracy of pose estimation in non-standard datasets by addressing the annotation inconsistencies and exploiting the available data more comprehensively.
limitations: limitations: - The starting point is good and the authors are trying to solve a new problem, which is interesting. I coarsely understand the problem setting of this work. However, the statements about the specific method are not clear. For example, I do not understand the training process or inference process according to Figure 3. It is the best way for the authors could clearly state the specific steps such as i) ii) iii) balabala... 
-  Another question is about the feedback learning. How do you update theta given data D in T steps? Is such an update simple to execute? The authors also show the distillation process in Figure 3 (c), which confused me as to why such a process is a distillation process. 
-  The paper could also benefit from a more detailed discussion on scenarios where FreeNet underperforms, including potential limitations in handling extremely diverse or rare species not represented in the training data. 
-  Figure 6 is not clear to recognize which one is better. The authors could enlarge the image size or highlight the differences. 
-  Table 2 shows comparisons with SOTA methods. It is better to show the model complexity for a fair comparison.