Learning Context with Priors for 3D Interacting Hand-Object Pose Estimation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 15:24:59

Abstract: Achieving 3D hand-object pose estimation in interaction scenarios is challenging due to the severe occlusion generated during the interaction. Existing methods address this issue by utilizing the correlation between the hand and object poses as additional cues. They usually first extract the hand and object features from their respective regions and then refine them with each other. However, this paradigm disregards the role of a broad range of image context. To address this problem, we propose a novel and robust approach that learns a broad range of context by imposing priors. First, we build this approach using stacked transformer decoder layers. These layers are required for extracting image-wide context and regional hand or object features by constraining cross-attention operations. We share the context decoder layer parameters between the hand and object pose estimations to avoid interference in the context-learning process. This imposes a prior, indicating that the hand and object are mutually the most important context for each other, significantly enhancing the robustness of obtained context features. Second, since they play different roles, we provide customized feature maps for the context, hand, and object decoder layers. This strategy facilitates the disentanglement of these layers, reducing the feature learning complexity. Finally, we conduct extensive experiments on the popular HO3D and Dex-YCB databases. The experimental results indicate that our method significantly outperforms state-of-the-art approaches and can be applied to other hand pose estimation tasks. The code will be released.

Main paper: 
 introduction: 
 The 3D hand-object pose estimation task simultaneously estimates the hand and object's poses in interaction scenarios. It has been widely applied in augmented and virtual reality [21, 59, 68, 79] , human-computer interaction [48, 57, 73, 86] , robotic manipulation [22] , robot-assisted surgeries [69] , and embodied artificial intelligence [37, 72] . While significant progress has been made in both 3D hand and object pose estimation [4, 5, 14-16, 18, 30, 55, Figure 1 : Illustration of attention maps produced by the context decoder layer in our Learning Context with Priors (LCP) framework. The four columns show the original images, attention maps for hand and object queries, and final pose estimation results. The hand attention maps cover the hand, object, and forearm. Meanwhile, the object attention maps highlight the object and the hand touching the object. 61, 66, 70, 71, 74, 85] , jointly estimating 3D hand-object poses remains challenging. This is primarily due to the mutual occlusion between the interacting hand and object, as illustrated in Fig. 1 . Occlusion leads to information loss and interference, significantly affecting the pose estimation accuracy. It is important to overcome this challenge for applications like robotic manipulation [22] and robot-assisted surgeries [69] , as inaccurate pose estimation may lead to safety issues.Since the hand and object poses are coupled when grasping occurs, their pose correlation is a valuable cue that relieves occlusion. Generally, existing approaches [45, 47] extract the hand and object features from their respective regions and then enhance each other using transformer-like modules [65] . These enhanced features are used to predict 3D hand and object poses independently. Although this paradigm promotes 3D hand-object pose estimation, it overlooks leveraging a broad range of image context, such as the pose correlation between the hand and arm (see Fig. 1 ).However, exploring a broad range of context for this task is challenging, as the model may struggle to identify useful cues in the entire image. Herein, we solve this problem by imposing priors in the context-learning process. First, our proposed approach utilizes the transformer decoder layers [2] . These layers are required for extracting the image-wide context, the fine-grained hand, and the object features by restricting the cross-attention operation scope. The disentangling operation relieves the latter two features from interference outside the hand or object regions. More importantly, we propose sharing the context decoder layer parameters between the 3D hand and object pose estimation tasks. This imposes a strong prior, emphasizing that the interacting hand and object are mutually the most important context for each other. This strategy enables extracting context features more robustly. Moreover, the obtained context features are transferred to the hand and object decoder layers as decoder embeddings. In this way, we fuse robust imagewide context and fine-grained hand or object features for 3D handobject pose estimation purposes.Second, since they play different roles, we provide customized feature maps for the context, hand, and object decoder layers, facilitating disentanglement between them. However, since the context and hand (or object) layers are in series as stacked decoder layers, the key to this approach is to ensure that the feature maps utilized are in similar feature spaces. Accordingly, we adopt the backbone proposed in [45] that not only disentangles the hand and object feature maps but also ensures them are in the same feature space. Then, the two types of feature maps are fed into the hand and object decoder layers, respectively. Furthermore, we propose concatenating the above hand and object feature maps along the channel dimension and then halving the channel number using an efficient 1 × 1 convolution layer. Then, the obtained feature maps are utilized as the value and key for the context decoder layer. The experimentation section shows that this operation significantly enhances the 3D hand-object pose estimation performance.To demonstrate the effectiveness of our approach, we conducted extensive experiments on two widely used databases: HO3D [19] and Dex-YCB [3] . The experimental results validate the effectiveness of each key design and show that our method consistently outperforms state-of-the-art approaches. Moreover, our method can also be applied to the 3D interacting-hands pose estimation, showcasing exceptional performance. 
related work: 
 3D Hand-Object Pose Estimation. Existing methods for this task typically employ a parametric hand model (e.g., MANO [32] ) and assume that the 3D object models are available. This enables them to focus on the 3D hand and object pose prediction. Furthermore, another line of closely related approaches [6-8, 24, 29, 33-35, 41, 63, 77, 78, 83] called 3D hand-object reconstruction exists. This approach does not assume the availability of the 3D object models; instead, it focuses on reconstructing the hand and object meshes. This paper targets the 3D hand-object pose estimation task with a single RGB image adopted as the input data. This section reviews existing research from model architecture, optimization strategy, and training data perspectives.For model architecture, previous approaches [11, 62] adopt a shared encoder and decoder for the hand and object pose estimation. However, they struggle to thoroughly explore the hand and object's unique characteristics. To address this problem, the following works [20, 23, 45, 47] adopt a shared encoder and separate decoders for the two sub-tasks. Given the strong correlation between the poses of an interacting hand and object, recent methods [20, 45, 47] usually utilize the hand and object features to enhance each other. First, they extract the hand and object features from their respective regions. Then, they use them to enhance each other using transformer-like modules [65] . For example, Hampali et al. [20] first detect 2D hand and object keypoints. Then, they extract each keypoint's features. In addition, Liu [47] and Lin et al. [45] extract the hand and object features from their respective bounding boxes. Lin et al. [45] devise a novel backbone model that disentangles the hand and object feature maps and ensures they are in a similar feature space, which facilitates the mutual enhancement between the hand and object features.Model optimization strategies typically utilize physically plausible constraints to refine the hand-object poses, particularly focusing on stable contact between the interacting hand and object. For example, some studies [1, 23] introduce the contact and repulsion loss functions that encourage stable contact and discourage repulsion between the hand-object surfaces. Also, some methods [17, 38, 64, 76] explicitly model the hand-object contact. For instance, Grady et al. [17] and Tse et al. [64] first estimate contact maps between the hand and object; then, the maps are used to optimize the hand poses accordingly. Another noteworthy approach [76] utilizes Contact Potential Fields (CPF) to refine the predicted hand mesh with attractive and repulsive energy terms between the hand-object meshes.The third category of methods addresses the difficulty on data annotation. For example, some studies [1] utilize multiple visual cues, such as the object detection results, 2D hand pose estimation, object instance segmentation, and hand-object contact area, to estimate 3D hand-object poses without supervision. This strategy does not require ground truth annotations. Some studies employ constraints based on temporal consistency to propagate the labels from sparsely annotated frames to the unannotated ones [23] or to enhance the quality of pseudo-labels [47] estimated by one off-theshelf algorithm [47] . Finally, Yang et al. [40] introduce an online data augmentation method that synthesizes training data with free labels to promote data diversity.In this paper, we explore model architecture. Compared to existing research on 3D hand-object pose estimation, we propose a novel approach that robustly leverages both image-wide context and fine-grained regional features. Transformer-based Pose Estimation. Detection Transformer (DETR) [2] and its variants [52, 87] utilize decoder layers and object queries to aggregate image-wide context efficiently for the object detection task. DETR has been applied to various pose estimation tasks, e.g., the 2D and 3D human pose estimation task [9, 12, 31, 39, 46, 51, 75, 80] . For instance, some approaches [39, 46, 51, 75] redefine the 2D human pose estimation task as a regression problem. Unlike traditional heatmap-based methods [60, 67, 84] , they utilize a query to predict one specific human keypoint. Similarly, other approaches [9, 12, 80] employ a query to predict the 3D coordinates of one specific human-body joint or vertex. Considering the increased number of vertices and joints, they design a series of methods to reduce model size and computational cost. For example, Huang et al. [31] use a limited number of queries to predict the pose and shape parameters of the human SMPL [49] model.In contrast to the above human pose estimation tasks, we discovered that aggregating image-wide context for hand-object pose estimation, which is more fine-grained and vulnerable to context inferences, is more challenging. This paper addresses this problem by imposing strong priors in the context-learning process. 
methods: 
 In this section, we first provide an overview to the 3D hand-object pose estimation problem and our Learning Context with Priors (LCP) framework in Section 3.1. Then, we introduce LCP's main This implicitly imposes a prior that the interacting hand and object are mutually the most important context for each other.The hand and object decoder layers extract fine-grained features from their ROI regions, respectively. The context and hand decoder layers share queries for hand pose estimation. Meanwhile, the context and object decoder layers share queries for object pose estimation.components in Section 3.2 and Section 3.3 using the ResNet-50-FPN model as backbone. Finally, we enhance the LCP's capacity using a stronger backbone in Section 3.4.We adopt the popular MANO model [32] M (𝜽, 𝜷) to represent the hand, where 𝜽 ∈ R 16×3 and 𝜷 ∈ R 10 represent the hand pose and shape coefficients, respectively. Meanwhile, the object mesh is assumed available, so we focus on object pose estimation, including the rotation R ∈ 𝑆𝑂 (3) and translation T ∈ R 3 parameters [71] . Similar to existing studies [45, 47] , instead of directly regressing R and T, we first estimate the 2D coordinates of object keypoints. Then, we employ the Perspective-n-Point (PnP) algorithm [36] to calculate the object pose. The goal of 3D hand-object pose estimation is to estimate 𝜽, 𝜷, R, and T, using a single unified framework with an RGB image as input.The overview of LCP is provided in Fig. 2 . We feed a single RGB image I ∈ R 𝐻 ×𝑊 ×3 into the backbone, which consists of a ResNet-50 model [26] and a Feature Pyramid Network (FPN) [44] . The FPN fuses four feature map scales from the ResNet-50 model and outputs F ∈ R 𝐻 /4×𝑊 /4×𝐶 , where 𝐶 denotes the channel number. Beside the backbone, LCP mainly consists of three components: the context, hand, and object decoder layers.The context, hand, and object decoder layers play different roles. They are required to extract the image-wide context, fine-grained hand, and object features, by constraining the scope of cross-attention operations. This disentanglement operation can free the latter two features from interference contained in the context. As illustrated in Fig. 2 , the context and hand (or object) decoder layers are in series as stacked layers. The obtained context features are transferred to the hand and object decoder layers as decoder embeddings. The key is how to obtain the context features in a robust manner for the hand and object, respectively.Naively employing the cross-attention operation to search for imagewide context is suboptimal, as the model may struggle to identify useful cues and be vulnerable to inferences. We address this problem by imposing priors for the hand and object, indicating that the interacting hand and object are mutually the most important context for each other. Moreover, the hand is flexible, and its pose is not only related to the interacting object but also to the wrist and forearm poses. In contrast, the object is rigid; therefore, its pose correlates with that of the hand only when the hand touches the object [45, 47] , as shown in Fig. 1 .Based on the above observation, we propose sharing the context decoder layers for the hand and object pose estimation tasks. This imposes a strong prior that the hand and object are mutually the most important context for each other. Specifically, we provide the context decoder layers with one group of learnable queries Q ℎ ∈ R 𝑁 ℎ ×𝐶 for the hand and another for the object Q 𝑜 ∈ R 𝑁 𝑜 ×𝐶 , where 𝑁 ℎ and 𝑁 𝑜 denote the number of queries for the hand and object, respectively. Then, the context decoder layers transform Q ℎ and Q 𝑜 into a set of decoder embeddings D ℎ 𝑐 ∈ R 𝑁 ℎ ×𝐶 and D 𝑜 𝑐 ∈ R 𝑁 𝑜 ×𝐶 , respectively. The meanings of the hand and object queries are described in Section 3.3. The above process is formulated as follows: where 𝑓 𝑐 (., ., ., .) denotes the stacked context decoder layers. D ℎ 0 and D 𝑜 0 denote the initial decoder embeddings (i.e., zero vectors). D ℎ 𝑐 and D 𝑜 𝑐 represent the output decoder embeddings by the context decoder layers. E pos ∈ R (𝐻 /4×𝑊 /4) ×𝐶 stands for the position encoding. The matrix M 𝑜 is a mask and ⊙ executes the Hadamard Product between M 𝑜 and each channel of F. Its purpose is to constrain the crossattention operation for Q 𝑜 within the object bounding box.EQUATIONEQUATIONThe hand and object decoder layers use D ℎ 𝑐 and D 𝑜 𝑐 as initial decoder embeddings, respectively. Furthermore, these layers continue to employ Q ℎ and Q 𝑜 as queries. Compared to the context decoder layers, they focus on extracting fine-grained features from the hand and object regions, respectively. This is described as follows:EQUATIONEQUATIONwhere 𝑓 ℎ (., ., ., .) and 𝑓 𝑜 (., ., ., .) denote the stacked hand and object decoder layers, respectively. D ℎ 𝑐𝑙 and D 𝑜 𝑐𝑙 stand for the final decoder embeddings for the hand and object pose estimation tasks. They contain both image-wide context features and fine-grained hand or object features. Similar to existing studies [45, 47] , F ℎ ∈ R 32×32×𝐶 and F 𝑜 ∈ R 32×32×𝐶 are obtained from F using the ROIAlign [25] operation, according to the hand and object bounding boxes, respectively. Finally, E ℎ 𝑝𝑜𝑠 and E 𝑜 𝑝𝑜𝑠 represent the positional embeddings for F ℎ and F 𝑜 , respectively.Notably, Q ℎ includes three types of highly correlated hand queries. The queries are utilized to predict the hand pose 𝜽 ∈ R 16×3 and shape 𝜷 ∈ R 10 of the MANO model [32] and 2D hand joint coordinates J ℎ ∈ R 21×2 . Accordingly, we adopt 16 pose queries to predict each of the 3D joint angles in the hand kinematic structure, where each 3D joint angle is represented by three parameters. Furthermore, we utilize an extra query for estimating 𝜷. Additionally, we add 21 more queries, each used to predict the 2D coordinates of one hand joint. These three types of hand queries are adopted in each context and hand decoder layer. The decoder embeddings obtained from these queries exchange information through the self-attention blocks in the decoder layers.In addition, Q 𝑜 incorporates a single type of queries for objects. It includes 21 unique queries, each used to predict the 2D coordinates of a specific object keypoint. The same as [45, 47] , the 21 keypoints include the projection of 8 corner points, 12 midpoints along the edges, and the central point of the 3D object bounding box onto the 2D image plane. Like the hand decoder layers, the decoder embeddings extracted by the 21 object keypoint queries conduct self-attention operations to capture the relationships in object structures.Finally, the decoder embeddings obtained by Q ℎ and Q 𝑜 make predictions via respective feed-forward networks (FFNs). In the training stage, we supervise the outputs of each context, hand, and object decoder layer. In the testing stage, we use the 𝜽 and 𝜷 coefficients predicted by the final hand decoder layer to represent the hand pose. Regarding the object, we follow existing works [45, 47] and adopt the PnP [36] algorithm to calculate the final object pose according to the object keypoints predicted by the final object decoder layer.The LCP model in the above subsections is based on the ResNet-50-FPN backbone. However, our context, hand, and object decoder layers play different roles; therefore, it is more reasonable to provide them with customized feature maps (CFM), facilitating the disentanglement between these layers. Moreover, since the context layer and the hand (or object) layers are stacked in series, it is essential to ensure that these customized feature maps are still in similar feature spaces.We achieve this goal with the help of the backbone proposed in [45] , which disentangles the hand and object feature maps and ensures that they share the same feature space. However, this backbone's output comprises only the hand and object feature maps. The two feature maps are denoted as F ℎ 𝑟𝑎𝑤 ∈ R 𝐻 /4×𝑊 /4×𝐶 and F 𝑜 𝑟𝑎𝑤 ∈ R 𝐻 /4×𝑊 /4×𝐶 , respectively. To integrate this backbone with LCP, we concatenate F ℎ 𝑟𝑎𝑤 and F 𝑜 𝑟𝑎𝑤 along the channel dimension and then halve the channel number by an efficient 1 × 1 convolution layer. The obtained feature maps F 𝑐 ∈ R 𝐻 /4×𝑊 /4×𝐶 are utilized as the value and key for the context decoder layer, while F ℎ and F 𝑜 are obtained using the ROIAlign [25] operation from F ℎ 𝑟𝑎𝑤 and F 𝑜 𝑟𝑎𝑤 , respectively. Finally, we employ the customized feature maps F 𝑐 , F ℎ , and F 𝑜 to our LCP model's context, hand, and object layers, respectively. The above process to obtain the customized feature maps is illustrated in Fig. 3 .The LCP's total loss comprises two parts: one for the hand and another for the object. It can be formulated as follows:EQUATIONThe details of the two parts are as follows:EQUATIONEQUATIONwhere 𝜆 𝜽 , 𝜆 𝜷 , 𝜆 V , 𝜆 J , 𝜆 J ℎ , and 𝜆 J 𝑜 are set as 10, 0.1, 10000, 10000, 250, and 300, respectively. 𝜽 and 𝜷 represent the ground truth MANO [32] pose and shape coefficients, respectively. In addition, V and J denote the ground truth 3D coordinates of the hand vertices and joints. They are obtained according to the MANO [32] model with ground-truth 𝜽 and 𝜷 coefficients. Furthermore, J ℎ and J 𝑜 represent the 2D coordinates of hand joints and object keypoints, respectively. Moreover, . denotes the predicted values. We impose the L2 loss function on 𝜽 , 𝜷, V, and J, and adopt the L1 loss for J ℎ and J 𝑜 . 
experiment 4.1 datasets and metrics: 
 HO3D. The HO3D database [19] We adopt two popular metrics to evaluate the hand pose estimation performance for HO3D and Dex-YCB. Specifically, these metrics include the mean per joint position error with procrustes alignment (PA-MPJPE) and the mean per joint position error without procrustes alignment (MPJPE). MPJPE measures the average Euclidean distance between the predicted coordinate of each hand joint and the ground truth in millimeters (mm), while PA-MPJPE corrects the MPJPE score using procrustes analysis.Similar to existing studies [45, 47] , we evaluate the object pose estimation performance only for the objects viewed during training, using the average distance of model points (ADD) as the metric. The ADD metric [27] assesses whether the average deviation of the predicted model points is within 10% of the object's diameter. Consistent with the approach in [45] , we adopt a symmetric version of the ADD metric, i.e., ADD(-S) [27, 28] , on the Dex-YCB dataset [3] , as some objects in this database are symmetric.All the experiments are conducted using PyTorch [56] and NVIDIA GeForce RTX 3090 GPUs. We consistently utilize the AdamW optimizer [50] for model optimization and set the batch size to 64. The initial learning rate is set to 2e-4. In addition, for experiments performed on the HO3D dataset [19] , we resize the images to 512 × 512 We conduct ablation studies on the large-scale Dex-YCB database and employ ResNet-50-FPN as the backbone unless otherwise specified. CFM refers to customized feature maps.Effectiveness of key components in LCP. The experimental results are summarized in Table 1 . To ensure a fair comparison with LCP, the baseline model contains four hand and four object decoder layers. These layers use F ℎ and F 𝑜 as the value and key in the cross-attention operations, respectively. Then, we replace the first hand and object decoder layers in the baseline model with a context decoder layer, respectively. The two context decoder layers do not share parameters. This experiment is denoted as 'w Context' and significantly reduced the hand's MPJPE by 0.65 mm and PA-MPJPE by 0.16 mm, with a slight performance improvement in the object's ADD(-S) score. This suggests that image-wide context is essential for hand pose estimation. Next, we introduce a mask to the object branch's context decoder layer, constraining the cross-attention operation in Eq. 2 within the object bounding-box area. This experiment is presented as 'w Mask' in Table 1 , and it slightly improves the performance of the object pose estimation, supporting our conjecture that rigid objects may not require extensive contextual information. Furthermore, 'w Sharing' implies sharing the context decoders' parameters with the hand and object. This operation reduces the hand's MPJPE by 0.16 mm and increases the object's ADD(-S) score by 1.3%. These results demonstrate that the interacting hand and object are mutually the most important context for each other.Finally, with the robust CFM backbone, the performance of LCP improves significantly, reducing the hand's MPJPE by 0.69 mm, PA-MPJPE by 0.19 mm, and increasing the object metric by 1.0%. The above experimental results validate the effectiveness of LCP. Ablation study on the context decoder layer. In Table 2 , we compare the performance of LCP's context decoder layer with two potential variants. In the first variant, a mask for Q ℎ is imposed on the context decoder layer, restricting the cross-attention operation within the hand bounding-box, which is similar to the operation defined in Eq.2. In the second variant, we enlarge the cross-attention operation for Q ℎ to cover both the hand and object bounding-box areas.As shown in Table 2 , the performance of both variants is lower than our method, especially on the challenging MPJPE metric. Indeed, without procrustes alignment with the ground-truth, the MPJPE metric requires more context for robust hand pose estimation. These experimental results justify the effectiveness of our designs. Effectiveness of using 2D hand-joint queries. Table 3 compares LCP's performance with and without the queries for detecing 2D hand joints. We observe that using the 21 hand-joint queries substantially improves the hand's PA-MPJPE and MPJPE metrics (i.e., 0.44mm and 1.44mm, respectively). This is because 3D hand pose estimation and 2D hand joint prediction are closely related. It also increases the object's ADD(-S) score by 0.6%. This may be because high-quality hand features provide more accurate context for object pose estimation. Ablation study on the CFM backbone. We compare CFM's performance with two possible variants, as displayed in Table 4 . The two variants' model structures are provided in the supplementary material. The first variant directly utilizes the disentangled hand and object feature maps produced by the original backbone in [45] for the context decoder layer. In other words, the hand and object queries utilize their respective feature maps in the context layer instead of our combined feature maps. The second variant replaces our concatenation operation described in Section 3.4 with simple element-wise addition between the hand and object feature maps.As shown in Table 4 , the performance of the original backbone model in [45] is significantly lower than ours, which may be because the disentangled feature maps in [45] lose the hand or object context. Moreover, the second variant's performance is only slightly lower than ours, meaning that our proposal can be achieved using the simple element-wise addition or concatenation operations. Number of decoder layers. In Table 5 , "Context," "Hand", and "Object" denote the number of context, hand, and object decoder layers, respectively. The table shows that increasing the number However, increasing the number of decoder layers also results in more computational costs. To strike a balance between performance and efficiency, we utilize only one context decoder layer and three hand-object decoder layers in the subsequent experiments.Comparisons on Dex-YCB. Dex-YCB [3] is a recently released database. Table 6 shows that LCP outperforms the state-of-theart 3D hand-object pose estimation methods [45] and surpasses approaches that only perform the hand pose estimation task [55] . Specifically, with the ResNet-50-FPN backbone and the backbone introduced in Section 3.4, LCP's MPJPEs perform better than the state-of-the-art method [45] by 0.06mm and 0.75mm, respectively; and LCP's PA-MPJPEs outperform those of [45] by 0.14mm and 0.33mm, respectively. Table 7 compares LCP's ADD(-S) scores for object pose estimation with those of state-of-the-art methods. Specifically, with the ResNet-50-FPN backbone and the backbone introduced in Section 3.4, LCP's ADD(-S) score significantly outperforms the stateof-the-art method [45] by 19.4% and 20.4%, respectively. This superiority can be attributed to the advantages of employing disentangled decoder layers in our approach. In particular, our method leverages both robust contextual information and fine-grained regional hand or object features. Simultaneously, the self-attention operations within the decoder facilitate learning the dependencies between object keypoints. Consequently, our approach shows stronger robustness on the challenging Dex-YCB dataset [3] compared to HFLNet [45] . Comparisons on HO3D. Table 8 shows that LCP achieves superior performance on hand pose estimation. Specifically, with very similar backbones, LCP † outperforms the recent method [45] by 4.7mm in terms of the MPJPE score. Compared with the PA-MPJPE score, MPJPE measures the mean joint position error without aligning the estimated hand pose with the ground truth; therefore, it may be more practical in real-world applications. We attribute the advantages of LCP † to the exploration of image-wide context, which facilitates holistic estimation of hand poses. Moreover, LCP achieves comparable performance even when compared with approaches that only focus on hand pose estimation (i.e., the "single-hand" approaches), as shown in Table 8 .Table 9 further compares the object pose estimation accuracy between our method and state-of-the-art approaches. Specifically, equipped with similar backbones, the average ADD score of LCP † Finally, since the HO3D dataset is relatively small, overfitting may occur on this database. To cope with this problem, we conduct an additional experiment that is similar to the one in H2ONet [74] . Specifically, we first pre-train LCP † and HFLNet [45] on the largescale Dex-YCB database, respectively. Then, we fine-tune each of them on the HO3D dataset. Experimental results show that with proper pre-training, LCP † outperforms HFLNet with very similar backbones.Comparisons on InterHand2.6M. We extend our approach to the 3D interacting-hands pose estimation task [13, 42, 54, 81, 82] . Specifically, we utilize LCP's hand and object decoder layers to predict the left-and right-hand poses, respectively. In this extension, we remove the masking operation in Eq. 2, enabling the right-hand to search for image-wide context. In Table 10 , we compare LCP's performance with state-of-the-art methods on the InterHand2.6M database [54] . Similar to ACR [81] , we evaluate our method using metrics such as MPJPE and Mean Per Vertex Position Error (MPVPE), as well as their scaled versions denoted as MPJPE-S and MPVPE-S, respectively. Table 10 shows that LCP continues to achieve the best performance across all metrics: MPJPE, MPVPE, MPJPE-S, and MPVPE-S. Moreover, existing 3D interacting-hands pose estimation methods usually do not assume the availability of hand bounding boxes. Therefore, to facilitate fair comparison with these approaches, we adopt the bounding box enclosing all 2D joints of a hand that are predicted by the context decoder layer as the bounding box for this hand. To promote hand joint prediction accuracy, we stack one more context decoder layer and employ the second context layer for hand joint prediction. We denote this LCP variant as LCP ⋄ in Table 10 . Furthermore, Table 10 shows that LCP ⋄ still outperforms state-of-the-art methods. Thus, the experimental results validate LCP's effectiveness.Fig. 4 illustrates the qualitative comparisons between LCP † and the state-of-the-art methods [45] . We compare their performance on images with severe hand-object occlusions. It is shown that LCP † estimates the hand and object poses more robustly. Since LCP † and [45] adopt similar backbones, we attribute the advantages of LCP † to its robust learning capacity of broad-ranged context and fine-grained regional features, facilitating the accurate estimation of hand and object poses. 
conclusion and limitations: 
 This paper explores robustly achieving a broad context range for the 3D hand-object pose estimation task. Our approach stacks disentangled transformer decoder layers to extract image-wide context, hand, and object regional features. By imposing priors to the context decoder layer, our model robustly extracts context for the hand and object, respectively. We also introduce customized feature maps for the three decoder layer types Finally, our approach outperforms existing methods on 3D hand-object and interacting-hands pose estimation. However, this study encounters certain limitations. For example, the utilized LCP queries remain the same for different images. In the future, we will enable adaptive adjustment of each image's queries according to its content to enhance pose estimation accuracy.  
 
 Appendix:  


Decision: This paper initially received 1 reject, but the reviewer revised their opinion. The AC agree with the reviewers' comments.

Reviews: review: paper_summary: paper_summary: To better model hand-object interaction, unlike most existing methods which only use the hand-object correlation as additional cues, the paper proposes to impose priors during the learning process so that the model could take full advantage of both hand and object context information, leading to more robust context features. 

The pipeline contains a context decoder along with hand and object queries for imposing the proposed interaction priors, and then hand and object decoder would be used for respective pose estimation.

The experimental results on HO3D, Dex-YCB are shown superior to previsou methods.
strength_weakness: strength_weakness: - The paper is easy to follow
- The experimental results for hand/object pose estimation show the effectiveness of the proposed pipeline compared with previous methods.
limitations: limitations: - The proposed idea: Using attention layer to capture correlation between hand and object and then condition respective pose decoder on such learned correlation is incremental to the hand-object interaction community. 
- How does the detection performance affect the context learning and final pose estimation?
- Table 9 shows a worse performance on cleanser after pre-training, could the author offer some insights?
- Could the authors provide results showing the generalization aiblity of the proposed method?
- Does MPJPE adopted in the paper mean calculating the error in camera space? If so, as shown in both Table 6 and Table 8, could the authors elaborate more on why MPJPE shows more performance gain than PA-MPJPE?
paper_summary: paper_summary: This paper addresses the challenge of 3D hand-object pose estimation in interaction scenarios, particularly focusing on the severe occlusion that occurs during interactions. Traditional methods leverage the correlation between hand and object poses as additional cues by extracting features from their respective regions and refining them mutually. However, these methods often overlook the importance of broader image context. To tackle this issue, the paper proposes a novel and robust approach that incorporates priors to learn a wide range of contextual information. Specifically, the approach uses stacked transformer decoder layers to extract both image-wide context and regional hand or object features by constraining cross-attention operations. The parameters of the context decoder layers are shared between hand and object pose estimations to avoid interference in the context-learning process. This imposes a prior that the hand and object are mutually the most important contexts for each other, significantly enhancing the robustness of the context features. Additionally, customized feature maps are provided for the context, hand, and object decoder layers, facilitating disentanglement and reducing feature learning complexity.
strength_weakness: strength_weakness: 1. Stacked Transformer Decoder Layers: The approach uses stacked transformer decoder layers to extract image-wide context and fine-grained features by restricting the scope of cross-attention operations, enabling detailed feature extraction.
2. Shared Context Decoder Layer Parameters: By sharing the context decoder layer parameters between hand and object pose estimation tasks, the method imposes a prior that the hand and object are the most important contexts for each other. This enhances the robustness of the extracted context features.
3. Customized Feature Maps: The method provides customized feature maps for the context, hand, and object decoder layers. It utilizes an efficient 1x1 convolution layer to concatenate the hand and object feature maps along the channel dimension and halve the number of channels, significantly improving the performance of 3D hand-object pose estimation.
limitations: limitations: 1. Insufficient Motivation: The motivation for the paper is inadequate. It does not clearly explain the benefits and necessity of using a broad range of image context. Additionally, there are no ablation studies to verify the advantages of incorporating this context.
2. Lack of Novelty: The core idea of the paper lacks originality, with limited innovation. The main modules, such as the stacked transformer decoder layers and context decoder layers, are not original to this paper. The LCP module is merely a targeted improvement based on the work presented in [45].
3. Insufficient Experimental Support: The experimental results are insufficient to support some of the claims made in the paper. There is a lack of comparison with both the latest and most classical methods. Tables 6 and 7 only provide comparisons with the method in [45], which is inadequate. Additionally, it is recommended to include the corresponding computational costs in Figure 5 to support the related claims in the paper.
4. Clarity and Organization: The clarity and organizational structure of the manuscript need improvement, and the overall writing quality should be enhanced. It is suggested to include a clear summary of contributions at the end of the Introduction to help readers understand the key points.
5. Definition of "Impose Priors": The term "impose priors in the context-learning process" is mentioned multiple times without a clear definition. It is recommended to provide a precise explanation of this term.
6. Explanation of Learnable Queries: In line 336, it is not clear how "one group of learnable queries Qh and Qo" is obtained. Please provide an explanation.
7. Definition of Position Encoding: Provide a formula or definition for the position encoding Epos mentioned in line 352.
8. Justification of Operations: In lines 352-355, explain the operations performed and why they are expected to achieve the desired outcomes as stated by the authors.
paper_summary: paper_summary: The authors propose a novel network to for joint hand-object pose estimation from a single image.  To leverage a broad range of the image contexts, they leverage the stacked transformer decoders layers to extract the features. This learned a hand-object interaction prior and reducing the feature learned complexity by disentangle hand and object. Their method is evaluated on the HO3D and DexYCB dataset, and the results significantly outperforms state-of-the-art approaches.
strength_weakness: strength_weakness: 1.	The proposed pipeline utilized the transformer decoder layer to model the hand-object interactions and extract the robust features.
2.	The paper conducts comprehensive evaluation and ablation studies.
limitations: limitations: 1.	Insufficient qualitative attention maps and their comparative analysis. The reasoning behind the network design remains unclear, and the insufficient provision of attention maps makes it challenging to assess the method's effectiveness.
2.	Lin et al. [34] exhibit significantly better performance than LCP in terms of PA-MPJPE on the HO3D dataset. Could the authors explain the reasons behind this observation?
3.	In comparison with Lin et al. [34], what contributes to the improvement in object pose estimation in this method? Given that Lin et al. employ a similar network architecture to LCP, an explanation for the observed enhancement would be valuable.
4.	The authors did not provide information about the running speed.
5.	It would be good to analyze failure cases to better understand the limitations of the proposed system.
paper_summary: paper_summary: Achieving 3D hand-object pose estimation in interaction scenarios is challenging due to severe occlusions that occur during the interaction process. Existing methods address this issue by leveraging the correlation between hand and object poses as additional cues. These methods overlook the role of context and instead use stacked transformer decoder layers to extract image-wide context and regional hand or object features. By sharing the parameters of the context decoder layer between hand and object pose estimation tasks, the prior knowledge that hands and objects are the most important context for each other is reinforced. Since the context, hand, and object decoder layers play different roles, the article provides customized feature maps for these layers to facilitate decoupling, enabling the model to more accurately learn and predict the poses of hands and objects. Finally, experiments were conducted on several different datasets, and the results demonstrate the superiority of this method.
strength_weakness: strength_weakness: 1. By stacking transformer decoder layers and limiting the scope of cross-attention, the method extracts image-wide context as well as regional hand and object features.
2. The robustness of context features is improved by sharing the parameters of the context decoder layer, emphasizing the interrelation between hands and objects.
3. Customized feature maps are provided for the context, hand, and object layers, which helps in decoupling these layers while ensuring they operate within a similar feature space.
4. Experiments on the HO3D and Dex-YCB datasets demonstrate that this method has improved performance compared to other methods.
limitations: limitations: 1. The paper mentions that the LCP framework uses the same queries for different images. This means that the model cannot adaptively adjust the queries based on the specific content of the image, which may affect the accuracy of the pose estimation for certain images. Future work will consider adaptively adjusting the queries according to the image content to improve the accuracy of pose estimation.
2. While the introduction of customized feature maps and the shared context decoder layer improves performance, it also increases the complexity and computational load of the model.
3. Although increasing the number of decoder layers can enhance performance, it also increases computational costs. In practical applications, a trade-off may need to be made between performance and efficiency.
4. The paper does not discuss the real-time performance of the model, which is quite important for many practical application scenarios, such as augmented reality, virtual reality, or real-time robotic manipulation.