Title: Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D Hand Reconstruction

Mamba

State Space Model
Abstract: 3D Hand reconstruction from a single RGB image is challenging due to the articulated motion, self-occlusion, and interaction with objects. Existing SOTA methods employ attention-based transformers to learn the 3D hand pose and shape, yet they do not fully achieve robust and accurate performance, primarily due to inefficiently modeling spatial relations between joints. To address this problem, we propose a novel graph-guided Mamba framework, named Hamba, which bridges graph learning and state space modeling. Our core idea is to reformulate Mamba's scanning into graph-guided bidirectional scanning for 3D reconstruction using a few effective tokens. This enables us to efficiently learn the spatial relationships between joints for improving reconstruction performance. Specifically, we design a Graph-guided State Space (GSS) block that learns the graph-structured relations and spatial sequences of joints and uses 88.5\% fewer tokens than attention-based methods. Additionally, we integrate the state space features and the global features using a fusion module. By utilizing the GSS block and the fusion module, Hamba effectively leverages the graph-guided state space features and jointly considers global and local features to improve performance. Experiments on several benchmarks and in-the-wild tests demonstrate that Hamba significantly outperforms existing SOTAs, achieving the PA-MPVPE of 5.3mm and F@15mm of 0.992 on FreiHAND. At the time of this paper's acceptance, Hamba holds the top position, Rank 1, in two competition leaderboards on 3D hand reconstruction.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Hamba, a novel technique for reconstructing 3D hand models from a single RGB image. This technique addresses the limitations of previous transformer-based methods, which struggle with occlusion, truncation, and capturing the intricate spatial relationships between hand joints. Hamba combines graph learning with Mamba state space modeling to create a Graph-guided State Space (GSS) block. This block effectively learns the structured relationships among hand joints and leverages both local and global features through a fusion module. The proposed framework utilizes significantly fewer tokens than traditional methods, improving both efficiency and precision.\n\nHamba has demonstrated its effectiveness through extensive benchmarking, outperforming state-of-the-art methods on metrics such as PA-MPVPE and F@15mm on the FreiHAND benchmark. Additionally, the approach promises scalability and adaptability, as the GSS block can be integrated into other tasks, indicating potential applications beyond hand modeling.\n\nSTRENGTHS:\n1.\tThe proposed network structure effectively improves prediction accuracy, as evidenced by results on the FreiHAND and HO3D leaderboards.\n\n2.\tThe experiments related to accuracy in hand pose estimation are thorough and detailed.\n\n3.\tThe method is easy to follow.\n\n4.\tThe implementation code is provided in the supplementary materials.\n\nWEAKNESSES:\n1.\tThe authors emphasize the efficiency of their method, but the paper includes only accuracy-related experiments. There are no ablation studies to demonstrate the claimed efficiency.\n\n2.\tThe authors assert that their method can serve as a “plug-and-play module for other tasks.” However, the paper only includes experiments related to hand pose estimation. The authors should at least attempt to apply their module to full-body pose estimation.\n\n3. The paper contains several typographical errors, such as the citation error on Line 92.\n\nQUESTIONS:\nMy questions based on the weaknesses are:\n\n1.\tCan the authors provide evidence of their method’s efficiency in terms of inference time and GPU memory usage?\n\n2.\tCan the authors attempt to transfer their module to full-body pose estimation or other scenarios to validate its plug-and-play capability?\n\nLIMITATIONS:\nThe authors discuss limitations and neglect the social impacts in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Hamba, a Mamba-based framework for single-view 3D hand reconstruction. Its main contribution is to introduce a graph-guided bidirectional scanning mechanism to fully exploit the joint relations and spatial sequences for accurate hand reconstruction. It additionally fuses global spatial tokens with local graph-based features to further improve the performance. The proposed state space block uses 88.5% less tokens than the attention-based methods, while achieving new state-of-the-art hand reconstruction accuracy on various benchmarks for single-image 3D hand reconstruction.\n\nSTRENGTHS:\n**(1) Technical novelty.**\nThis paper proposes a quite novel image feature extractor that is particularly effective for hand reconstruction; it modifies the Mamba framework to further exploit the graph-based joint relations. To the best of my knowledge, this is the first work to demonstrate that visual Mamba can be effective for image-based articulated shape reconstruction task. I wonder if this method works well for, e.g., human body reconstruction as well.\n\n**(2) Strong experimental results.**\nThe proposed method achieves strong experimental results across various widely-used hand reconstruction benchmarks. The comparisons are also done against the competitive baselines that are very recently proposed (e.g., HaMeR [57]).\n\n**(3) Good presentation.**\nOverall, the paper is well organized and easy to read. The figures are also clearly presented. \n\nAdditionally, I appreciate the authors for also submitting the code for the reproducibility of the proposed method.\n\nWEAKNESSES:\n**(1) Ablation study with Transformer + GCN.**\nI’ve found that the motivation for using Mamba and GCN - to capture both long-range dependencies and graph-based local dependencies - is very similar to the motivation for Graformer [R1, R2], where Transformer (instead of Mamba) and GCN are used together. Additional ablation study with Graformer-like architecture (e.g., replacing the state space block with self-attention block in the proposed architecture) would be informative. \n\n[R1] Gong *et al.*, DiffPose: Toward More Reliable 3D Pose Estimation, In CVPR, 2023.\n\n[R2] Zhao *et al.*, Graformer: Graph-oriented transformer for 3d pose estimation, In CVPR, 2022.\n\nQUESTIONS:\n**(1) Justification for the intermediate 3D reconstruction.**\nTo obtain the 2D joint positions used for image feature extraction, the model intermediately performs the full 3D reconstruction via regressing MANO parameters (Equation 4). I am not sure why this is less overhead compared to employing off-the-shelf 2D joint detectors (lines 151-152).\n\nLIMITATIONS:\nAuthors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach for 3D hand reconstruction from a single view. The main idea is to introduce a graph-guided Mamba framework in the model for hand reconstruction, by bridging graph learning and state space modeling. Building on top of the recent Hamer approach, the final proposed model, Hamba is evaluated on various datasets where it demonstrates strong performance.\n\nSTRENGTHS:\n+ The quantitative results show improvements over the baselines.\n\nWEAKNESSES:\n- I am not sure I follow the motivation of the paper. We read in the paper that \"existing [..] methods fail to capture the semantic relations between different joints\" (Ln 4) which is mentioned again later as \"lack of understanding the semantic relations between hand joints\" (Ln 38) or \"applying attention to all tokens does not fully utilize the joint spatial sequences, resulting in an inaccurate 3D hand mesh in real-world scenarios\" (Ln 41). I am not sure how the proposed approach improves over these observed weaknesses. I can see some quantitative improvements, but I do not think that we get enough support for these arguments, i.e., how Hamba improves on these issues, more specifically over the baseline approach (Hamer [57]).\n\n- The improvements over the baseline are consistent, but relatively minor, particularly on the HInt dataset.\n\n- Based on the training details of the supplementary, it looks like the method is starting from the Hamer checkpoint and finetunes it for 170k iterations. What is the performance of the Hamer model if it also finetuned for 170k iterations? It would be interesting and fair to see this comparison as well.\n\n- There are very few qualitative comparisons with the baseline Hamer model.\n\nQUESTIONS:\n- Why there are no results on the Ego4D subset of HInt?\n\n- The paper mentions that the GSS block uses 88.5% less tokens. How does that affect the final system in terms of runtime, number of parameters and FLOPs, etc.\n\n- Could you clarify their motivation that I discuss in the weaknesses?\n\n- Is it possible to see the additional results of the Hamer model when finetuned the same way that Hamba is finetuned?\n\nLIMITATIONS:\nI think the discussion on limitations could be longer. The paper shows a few failure cases, but the limitation section reads like an afterthought. It could be extended with more of the observed failure cases, as well as other limitation/weaknesses (potentially runtime? number of parameters? reliance on a hand detector? etc).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}