BiDRN: Binarized 3D Whole-body Human Mesh Recovery

Authors: Zhiteng Li, Yulun Zhang, Jing Lin, Haotong Qin, Jinjin Gu, Xin Yuan, Linghe Kong, Xiaokang Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D whole-body human mesh recovery, Binarization

Abstract:
3D whole-body human mesh recovery aims to reconstruct the 3D human body, face, and hands from a single image. Although powerful deep learning models have achieved accurate estimation in this task, they require enormous memory and computational resources. Consequently, these methods can hardly be deployed on resource-limited edge devices. In this work, we propose a Binarized Dual Residual Network (BiDRN), a novel quantization method designed to estimate the 3D human body, face, and hands parameters efficiently. Specifically, we design a basic unit Binarized Dual Residual Block (BiDRB) composed of Local Convolution Residual (LCR) and Block Residual (BR), which can preserve as much full-precision information as possible. For LCR, we further generalize it to four kinds of convolutional modules so that full-precision information can be propagated even across mismatched dimensions when reshaping features. Additionally, we also binarize the face and hands box-prediction network as Binarized BoxNet, which further reduces the model redundancy. Comprehensive quantitative and qualitative experiments demonstrate the effectiveness of BiDRN, which has a significant improvement over state-of-the-art binarization algorithms. Moreover, our BiDRN achieves comparable performance with the full-precision method Hand4Whole while using only **22.1%** parameters and **14.8%** operations. We will release all the code and pretrained models.

Decision: reject
Reviewer scores: 5, 5, 3, 8

Reviews and discussion:
<thread>
creation date:
1729991214913

thread_type:
official_review

<thread_object>
creator: reviewer_zufd

<content>
summary:
In this work, the authors focus on the 3D HMR (human mesh recovery) task. They propose a method named BiDRN based on a Binarized Dual Residual Network. They propose to use a binarized boxnet followed by three BiDRNs for the face, hand, and body, respectively. These features are then projected to SMPL-X parameters to animate the human object. As suggested by the authors, they are the first to propose a binarized neural network for the HMR task. The authors have conducted extensive comparisons on EHF and AGORA benchmarks and compared both binarized and non-binarized methods.

strengths:
+ The proposal of BiDRN is one of the first implementations of a Binarized Neural Network for the HMR task. The motivation is solid (due to rendering time requirements for VR), and the paper is well-written and easy to follow.
+ The authors have conducted comprehensive comparisons with other methods, including both BNN-based and non-BNN-based methods. They show significant improvement over other BNN-based methods, while performance is close to float32 networks, with a huge reduction in computational requirements.
+ The overall improvement is validated with extensive ablation results.

weaknesses:
This work is interesting, and I have no major concerns. For further improvement, here are some suggestions:

+ For the results shown in Table 1, please consider including more recent BNN-based methods. Currently, the latest work in the table is from 2021, which is three years old. Including more recent work would make the comparison more reasonable, as the BNN domain has advanced in recent years (as stated in the related works).
 + As BNNs significantly reduce computational requirements, it would be helpful to check the FPS rate for a video for the BNN network compared to the original network, which could better illustrate whether the BNN-based model can achieve real-time HMR.

questions:
If authors are able to provide the FPS rate for the reconstruction for a video, along with the resolution of the frames and computational resources, it can be very helpful to further validate how fast BNN can achieve for HMR task.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730640469042

thread_type:
official_review

<thread_object>
creator: reviewer_ywt4

<content>
summary:
This paper investigates the binarized network for whole-body 3D human mesh recovery. The motivation is to reduce the high computational cost of existing multi-stage pipeline, so that the model can efficiently run on mobile devices. While directly binarizing the backbone causes severe degradation on precision of mesh regression. Built on the baseline, Hand4Whole, this paper redesigns the architecture of backbone, bounding box detector, and decoding heads to reduce the computational costs while alleviating the reduction in precision.  The experiments are conducted on EHF and AGORA to testify the effectiveness of the proposed designs, compared with other common designs. Compared to the previous binarization design, ReCU, the proposed method reduces 17.1% and 4.5% on EHF and AGORA in terms of whole-body MPJPE, meanwhile, reducing the computational costs to 27.8% of the original Params and 15.6%  of the original OPs.

strengths:
1.	This paper addresses the underexplored challenge of enhancing computational efficiency in whole-body 3D human mesh recovery.
2.	Unlike previous binarization methods, the proposed approach mitigates degradation in mesh regression to an obvious extent.
3.	The reduction in model parameters and operations is substantial, making this method valuable for deployment on mobile devices.

weaknesses:
This paper introduces a binarization model based on the state-of-the-art multi-stage method, Hand4Whole. The experiments and binarization techniques are tailored specifically to optimize this baseline. However, it remains unclear whether the proposed binarization approach is compatible with more efficient one-stage methods, such as MultiHMR or AiOS, which may offer greater potential for faster performance on mobile devices. If the proposed binarization method cannot be applied to other whole-body HMR methods, its novelty and contribution to the field would be limited.

The challenges are described as engineer problems, such as the quality of backbone features or dimension mismatching. The motivation behind all the binarization designs hasn’t been discussed in an obvious way.

questions:
The scientific value of the proposed method might be determined by whether the proposed method can be generalized to the other HMR methods, especially the one-stage methods, AiOS / MultiHMR, which adopts a very different architecture.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699336910

thread_type:
official_review

<thread_object>
creator: reviewer_kuyg

<content>
summary:
This paper introduces a binarized network for 3D whole-body recovery from images. The core design of the proposed network is a binarized module consisting of (i) the local convolution residual with hardtanh pre-activation to alleviate binarization failures, and (ii) the block residual with a full-precision shortcut to maintain information. The proposed network is compared with several general binarization methods on two public datasets. The experimental results show that the proposed network outperforms the general binarization methods.

strengths:
\+ The presentation of this paper is good and easy to follow.

\+ This paper provides a good engineering solution for efficient human mesh recovery. It raises the performance of previous binary networks on the EHF and AGORA datasets.

weaknesses:
\- The proposed method is not clarified clearly. In the down sample residual block (Figure 4, left), why the dual binarized convolutions with the same inputs, kernel sizes, and strides are needed? How does it differ from a grouped binarized convolution (i.e., with 2 groups)? As for the FullPrecision convolution in the block residual, its kernel size and stride are 1 and 2, respectively. Will this result in checkboard artifacts? 

\- None of the competitors is designed for human mesh recovery. I wonder whether the proposed method still has significant advantages in efficiency compared with those tailored for this task (in terms of GPU memory and FPS), such as [1-2].

[1] Dou, Zhiyang, et al. Tore: Token reduction for efficient human mesh recovery with transformer. ICCV 2023.

[2] Zheng, Ce, et al. Potter: Pooling attention transformer for efficient human mesh recovery. CVPR 2023.

\- The qualitative results show that some meshes predicted by the proposed method do not align with images well (e.g., hands and faces in Figure 7).

questions:
I have a few concerns that I wish could be addressed. I may change my decision after reading the rebuttal and other reviewers' comments.

Q1: It would be better to include the results of the baseline (Hand4whole) in the visual examples.

Q2: Why binarized deconvolution and linear layers even yield better performance (L345-346)?

Q3: How is the prediction head of SMPLX parameters binarized?

Q4: A visual architecture comparison with other binarized methods could help to capture the gist and novelty of the proposed design, especially when previous methods have similar components, e.g., Bi-real also adopts a piecewise polynomial function and shortcuts. 

Typos: "e.g.the" -> "e.g., the", "Hands regions" -> "Hand regions".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719495372

thread_type:
official_review

<thread_object>
creator: reviewer_fsnc

<content>
summary:
This paper proposes a binarized network for the task of human shape recovery from images. The main design in the proposed Binarized
Dual Residual Network (BiDRN) is the Binarized Dual Residual Block (BiDRB), which is further composed of Local Convolution Residual (LCR) and Block Residual (BR). The LCR is extended to convolution operations with down scale, down sample, fusion up, and fusion down. The design of LCR includes using Hardtanh as pre-activation function and the adoption of RPReLU activation. Experiments are conducted on two datasets to show the effectiveness of the proposed method compared with other binarization methods.

strengths:
This paper presents detailed design of the binarized dual residual block, which covers the design of the pre-activation function, local convolution residual with channel-wise RPReLU, and block residual.

The Local convolution residual is extended to scenarios where the residual does not match the dimension of the outputs of convolutions.

Both the backbone of the reconstruction task and the face and hand detection network are considered in the network binarization.

Experiments on EHF and AGORA show performance advantages over other binarization methods.

weaknesses:
The novelty and contribution of the proposed binarization is limited as most of the design is quite straightforward. The block residual is something new to me. 

The designed residual block is quite specific for ResNet. This limits the scope of the proposed binarized dual residual block. 

The experiment is conducted with comparison to other standard binarization methods, which are designed for general networks. This comparison is kind of unfair as the proposed binarization only works for this particular model, or maybe a broader ResNet-like architecture. A comparison to other binarization/quantization methods for networks in 3D human reconstruction or ResNet-like models is needed.

questions:
It is not very clear why the proposed binarization method target at the Hand4Whole method. There are other (latest) methods (with better performance) in the field.

Latency. While the OPs is provided to have a rough idea of the number of operations, It is also necessary to see the change of speed with the proposed binarization.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732260690073

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers and area chairs for a brief summary

comment:
Dear reviewers and area chairs,

We sincerely thank all reviewers and area chairs for their valuable time and insightful comments.

We are pleased to note that:
1. Reviewers kuYg and zufd find our method easy to follow, with reviewer zufd also recognizing the solidity of our motivation.
2. Reviewer Ywt4 acknowledges that our method effectively addresses an underexplored challenge in the binarized HMR task.
3. All reviewers appreciate the superior performance of our method compared to previous BNN approaches.
4. Reviewers Ywt4 and zufd highlight that our method achieves significant reductions in both parameters and computational operations.

We have responded to each reviewer individually and would like to summarize our responses here:
1. We clarify that our method is applicable to a wider range of **CNN-based** backbones.
2. We perform an additional experiment to **compare our BiDRN with the more recent binarization method BBCU**.
3. We adapt our method to the **latest one-stage HMR method, MultiHMR**, further showcasing its generalizability and effectiveness.
4. We include the **visual comparison of the full-precision method Hand4Whole**. 
5. We include a **visual architecture comparison** between our BiDRN and other BNN methods to better illustrate our novelty.
6. We clarify details in the paper, including the **latency improvement**, **design of down sample residual block**, **comparison to efficient methods**, **qualitative results**, and **motivation of binarizing BoxNet**.

We extend our gratitude again to all reviewers and area chairs!

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a binarized network for whole-body 3D human mesh recovery. The proposed method is designed to reduce the high computational cost of existing multi-stage pipeline, so that the model can efficiently run on mobile devices. Built on the baseline, Hand4Whole, this paper redesigns the architecture of backbone, bounding box detector, and decoding heads to reduce the computational costs while alleviating the reduction in precision.
Although the performance gain by the proposed method over existing methods should be appreciated, the reviewers feels that the paper provides an engineering solution to the task.  Indeed, the reviewers except for Reviewer zufd raised common concerns regarding limited novelty, insufficient validation, and lacking detailed explanation of the method.  On the other hand, Reviewer zufd appreciates the first implementation of binarized network for this task and good performance.  The architecture is straightforward and the design choices are not well justified.  Moreover, it is specific to CNN-based backbone.  Compatibility with one-stage methods is unclear.  It is also suggested to choose appropriate methods for HMR to compare.  The authors tried to address the raised concerns by providing additional experiments; however, the reviewers except for Reviewer zufd have not been convinced.  In fact, newly provided experiment on human keypoint detection is not in the focus of this paper, meaning significant changes of the scenario will be required.  Reviewer kuYg’s concern on missing comparison with methods for SMPL is reasonable.  Even though the aim is not to create a general binarized framework, comparison is necessary to convince the effectiveness of the proposed method.  The authors say that the Local Convolution Residual and Block Residual techniques are compatible with one-stage methods, but no evidence is provided.  It is unconvincing to focus on the model with CNN backbone as there are more advanced methods with ViT backbones.  Simpleness and effectiveness cannot be significant contribution.  Clarifying the design ideas behind should be clearly explained with evidence to bring some insight to the community.  Otherwise, such a paper cannot be solid.  The claims of the paper should be properly justified with thorough evaluation.  For this, substantial improvements and another round of reviews are required. This paper cannot be accepted, accordingly.