Title: Incorporating Test-Time Optimization into Training with Dual Networks for Human Mesh Recovery
Venue: NeurIPS 2024
Accepted: yes
Keywords: Human Mesh Recovery; Meta-Learning; Test-Time Optimization; Training Optimization
Abstract: Human Mesh Recovery (HMR) is the task of estimating a parameterized 3D human mesh from an image. There is a kind of methods first training a regression model for this problem, then further optimizing the pretrained regression model for any specific sample individually at test time. However, the pretrained model may not provide an ideal optimization starting point for the test-time optimization. Inspired by meta-learning, we incorporate the test-time optimization into training, performing a step of test-time optimization for each sample in the training batch before really conducting the training optimization over all the training samples. In this way, we obtain a meta-model, the meta-parameter of which is friendly to the test-time optimization. At test time, after several test-time optimization steps starting from the meta-parameter, we obtain much higher HMR accuracy than the test-time optimization starting from the simply pretrained regression model. Furthermore, we find test-time HMR objectives are different from training-time objectives, which reduces the effectiveness of the learning of the meta-model. To solve this problem, we propose a dual-network architecture that unifies the training-time and test-time objectives. Our method, armed with meta-learning and the dual networks, outperforms state-of-the-art regression-based and optimization-based HMR approaches, as validated by the extensive experiments. The codes are available at https://github.com/fmx789/Meta-HMR.


Peer reviews:
comments: SUMMARY:
The paper presents a new training paradigm for better test-time optimization performance at test time. Specifically, two strategies are proposed. (1). integrate the test-time optimization into the training procedure, which performs test-time optimization before running the typical training in each training iteration. (2). propose a dual-network architecture to implement the proposed novel training paradigm, aiming at unifying the space of the test-time and training optimization problems.

STRENGTHS:
1. The paper proposes a new training paradigm that can integrate test-time optimization into the training procedure.
2. The overall procedure is clear and easy to understand.

WEAKNESSES:
My biggest concern is that the performance improvement is not significant when compared with the existing method.  The Table 1 compares the results of an existing method PLIKS and the method proposed in this paper. However, the comparison of 3DPW is not fair, since the results of the last two lines use 3DPW for training, while the reported PLIKS does not use 3DPW for training. PLIKS trained on 3DPW can achieve 40.1 PA-MPJPE and 73.3 PVE, which greatly outperforms the results in this paper.

QUESTIONS:
see weakness

LIMITATIONS:
see weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

comments: SUMMARY:
This paper targets on the Human Mesh Recovery task. The authors adopt a test-time per-instance optimization method similar to EFT, but proposes a meta-learning-like framework to pre-optimize the pretrained human pose estimation network, aiming to provide a better starting point for test-time optimization. In addition, the authors also propose a dual-network mechanism to solve the test-time and training-time objective misalignment. The idea is novel and interesting. Experimental results also fully demonstrate the effectiveness of the propsoed method.

STRENGTHS:
* Incorporating meta learning idea in test-time optimization is interesting and demonstrates leading performance.
* The authors conduct extensive ablations to validate the proposed method.
* Paper is well organized and written.

WEAKNESSES:
* Some experimental results were not clearly explained. E.g. In Fig. 3, EFT gets worse after several optimization steps, while the proposed method keeps getting better. Could you please give some intuition on this? Why EFT gets worse and which design of the proposed approach alleviate this issue?
* Fig. 1 is a bit dense and difficult to follow. Perhaps the authors could try to reorganize and simplify it.
* The paper consists a large number of experiments. However, the authors don't provide statistical significance for any of the experiment as suggested by the NeurIPS Paper Checklist 7. It is suggested to at least include the standard deviation or the standard error for the main results.

QUESTIONS:
Please see above.

LIMITATIONS:
The authors didn't include Broader Impacts in the paper. Although two failure cases are provided, more are encouraged to give a comprehensive overview.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nHuman mesh recovery (HMR) is improved by incorporating test-time optimization into the mesh prediction model’s training loop, allowing the learned parameters to potentially be more readily adapted to examples at test time. This is referred to as a meta learning approach, and it departs from prior work that also introduced this meta learning strategy by incorporating a secondary network to make predictions for pseudo ground truth meshes during the test time computations (during training and actual testing). This novelty as well as the design choices of the approach as a whole are validated through ablation studies, and the final method is shown to perform strongly relative to SOTA competing methods.\n\nSTRENGTHS:\nOriginality: Adding to the meta learning approach an auxiliary network that provides a pseudo mesh for the test time adaptation is a novel contribution. Ablation studies illustrate the effects of key method hyperparameters.\n\nQuality: Incorporation of the test-time adaptation into the training procedure is well motivated. The method is tested with multiple backbone networks on multiple datasets, against multiple strong baseline methods of varying types (optimization and regression). Quantitatively, it significantly outperforms the state of the art baselines. The ablation studies support the contention that the proposed learning approach will be effective as joint estimation and mesh prediction models advance. The results (especially Figure 3 and Table 3) support the idea that meta learning is a critical framework for solving the HMR problem. Table 3 also supports the use of the dual network approach.\n\nClarity: The paper is reasonably well written and the figures/tables helpfully convey key concepts and results. \n\nSignificance: The problem of predicting human mesh parameters from image data is actively studied, and progress on it impacts various applications. The submission makes notable contributions that improve the ability to solve this problem.\n\nWEAKNESSES:\nThe main weakness is presentation, which I think can be improved in a few ways. I outline these below and in the Questions section. Importantly, I think the results in this paper are strong, and I would be happy to raise my score if these weaknesses are addressed.\n\nMost importantly, the contextualization relative to the prior work “META-LEARNED INITIALIZATION FOR 3D HUMAN RECOVERY” (Kim et al., 2022) needs to be improved. \n\n - The related work section references Kim et al. (2022), which also uses meta learning for HMR, but other than that reference the phrasing in the submission suggests that the meta learning idea is a novel contribution of this submission. I think this phrasing should be adjusted to better reflect that the meta learning component here is not novel. Some examples of phrasing I would change follow: the submission says that the relevance of meta learning to HMR is its “key observation”, that it is “rethinking” the HMR problem by introducing meta learning, that it “proposes” to use meta learning, etc.\n\nQUESTIONS:\nLine 153 and Algorithm 1: the referenced meta learning approach MAML involves computing gradients through other gradient computations (i.e., second order derivatives). Could you please confirm my understanding that Algorithm 1 does not do this, and (if it does not) explain why it's described as “simulating” this meta learning approach? \n\nPlease add a quantitative comparison to the prior work that introduced meta learning to this problem, Kim et al. (2022), using the same training/testing data for each method. Also, please highlight the differences between these methods when discussing their different results, and connect this discussion to Table 3 because I believe its row 2 corresponds to Kim et al. (2022). \n\nLine 251: I would add quantitative results to this section or avoid claiming strong OOD performance (e.g. relative to EFT). The provided qualitative results with 2 images do not provide sufficient support for the OOD claims.\n\nConsider replacing the backwards sum notation in Figure 1 by placing the usual notation on the left side of the summands.\n\nLIMITATIONS:\nThe main text does not discuss any limitations. The appendix has some discussion of them that should be moved to the main text.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method to enhance Human Mesh Recovery (HMR) from images. The approach integrates test-time optimization into the training process, inspired by meta-learning. The dual-network structure is designed to align training and test-time objectives, improving the starting point for test-time optimization. Extensive experiments demonstrate that this method outperforms state-of-the-art HMR approaches, providing higher accuracy and better generalization to test samples. The work emphasizes the advantage of combining test-time and training-time optimization for robust human mesh recovery.\n\nSTRENGTHS:\n1. The authors introduce an innovative approach by incorporating test-time optimization into the training phase, which addresses a gap in existing HMR methods. This blend of meta-learning principles ensures the optimization process at test time is more effective, enhancing performance and accuracy.\n\n2. The proposed method integrates seamlessly with existing HMR models using a dual-network structure, ensuring minimal additional computational load during training. This efficiency is achieved without compromising result quality, making the approach practical for real-world applications.\n\n3. The authors present their work in a well-structured and accessible manner. The logical flow, clear explanations of technical terms, and use of diagrams ensure that even complex concepts are easily understood, enhancing the overall impact of the paper.\n\nWEAKNESSES:\n1. The inclusion of dual networks does not fully achieve the goal of matching training and testing objectives. There remains a discrepancy between the training loss (using ground truth labels) and the testing-time loss (using pseudo labels). There is no guarantee or analysis provided that the gradients from ground truth labels and pseudo labels align.\n\n2. Using dual networks will introduce the ensembling effect. The second term of $\\bigtriangledown_{\\omega} L_{test}$ ($\\bigtriangledown_{\\omega} L_{3D}$) acts as an adjustment towards an intermediate estimation between the two networks. To fully eliminate the ensembling effects, another row in Table 3 should be included showing the results of EFT_CLIFF with two networks (averaged results) for a more accurate comparison.\n\n3. While the last subsection in Section 4 is a good start, it is insufficient. Test-time optimization (TTO) methods typically perform significantly better than their counterparts on out-of-domain datasets, which are critical for demonstrating the full potential of TTO solutions. A quantitative comparison with EFT on the LSP-Extended dataset would provide a more comprehensive validation of the method's effectiveness in OOD scenarios.\n\nQUESTIONS:\nIn line 7 of Algorithm 1, why use the gradient w.r.t. $w'$ to update $w$ (similar to Reptile), instead of the gradient w.r.t $w$ (as in MAML)?\n\nLIMITATIONS:\nNo negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}