Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors present a technique to estimate the underlying pose and shape of a body from multiview images without requiring calibration information. The technique uses the human body as a calibration target and performs fusion of local and global features via a self-attention mechanism. The authors demonstrate the efficacy of the approach on standard benchmarks. 
strengths_and_weaknesses: Strengths:
1. The idea of using the body as a calibration target is novel. 
2. The method shows comparable accuracy to methods requiring calibration which is not always available. 
2. The authors have collected a new dataset (Social Datasets) to qualitatively evaluate the proposed approach. 

Weaknesses:
1. The approach is not sufficiently motivated. Such a multiview approach applies to a multi-camera rig setting, and calculating the calibration is straight-forward. The authors should provide a more compelling discussion on why the method is of practical importance?  
2. The method stands to be most useful in the context of the Social videos dataset. However, no quantitative evaluations exist on the same. 
3. The assumption in line "141" of "We assume the 3D body vertex index i are consistent across people, views, and poses" is a strong assumption and forms the basis of the approach. Some quantitative analysis or commentary on this assumption will be useful for wider adoption. Some failure cases should also be shown. 
4. The self-attention mechansim proposed in the paper serves to remove dependence on calibration and is the main contribution of the paper. The authors could do a better analysis of the usefulness of self-attention mechanism. 
5. The architecture has limited novelty. For e.g., the self-attention mechanism could be reformulated as a transformer based multi-headed attention module. It would be useful to understand if such a formulation is beneficial. 
6. The complexity and run-time of the method is not discussed. 

Post rebuttal:
The authors have adequately responded to my concerns. I would have liked to visually see the failure cases. I am slightly increasing the rating of the paper based on addressing of my concerns. 
questions: 1. Did the authors try to replace the self-attention mechanism with an explicit calibration based fusion mechanism? It would be useful to understand the similarities and differences of the self-attention with such an explicit fusion mechanism beyond average and max pooling discussed in the paper. 
2. Please perform a quantitative evaluation on the social videos dataset as the the method is best suited for such a setting. This would entail calculating the relative pose between multiview images for comparison to the other methods, but this is a critical contribution and the wider audience would benefit from an analysis on "in-the-wild" dataset.
3. How does the method compare in run-time to other comparable methods?Did the authors experiment with bulkier or leaner verions of the architecture? A variant that runs in real-time would be especially useful for social videos setting. 
4. Why havent the authors evaluated against [34] which seems to be solving the problem in a more relaxed setting? The authors should show some examples with single camera videos which are multiview over time, albiet with pose changes. The behavior of the method in such a setting, along with associated failure modes would be useful to understand from an application perspective.  
limitations: The authors haven't discussed the failure modes of the approach adequately. The social videos dataset presents a good opportunity to qualitatively demonstrate when the method fails as mentioned in the limitations section (e.g. under severe occlusion). The authors haven't discussed the performance of the method on single camera videos which have greater practical appeal. Also the possibility to futher improve the approach with calibration by adapting the self-attention mechanism warrants some discussion to guide future work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: A 3D human body reconstruction method using multiview images without calibration is presented. The method leverages semantic calibration to align visual features learned from multiview images by dense keypoints detection.  
strengths_and_weaknesses: Strengths:
- The idea of using semantic calibration for human body reconstruction is sound and novel. Leveraging 2D/3D keypoints detection as well as multiple datasets make it an interesting approach. 
- Main concepts and motivations are well and clearly delivered.

Weaknesses:
- The application of multiview reconstruction requires that sequences are temporally aligned. Comparing the single view methods, this might limit the use cases of the proposed method.  
questions: - It is not clear how the global features and local features are used in this method. Could you add more details?
- The idea of leveraging multiple datasets with different GT is very attractive. Could the authors provide details on how the training is performed?
 
limitations: - Qualitative results are shown to validate the multiview methods advantageous over single view in terms of consistency across views. This point is obvious. Thus, more qualitative results might be more valuable. 

- Semantic calibration idea is sound for multiview reconstruction. If the idea could be extended to single view, or video-based reconstruction, that will be more practically useful.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper describes human pose and reconstruction with a multi-view method that can handle up to an arbitrary number of uncalibrated camera views. 
The proposed method maps visual features from different views to a canonical body surface to reconstruct a fused 3D SMPL parametric model. 
Depending on each view, a learnable parameter assigns confidence to the fused features, playing the role of attention. 

Human3.6M, UP-3D, MARCOnI and VBR are the datasets taken into account for collecting different views. Experiments are done solely on Human3.6M,  with the P1 protocol. Concerning the considered competitors, the proposed method shows the best performance on MPVPE-PA, both for single and multi-view approaches.  Qualitative results are also shown on Social Videos dataset.

strengths_and_weaknesses: The merit of the paper is to devise a multiview method using only key features and mapping them to a canonical model, which should, in principle, solve self-occlusion problems.
The paper is well structured, and the images are good, though it is not so well written.

A relevant weakness is a choice of experimenting only on Human3.6M.  
Why not also use 3DPW, which consists of 35515 RGB images of 7 subjects with paired ground-truth SMPL parameters.
Many recent contributions use 3DPW for testing. 

Furthermore, to compare with the single view methods, the authors report the mean and standard deviation of the error of each method tested on several individual inputs. 
The comparison might be unfair since a single view method does not take advantage of several views. 

Despite multi-views being claimed to be relevant for occlusions, there are no ablation studies on this issue.
Recent contributions such as “End-to-End Human Pose and Mesh Reconstruction with Transformers” and “Probabilistic Modeling for Human Mesh Recovery” have better results on single images than the authors. The authors should compare their method with the most recent works.

questions: There is minimal information about the experiments. Essential facts are the number of images used and whether different runs have been made with a changing number of images for the multi-view testing. 
Similarly, for training, the method can randomly choose images to construct a model from multi-views, what kind of protocol is used to choose images, and how many samples are collected, especially in training. 

The authors suggest that multi-view is relevant for solving occlusions, and in their Conclusions the authors write that there are cases in which it cannot be solved. A comparison with other methods, such as “PARE: Part Attention Regressor for 3D Human Body Estimation” (which uses a weak perspective camera), and a discussion on the conditions under which the method fails to solve occlusions is relevant for a complete understanding of the method.  

The proposed loss uses both 3D and 2D information and the SMPL parameters \Theta; given that recently, many methods do not require paired 2D-3D data and only use weak – supervision, would the proposed multi-view implant be able to extend to unpaired 2d-3d data, using the multi-view?
What about runtime and number of parameters:  is the model competitive in terms of computation time and size? 

limitations: Limitations are discussed in particular about self-occlusions
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a calibration-free method to reconstruct 3D human body pose and shape from multiview images. The proposed method is motivated by the observations that the human body can be leveraged as a semantic calibration target. Experimental results validate the effectiveness of the proposed method. Both qualitative and quantitative results are promising.
strengths_and_weaknesses: Generally, this paper is well-motivated, well-written, and technically sound. It is quite smooth to read this paper.

Overall, this paper presents an interesting solution for uncalibrated multi-view human pose and shape estimation.

+ The motivation of leveraging the human body as a semantic calibration target is appreciated.

+ The approximation of the discrete feature mapping via continuous mapping is interesting, which is the key to achieving differentiable feature learning. It is reminiscent of the existing differentiable rendering approaches.

Some concerns regarding the current version of the paper:

- Some part of the technical details is not very clear. It is confusing to me how to concatenate the per-pixel features with the global features?

- How about the proposed method when the camera calibration is available? The continuous mapping may also contribute to better fusion when the camera parameters are known. Adding such a setting in the ablation study should strengthen the paper.




questions: - The supplementary material provides a version using GCN to regress mesh vertices directly, but the quantitative results are counterintuitive as the single-view methods with non-parametric representations typically outperform the parametric counterparts numerically. But it makes sense that there are artifacts in the reconstructed meshes of the non-parametric solutions, as shown in Figure 3 of the supplementary material. It is also recommended to add ablation experiments that use the GCN mesh decoder when camera parameters are available.

- The related work could be improved to be more comprehensive.  It is recommended to include [A] in the related work, as it is a recent state-of-the-art multiview reconstruction method when camera calibration is available.

[A] Lightweight multi-person total motion capture using sparse multi-view cameras. ICCV 2021.

limitations: Please see the comments in Weaknesses and Questions.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new method for 3D human pose reconstruction give multiple (or a single) image without the need for camera parameters. The method is based on doing dense pose estimation, then predicting local and global features for how salient parts of an image are from a specific view. Finally it regresses parameters of a SMPL model which parameterizes the human body.

------------
Post rebuttal: Changing score from 6 to 7. I think the authors can improve the presentation of the paper and with that I would rate it a 7. I like the approach and design along with the motivation of using the human body as the semantic target. 
strengths_and_weaknesses: Strengths:
The intuition behind the paper makes a lot of sense especially the way it was presented in terms of global features and local features with an attention map. I particularly liked the experiments section. There were detailed comparisons to other methods, and the ablation study did a good job of justifying the different design decisions. I would have liked to see a comparison with SMPLify in the table as well, but a number of other single view baselins were presented.

Weaknesses:
The presentation of the paper could be improved, especially the methods section. For example, figure 4 which shows the full method overview should appear near the top, especially before figure 3 and 2. Seeing the system overview is important before diving into the specific pieces. Figure 4 should also include $F_A$ so we can see where those weights are. It also was not clear how the weights for $F_A$ are learned. Eq. 7 has two unknowns: $a_c$ which is the attention score and $F_A$ which is the weights of the attention module. Do we have access to the ground truth attention score during training in order to learn $F_A$?

The other main weakness is that the results shown in the video still contain considerable artifacts especially in the way the hand moves. Compared to other methods for related 3D reconstruction tasks, such as Nerf based methods, the artifacts are very noticeable. The results are better than the single channel baseline, but could still be better especially given the number of camera views.
questions: What's the reason why the final parameters were regressed instead of using an optimization based approach like SMPLify? It should be possible to do that without a calibration (optimize the 2D pixel loss from the output to the dense keypoint mapping) and it would ensure the output matches up with the visible images as much as possible.

It seems like the main proposed applications is for videos, and there is a large body of work on 3D pose from video. Is there a reason the authors chose to specifically focus on multi-view images without using temporal information?
limitations: Limitations are well addressed in the final section.

If the authors can address questions about the presentation and clarity, and include that in a revised version, then I would consider increasing my review score given that the contribution and method are interesting.
review_title: null

Meta-review: metareview: This paper presents a calibration-free approach (as traditionally understood) for 3D modeling of humans using multi-view images. The idea of using the human body as a means of providing "semantic calibration" is interesting. The method is scalable with the number of cameras. The paper received three 7s and two 6s. The authors and reviewers had several engaging conversations. The reviewers have also provided several suggestions for improving the presentation of the paper. 
confidence: Certain
Decision: Accept