Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper presents a method for 3D shape and pose estimation of multiple persons from RGB images. The network utilizes numerous off-the-shelf techniques available from the literature and employs a transformer network to combine this information aided by a self-collision and an interpenetration loss.
main_review: Strengths:

1. The paper is fairly well written.
2. The paper addresses an important and a valid problem.
3. The model seems to achieve good results in challenging conditions.

Weaknesses:

1. The paper uses a collection of state-of-the-art models, but the actual novelty of the method itself is limited.
2. The performance of the approach is directly limited by the performance of used off-the-shelf methods.
3. The method seems to consist of a lot of steps without proper explanations or motivations (at least the authors have failed to describe them). For example, Sec. 3.3. presents a lot of design choices but lacks an explanation on "why?". For each design choice given in Sec. 3.3, there are many alternatives. The authors should justify these choices either theoretically or using a thorough ablation study. Table 3 presents an ablation study, but it only shows the impact of loss components, which is not enough.
4. There are many different moving parts, hyper-parameters, and design choices associated with the presented method. This raises serious doubts about the robustness of the method against the slightest changes of hyperparameters and conditions (or different datasets). At least the authors should have discussed some guidelines or motivations (or at least the effects of) toward choosing hyper-parameters. In general, the performance of any model is sensitive to hyper-parameters and design choices. But in a model that is a collection of a lot of different components, this is a significant problem that should be clearly discussed. 
5. The experiments section is not thorough. More experimental validation is necessary to conclude the usefulness of the model. If the paper presented a theoretically rigorous model, it is acceptable to weaken the experiments section. However, since the proposed model is mainly based on heuristics, the experiment section needs to highlight the strengths of the approach more. 
limitations_and_societal_impact: The paper has not discussed the limitations of the method adequately. The authors should discuss limitations associated with the points mentioned under "weaknesses" more comprehensively.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method to reconstruct multi-people in 3D from a single image. Different from popular approaches where a neural network pose estimator focuses on a single person given a bounding box, the proposed method estimates multiple people given multiple boxes at once, in particular considering their spatial layout and physical consistency. For that objective, the paper presents several novel aspects including a transformer framework, collision and interpenetration loss functions. Comparisons over previous approaches in the similar domain (multi-person) are performed in the public benchmarks (Panoptic[32] and CHI3d[3]) and the proposed framework outperforms the previous approaches. 

Overall, the paper is in a good shape, containing several interesting components. Additional experiments on the other more popular dataset (3DPW) would be still needed. 
main_review: Strengths:
- The use of transformer architecture for multi-people 3D pose estimation purposes is interesting. I think that this is the first paper in that direction. The use of global image features with box features are convincing for me, to better incorporate the contextual information of the scene. 

- The experiments on public benchmarks (Table 1 and 2) support the strength of the method over approaches in the same domain. The ablation study performed in Table 3 is also helpful to better understand the proposed models. 

- This paper also puts effort into other details such as introducing self-collision loss and interpenetration loss. This is encouraging, even though additional experiments over alternative options would be necessary to show clearer advantages. 

- I also like the fact that the paper is based on GHUM over the more popular human model, SMPL, particularly in terms of diversifying the framework in the 3D human pose estimation field. 

Weaknesses:

- Missing references: There are recent work where transformer architectures are used in different ways. It would be helpful to include them with discussions. 
Pose and Mesh Reconstruction with Transformers, Lin et al., CVPR 2021
Lifting Transformer for 3D Human Pose Estimation in Video, Li et al., 2021

- The paper ignores other popular 3D pose estimation benchmarks such as H36M and 3DPW. Even though the paper focuses on multi people scenarios, it would be still necessary to evaluate its performance in these datasets. In particular, note that the similar goal can be achieved via regression + optimization, for example, by (1) first estimating individual people per box (2) adjusting their spatial layout via an optimization. See the following paper:

Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild, Zhang et al., 2020

I understand the proposed method is trained without any 3D dataset (e.g., no H36m)? In this case,  I am concerned about its accuracy over the recent SOTA paper such as SPIN or VIBE. 

limitations_and_societal_impact: Yes. In section 6.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method for estimating multiple human pose & shape from monocular RGB input, which leverages a novel convolutional-transformer structure to optimize the detector-based human pose, shape and RT information. Also, it introduces a mesh-approx. based method for jointly formulating self-collision and interpenetration, which tackles the problem of treating two kinds of collisions in the same framework. Combined those methods, this paper tackles the problem of multiple human pose & shape estimation under challenging scenes.
main_review: The paper shows very well innovation, reasonable technical ideas with robust results and sufficient analysis. The reference is comprehensive.
According to the demonstrated results, this paper shows robust results dealing with different challenging scenes, including multiple human strong occlusion, incomplete human body capturing, and strong interactions between humans. The reconstructed human poses and 3D position relations between different humans are visually accurate according to the input images. The comparisons show its robustness compared with previous works.
The proposed convolutional-transformer structure combines the architecture of CNN and transformers, and takes the CNN features and human-detection features to initialize the transformer inputs, which may inspire future works on studying better detection-reconstruction combination methods. The proposed mesh-approx. based collision resolving method moves a step forward for formulating 3D collision in deep learning networks.

limitations_and_societal_impact: The first concern is that, although the pose and positions are estimated accurately, the estimated human shapes are slightly inaccurate, which may be further improved. 
Another concern is that, although it is not fair to compare this work with monocular single-person pose & shape estimation methods, this paper will show more profundity by comparing with those methods and analyzing the results.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a network and learning scheme for reconstructing multiple people from monocular images. The proposed approach includes a transformer to iteratively refine the reconstruction results, self-collision and interpenetration-collision losses based on the generalized winding numbers, and self-contact and interaction-contact losses. Qualitative results are impressive and strong.
main_review: This submission basically includes incremental upgrades of existing solutions for single/multiple person reconstruction and mesh collision detection, though these solutions bring performance gains over previous approaches for multiple person reconstruction.

Methodologically, though using the transformer to leverage spatial features and update the parameters of the reconstruction is somewhat interesting, the transformer for mesh reconstruction has been investigated in [Lin et al.], which is not discussed in the paper. 

[Lin et al.] End-to-End Human Pose and Mesh Reconstruction with Transformers 

For collision detection, exploiting generalized winding numbers is appreciated, which provides a more efficient solution for self-collision and interpenetration penalization in multiple person reconstruction. However, it only affects the runtime during training. There is no ablation study or discussion on the performance of different solutions for collision detection. 

Other minor issues:

Section 3.3 introduces the network architecture of using a transformer to predict the reconstruction results. Adding an illustration figure for the transformation of different features should make this part more readable.

There are some typos in the manuscript:
L35: deffer -> defer
L38: "so their 3D spatial relations are plausible, and the alignment with the image evidence is good.”?
limitations_and_societal_impact: In Line 114, does the $N_P = 16$ mean that the maximum number of people in an image is 16? Do the number of people in the input image affect the performance of the transformer? Moreover, how do the solution and scale of the person in the image affect the reconstruction?
review_title: null

Meta-review: This submission received 4 diverging final ratings: 4, 6, 7, 6.
On the positive side, the reviewers mentioned importance of the problem, an overall interesting approach to the multi-person setting, strong performance and clear presentation. At the same time, some of the reviewers were skeptical about overall novelty (collection of known components), insufficient motivation of certain design choices and high overall complexity, lack of important ablations and evaluation on common datasets. Some of these concerns were addressed in the rebuttal, as acknowledged by the reviewers. The most skeptical reviewer did not engage in further discussion with the authors.
Overall, the AC agrees with the positively-inclined reviewers that the strengths of this work outweigh its weaknesses, and the paper presents an interesting approach that can potentially have an impact. The authors are highly encouraged to address the remaining concerns in the camera ready version. The final recommendation is to accept as a poster.
Decision: Accept (Poster)