Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents an approach for 4D garment reconstruction. To this end, it first presents a statistical shape model for garments which is learned using sequences of garment meshes. Since garment meshes across different sequences may have different topologies, the paper aligns them by first registering them to a template mesh and then re-meshing using barycentric interpolation. Given the statistical garment model, the paper then presents an approach for 4D garment reconstruction from a point-cloud sequence as input. The point cloud is first segmented into different body parts and then fed into an MLP to predict the PCA coefficients of the garment model. This provides the garment mesh in the canonical pose. Subsequently, for each body pose, the canonical mesh is posed using the LBS skinning weights interpolated from the body model. These posed meshes are further refined by iteratively predicting additional displacements using hierarchical feature extractors and a transformers-based temporal model. Experiments are performed on the CLOTH3D dataset and compared with adapted baselines. The proposed method is shown to outperform the baselines. Additional ablative experiments demonstrate the importance of different design choices. 
main_review: **Pros:**
- The paper addresses the problem of garment reconstruction from point-cloud sequences which has not been addressed previously and is of great interest, though I am not sure if the paper aligns well with NeurIPs (may be better suited to CVPR, etc.) 
- The proposed approach is intuitive and makes sense, although it's incremental in nature. 
- Appropriate baselines are developed to validate the contributions of the paper. 
- Sufficient experiments are performed to validate the design choices. 
- I like the idea of interpolated LBS to avoid cliff-like skinning artifacts. 
- I like the idea of using interpenetration loss to explicitly build a relationship between body surface and clothing. 

**Cons:**
- No qualitative samples for real data are provided. 
- Also, no examples of very fast body movements is provided in the supplemental video. I mainly wanted to evaluate the impact of laplacian smoothening. For slow motion, the resulting reconstruction looks good, but how does it look for fast motion? Is there any lag in general?
- The paper is a bit hard to follow at times. I sometimes missed the intuitions behind different design choices which makes the paper less interesting to read. Also, the paper requires a thorough revision. I have provided some typos below. 
- In Section 3.4, when replacing a specific component of the proposed approach, it's not clear what is used instead. Please make it clear by also providing the details of alternative components. 
- The related work is not enough and requires a thorough revision. 

**Questions:**
- For the BC term in eq (6), how are the corresponding boundaries obtained?
- How are the human body meshes obtained? Are ground-truth meshes used in this case? 

**Missing References:**
The following papers should be referenced in the paper as they also tackle estimation of clothing: 
1. MonoClothCap: Towards temporally coherent clothing capture from monocular RGB video, 3DV'20. 
2. DeepCap: Monocular Human Performance Capture Using Weak Supervision, CVPR'20.

Also, 27 references are too few to do justice with the large amount of literature on human clothes modeling. Please thoroughly revise the related work. 

**Typos:**
L67: learns -> learn 
L76: are formulated -> is formulated. 
L77: afterwards and successively are redundant. 
L87: re presents -> represents 
L117: normalized -> normalizes 
L174: Biases
L228: homotopy -> homotopic 
L229: of all three aspects. 
L334: and help in capturing the deformations caused by body movements. 

**Other suggestions:**
The work canonical-posed can be replaced with `Canonical` for ease of reading. This, however, is a personal preference and not critical. 
limitations_and_societal_impact: No limitations and societal impacts are provided. I suggest authors to kindly address this by providing results on faster motions, different body shapes, and providing additional comments about societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This manuscript designs a parametric garment model and utilize techniques such as point cloud network and GCN to reconstruct garment mesh. Quantitative and qualitative experiments on synthetic data show the effectiveness of this method to a certain extent. Compared to MGN [16], the method can better process loose garments and produce more natural reconstructions. The technical description is also clear and reproducible.
main_review: I have several concerns:
1. Its parametric garment representation, including the interpolated LBS skinning weights computation, is very similar with the garment model of BCNet[17], except it adds displacements on posed garments. I think this will decrease the manuscript's novelty.
2. The authors claim that they are the first to explore garment reconstruction from point cloud sequences. It may be true while I do not think it is very importmant. First, some works have explore the clothed body reconstruction from point clouds [A,B], while they use implicit representations. Second, point cloud input can bring other problems. For example, real point cloud data often has holes and may be partial, these can greatly affect the prediction of the network, while these situations are not analysized in the manuscript.
3. If I understand correctly, the method not only needs the garment point cloud, but also the dressed body point cloud, which is used in the Hierarchical Body Surface Encoding. I think this requirement severely restricts the application of this method to real scene. Because in a realistic scanning environment, it is difficulty to get the body scanning hidden under the garments. Therefore, I doubt the practicality of this method.
4. All the experiments are based on the synthetic data, while some real point cloud should be tested to present the practicality. Besides, the demo presented reconstruction results are not satisfactory  for me.

[A] Saito S, Yang J, Ma Q, et al. SCANimate: Weakly supervised learning of skinned clothed avatar networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 2886-2897.
[B] Chen X, Zheng Y, Black M J, et al. SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes[J]. arXiv preprint arXiv:2104.03953, 2021.
limitations_and_societal_impact: As discussed above, the authors should analyze more about the practicality and scalability of the method.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed a framework, Garment4D, to reconstruct 3D garment models from 3D point cloud sequences of dressed humans instead of 2D RGB images. There are three key components in the proposed framework, including garment registration, garment shape estimation in the canonical pose, and garment reconstruction in the target pose. Experiments demonstrate the effectiveness of the proposed framework with more accurate and natural reconstruction results.
main_review: This submission claims itself as the first attempt to reconstruct the 3D garment model from 3D point cloud sequences, which eliminates many limitations of those solutions using 2D images. The three key steps in the proposed framework also include new technical designs to address the challenges in modeling the detailed deformation and garment dynamics. Overall, this paper is well-written and technically sound.

There are a few concerns regarding this paper:

- Guo et al. have also investigated using point cloud as input for the reconstruction of dynamic geometry of clothes. Though it seems that these two papers are concurrent with each other, it is recommended to add discussions or comparisons with Guo et al. in the final version of the submission. 
[Guo et al., Inverse Simulation: Reconstructing Dynamic Geometry of Clothed Humans via Optimal Control, In CVPR, 2021]

- In Sec. 2.3, point-wise semantic segmentation is needed to select the garments from the input point cloud. In practice, it may be harder for algorithms to perform clean segmentation from real-world point clouds. How about the robustness of the estimation results over the segmentation errors?

- In Sec. 2.4 and Fig. 3, are all hierarchical garment and body features concatenated together as the input of GCN? It is not very clear in the presentation and figure since the vertex numbers of the garment and body are different.

- In Sec. 3.4 and Fig. 6, how about the trade-off between the K value and the reconstruction performance? How about the additional computational cost raising by the increase of K?
limitations_and_societal_impact: Reconstructing 3D garment models from 3D point cloud sequences is interesting and appealing. However, this submission does not show or disccuss the performance on real-world point cloud sequences. In comparsion of the solutions using 2D images as inputs, the generazation of the proposed framework remains unclear.
review_title: null

Meta-review: 
This paper proposes a framework  to reconstruct 3D garment models from 3D point cloud sequences of dressed humans. The paper raised concerns regarding robustness of the model to segmentation errors, limited real world results, similarities to recent works, requirement for body point clouds (under the garment), and accuracy of the proposed method. The rebuttal submitted by the authors addressed these concerns by showing real world results and the feasibility of obtaining body pointclouds using the SMPL model. Though the relevance of the paper to a wide NeurIPS audience maybe still be under question,  all reviewers are positive about the paper, and it is suggested for publication.  

Decision: Accept (Poster)