Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work focuses on modeling 3D garments with a novel neural sewing machine (NSM).  The NSM is a learning framework that can effectively encode garment representations, which could facilitate 3D garment construction and manipulation.  Firstly, it utilizes a unified sewing pattern encoding module to embed sewing patterns.  Then, the embeddings are decoded into a 3D garment by using the UV-position maps with masks.  A few loss terms are introduced to preserve the inner-panel structures of the 3D garments.  Extensive experiments validate its superiorities over previous SoTA methods.
strengths_and_weaknesses: Strengths:
1. Using sewing patterns for 3D garment modeling is a good idea and also a nice contribution, which supports both 3D garment reconstruction and controllable editing.
2. Extensive experiments show that the proposed method outperforms previous SoTA methods quantitatively and qualitatively.

Weakness:
1. This manuscript claims that previous methods failed to learn reconstructable and manipulable representations with complicated topology.  Maybe a few more examples with complicated or straight shapes can be provided.  

2. Also, all the experiments are conducted on synthetic data, and it would be nice if real-captured or even in-the-wild images can be used for evaluation.

3. It seems like this work can only reconstruct the garments with the canonical pose, and many local details (e.g. wrinkle) are missing even if they can be observed in the input image.  And the possible penetrations between the reconstructed 3D garments and the 3D human body are not considered during reconstruction and manipulation.
questions: Generally, I think it is a nice paper, and maybe the authors can provide more real-captured images as I mentioned in the ``Weakness''.
limitations: The manuscript has a section to discuss the limitations and broader impact.  But the method limitations have not been properly discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper looks into the problem of 3D garment modeling and reconstruction which is a challenging problem within the wider area of 3D generative modeling and reconstruction. The problem is especially challenging because it is unclear as to what constitutes a good representation for 3D clothes that captures the wide variety of garments we see around us. This paper proposes to use sewing patterns as a representation of 3D garments and proposes a learning based framework - Neural Sewing Machine - capable of generating garments of varied topology. The authors discuss challenges and propose solutions to both encoding and decoding using this representation. They additionally propose three kinds of losses – inner-panel structure preserving loss, inter-panel structure preserving loss and surface normal loss. The authors claim that their method outperforms the prior state of the art. 

strengths_and_weaknesses: Strengths: 

In my evaluation, the proposal of this paper is very good. Indeed, representation of 3D garments is a challenge and this paper provides an interesting sewing pattern based representation. The authors also do a great job of highlighting various design choices involved in both encoding and decoding this representation and thus building their proposed – Neural Sewing Machine. In particular, I believe the following are the set of strengths of this paper: 

1. Unified Sewing Pattern Encoding: I like this encoding scheme as it allows for representing clothes of diverse categories, shapes and topologies. The fact that the authors chose PCA to come up with the encoding is again very interesting – something well suited for this problem setting as there is only so much variety in clothing. 

2. 3D Garment Decoding: The authors provide a principled approach to decode the encoding by learning a 2D UV map and a mask map which together represent the various panels. 

3. Structure Preserving losses: The proposed losses are intuitive and crucial for getting good results. The ablation that the authors provided validates their importance. 

4. Mean Geodesic Length Error: I particularly liked this metric to evaluate the quality of topology and the authors show that their method gives good results on this metric. 

Before I mention the weaknesses of the paper, I want to emphasize that the authors have presented an interesting method to model and generate 3D garments and I personally feel that the idea has a lot of potential. However, I think that authors have left out several important experiments and their analysis does not adequately back the claims they are making. Moreover, I don’t find their choice of baselines very convincing. I also find the paper is hard to read in several places with figures missing captions/ not being clear enough. 

1. I think the authors should do a better job at describing the related work. They mention them all in a single line L112, L115 but that’s pretty much what the reader gets. At the present moment, it is hard to properly contextualize the contribution of this paper with respect to existing literature. 

2. The authors show comparison with general purpose and ‘freeform’ baselines such as pixel2mesh and anchorUDF which is good to know but by no means is a fair comparison in my opinion. Such weak baselines tell little about the efficacy of the proposed method. The authors dedicate half a page in the related work on template-based models and garment structure modeling – which clearly shows that there are plenty of closely related recent papers (with open source code) that specifically target the garment generation problem. However, the authors don’t provide any comparisons for these. Therefore I am not convinced that this paper is really state of the art (not that it matters). 

3. I am also curious to see an ablation for the UV position maps as an intermediary representation. Particularly, how well does it compare against those approaches that used an alternative image based representation. 

4. I think the paper would benefit if authors spend some time in making the figures more explanatory. For example, in Fig 6 there are no captions as to which one is the prediction and which is ground truth. Moreover, I qualitatively cannot make sense what are the artifacts manifested by each loss term ablation. Perhaps an inset focusing on the artifact will help. 

5. Novelty on surface normal loss – I think authors should not claim novelty on this one as it is pretty much well known in the area of 3D reconstruction. 

6. While the authors introduce a new metric – Mean Geodesic Length Error (MGLE) to measure the quality of topology, I believe that the authors should have provided more analysis for the behavior of this metric such as the effect of the number of samples K. This is crucial because this metric seems to be important to show the superiority of the proposed method and I suspect that for different values of the sample K, the results will be very different. I hope to see results with a much larger value of K (on the order of a few 1000s atleast). 

questions: 1. L59 : “Our NSM guarantees …” – I don’t understand how this statement is true? It’s not that the 2D UV map is an isometric parameterization of any kind. Maybe I am missing something?

2. The authors make comments about physics-based simulators in several places and believe that their method is superior to this line of work. I have absolutely no clue how using physically based simulators for interacting with the clothes is a “contemporary line of work” to the proposed method, as the two are completely different items? What’s the context or argument here? Also, the authors claim that their method creates garments which can be manipulated but nowhere do they qualify or provide any evidence for this one. 

3. In the dataset, the authors say L245 - “ We use the first 90% part of the data for each base type … and remaining as the test set”. Where is the validation set? If the authors fine tuned their model on the test set then this basically throws all of experimental evaluation into jeopardy. 

4. I also didn’t understand the section on controllable garment editing. It looks interesting but I could not parse through the procedure involved in getting that. 

I firmly believe that the idea has a lot of potential, but I am not convinced with the experimental evaluation provided here. Unless the authors can address the concerns (primarily with comparisons to relevant methods) I don't think there is any merit to accepting paper. This is simply because the main hypothesis of this paper is that using sewing patterns as a representation is inherently superior to other kinds of representation. And in the absence of state of the art results on relevant comparisons, I don't see how else the authors would validate this hypothesis.  
limitations: I am surprised that the authors didn't talk about obvious limitations of their work — both technical and in terms of its social impact. 

Due to the choice of their representation, I doubt that their method generalizes to other clothing styles and the authors neither experiment with the generalization nor provide any comments on it. This to me seems like a severe limitation of the proposed work. However, I don't think this is any reason to reject the paper or the idea as this is not the main hypothesis/claim of the paper. 

The dataset used in this paper contains clothing styles which are not very inclusive. In particular, there are several garments found in South Asia, East Asia and Africa which would be hard to be captured by the PCA basis used in the paper — which primarily contains popular garments worn in the western countries. I believe that the lack of inclusivity of garments found in other (and often underrepresented) regions of the world fundamentally limits this work and moreover prevents even a discussion about the needs of such underrepresented regions and cultures as it relates to automatic 3D garment representation and generation.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a network model to reconstruct 3D garments based on sewing pattern input. The sewing patterns are first gone through PCA for coefficients to compose the embedding. Later, masks and uv position maps are generated using the coefficients. The pixels from the masked maps are used to reconstruct the 3D mesh. The paper introduces new loss functions to regularize the model behavior using deformation and sewing edge alignment rules, as well as normal consistentcy. Results show that the proposed method works better in a single-view garment reconstruction task and is capable of controllable editing and interpolation.
strengths_and_weaknesses: Strengths:
 - The paper is the first one that I know of that addresses the importance of modeling garments in the context of sewing patterns, which is the key to develop a physically correct as well as easy-to-use garment generation models.

Weakness:
 - The method does not model tightness of the cloth. What it really does is merely a *garment template mesh* generation model for a given sewing pattern shape. While it is not a critical component, the authors should discuss it either in future work or in the limitation section.
 - It is still not clear how the position maps compose the final 3D mesh. Although there are several sentences in the caption of the figure in the appendix that mentions it, it is not in sufficient detail. How is the mesh triangulated around the sewing pattern boundary where the edges are defined in lines but the inner vertices are defined as grid points? What does the algorithm do when the 3D coordinates of the sewing edges from two patterns are different? Is there any post-process to smooth out the neighboring vertex locations caused by the difference? If not, would that generate unnatural artifacts? There are a lot of critical details on this topic but the authors did not discuss them in detail.
 - The fabric material is not modeled. This is not a necessary component but should be mentioned in the future work section.
 - It is not clear how the basic panel groups are defined and it is not shown that what has been included would be sufficient to support *generic* garments. There seems to be only 9 basic panel groups. How are they chosen? Which factors are considered? Which garments can be supported by the 9 groups and which cannot? For example, can the model support T-shirts with pockets or dresses with lace/ribbon decorations using the 9 groups? It is a critical component of the paper but it is not discussed in depth.
 - It seems in Fig. 8 that the garment editing for the panel class is manual. It doesn't actually help too much because the user will have to specify the new pattern shape themselves which could be tedious.
questions:  See above.
limitations:  See above.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes the Neural Sewing Machine (NSM) for structure-preserving 3D garment modeling. Previous methods mostly model garments in a class-specific way, which is not flexible and easily extendable. In this work, each garment is modeled as several basic panels and their stitching relations. The basic panels are encoded by PCA from a large-scale dataset. The stitching relations as well as the 3D structure of each panel are encoded in the UV-position maps. For each image input, NSM predict PCA coefficients for each basic panel that corresponds to the observed garment. Based on the predicted PCA coefficient, a CNN-based network is trained to predict the UV-position maps. Four training targets are introduced to facilitate the supervised training. The reconstruction results are the best compared with previous baselines. Utilizing the structure-preserving modeling, controllable garment editing and interpolation are also shown to demonstrate the versatility of this kind of garment modeling.
strengths_and_weaknesses: ## Strength
1. Considering the sewing pattern into the garment modeling process and unifying modeling for basic garment classes are important contributions towards practical real-world garment modeling.
2. Both qualitative and quantitative results surpasses previous SOTA methods.
3. Ablation study proves the effectiveness of the proposed training targets.
4. The application of controllable garment editing is interesting.

## Weaknesses
1. The experiments are all done on a synthetic dataset, which might not be enough to fully prove its practicality when facing real-world data. For example, given a real photo of a clothed human, can NSM still give correct reconstructions on the clothes?
2. It seems like the sewing pattern/ topology of several basic panels are solely represented by the predicted UV-position maps. It might lead to two problems. a) For two corresponding edges, the predicted UV-position are not guaranteed to be perfectly aligned, which would give broken geometry when stitching panels. b) For problem a, a solution would be to find correspondences and force them to align spatially. However, finding such correspondences is not trivial under the current modeling. The edge points in the predicted UV-position map are not necessarily spatially ordinal.
3. It seems like NSM is only capable of reconstruct clothes in the canonical space. Can NSM takes photos of a posed clothed human as input? Can NSM output posed clothes as observed in the input image? A simple synthetic dataset for this would be Cloth3D.

## Typo
Table 3, headers, four L_{rec}.

## Missing References
[1] Xiang, D., Prada, F., Wu, C., & Hodgins, J. (2020, November). Monoclothcap: Towards temporally coherent clothing capture from monocular rgb video. In 2020 International Conference on 3D Vision (3DV) (pp. 322-332). IEEE.

[2] Hong, F., Pan, L., Cai, Z., & Liu, Z. (2021). Garment4D: Garment reconstruction from point cloud sequences. Advances in Neural Information Processing Systems, 34, 27940-27951.

[3] Bhatnagar, B. L., Sminchisescu, C., Theobalt, C., & Pons-Moll, G. (2020, August). Combining implicit function learning and parametric models for 3d human reconstruction. In European Conference on Computer Vision (pp. 311-329). Springer, Cham.

[4] Alldieck, T., Magnor, M., Bhatnagar, B. L., Theobalt, C., & Pons-Moll, G. (2019). Learning to reconstruct people in clothing from a single RGB camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1175-1186).
questions: 1. Section 4.2, what is the input for the reconstruction from sewing pattern?
2. Why inner-panel structure preserving loss learns an isometric mapping between UV and 3D coordinates?
3. For the controllable garment editing, can we draw the panel shape by hand?
limitations: The limitation is not discussed. Please discuss more on the limitation. Please see the above 'Weakness' section for my comments on the proposed garment modeling.
review_title: null

Meta-review: metareview: This paper was reviewed by four experts in the field. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to NeurIPS 2022.

The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. For example, 1) the evaluation on real-world datasets can be incorporated, 2) more discussion can be added on the reconstruction of garments with non-canonical poses. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
confidence: Certain
Decision: Accept