Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposed a framework for  virtual try-on with multiple garments. The idea is first encode the the open surface implicitly with signed distance field and covariant field, and to combine multiple garment with collision free condition, the paper further proposed a formulation to use neural network to predict the untangle layered implicit surfaces, comparing with previous work (Buffet et al. [8]) which uses optimization process to achieve collision free. The final algorithm is formalizing the VTO as an optimization problem (Eq. 10), which needs to achieve both collision free and minimize distortions of the triangles. Qualitatively, the results achieves better performance comparing with previous works on VTO. 
strengths_and_weaknesses: Strength:

1. The paper is well written and easy to follow. 

2. The final performance on VTO looks better than baselines, qualitatively speaking. 


Weakness

1. Novelty. As summarized above (and also in the paper), there're two new things proposed in the paper: a) A neural field formulation to encode open surfaces, b)  utilizing a network to predict the untangled distance field, instead of optimization procedure. However, are these two novel? Why using them is correct choice? Regarding the encoding of open surfaces, there're already many published works [a, b], why the proposed method is better than these work? The paper didn't provide an explanation on this, nor comparisons. Regarding using a network to predict untangled distance field, this is a new approach to me, however, Buffet et al. [8] can solve the fundamental limitation mentioned in the abstract "the interaction between layered neural fields.", to me, the benefit is only using a neural network is much faster than [8], are there other advantages?

2. Limitation. There are a bunch of limitation of proposed methods:

- The experiment is only designed for canonical pose, extending to arbitrary poses is not clear (It's not as simple as mentioned in the Line 160-161), different poses will make the garment fitting more complicated,  and ULNef module might also have generalization issue.
- The ULNef is limited to a fixed number of garment, the input is N dimension vector and output is also N dimension vector, how does it generalize to different number of garments?
- The model would require to fit each module for each garment, how could this scale up to real world application, where we can have thousands of different clothes. 
- Generalization of ULNef. It's hard to say whether ULNef can generalize to different garments, e.g. trained one one set of garment, but tested on a different set of garments (there's also no analysis of this)

3. Expeirment. The experiment has many problems:

- The biggest issue for experiment is the model is only trained and tested on 5 garments (See Supplement Sec 1), this is very limited in evaluating the performance to check the whether the result can generalize in a larger scale. 
- It would be better to ablate each module proposed in the framework. I think there are missing at least two baselines for comparisons: i) To show the effect of proposed idea of encoding open surfaces, it would be better to compare at least one of [a], [b]. ii) To show the effect of ULNef, one baseline is not using a neural network to predict $\textbf{f}$, but use [8] to get $\textbf{f}$

 
In summary, this paper has many problems in both the method and experiment side, and I think it's not ready for publication at current status, so I vote for a reject. 

[a] NEURAL UNSIGNED DISTANCE FIELDS FOR IMPLICIT FUNCTION LEARNING
Julian Chibane, Aymen Mir, Gerard Pons-Moll
NeurIPS 2020

[b] 3PSDF: Three-Pole Signed Distance Function for Learning Surfaces with Arbitrary Topologies
Weikai Chen, Cheng Lin, Weiyang Li, Bo Yang
CVPR 2022

[POST REBUTTAL]
After reading the rebuttal and other reviews, I'm convinced of the results provided in the paper, but I still do not think the novelty is great enough for neurips, so I only raised my score to boardline accept. 
questions: See Weakness section above
limitations: See Weakness section above
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Based on recent success on neural cloth simulation for a single garment, the authors extend previous works to solve multi-layer cloth prediction while avoiding collisions. The author leverages the untangling power of the implicit surface model from Buffet et al. and neural cloth representation from Santesteban et al., designing a new neural untangling layer directly predicting collision-resolved multi-layer cloth pieces. The authors have shown a cool demo in the video that they can directly predict multi-layer cloth on the human bodies while tuning the human height and width.
strengths_and_weaknesses: Pro:

1. To extend a neural network-based virtual try-on system to a multi-layer cloth system is very novel and ambitious. The method utilizes a neural implicit representation of cloth and uses an optimization-inspired method to untangle different cloth pieces. The authors have shown cool visualization results that they can handle up to ~5 layers of cloth pieces stacking on a human body. In the ablation study, they also include results that using Fourier features and gradient loss can generate a better implicit representation of the garment model.

2. Another exciting part of this paper is the authors have built a comprehensive pipeline for the mix-and-match VTO task. From neural explicit/implicit representation to untangling, to find the final zero-set surface using optimization. It's not easy to fuse several challenging components to a final runnable pipeline.

Con:

1. Sec 4.2 about optimization of untangled garments is not well-written and very unclear. In line 242, they mentioned that "evaluating per-garment fields, followed by the neural projection (untangling I guess?)", not sure if it is only queried once to find the zero-level surface or done repeatedly during the optimization. Also, it is unclear if it is solving the {X_i} with all the pieces together or only finding layer by layer. I suggest the authors add an algorithm section to detailly include how they combine L-BFGS and the untangling operator.

2. Current results are limited by the evaluation, the authors simplified the problem to T-pose (with only height, and width), without pose parameters.

Some related work:
Collision avoidance based on neural network and optimization:
 LCollision: Fast Generation of Collision-Free Human Poses using Learned Non-Penetration Constraints
Q Tan, Z Pan, D Manocha
AAAI Conference on Artificial Intelligence
questions: 1. Sec 4.2 and Sec. 3.2 both are inspired by numerical optimization, why not also trained a network to replace the optimization in  Sec. 4.2?

2. In Eq. 9 the X(beta) is already dependent on shape parameters, why also need a diffusion step? Directly train a network to replace Mg(x, beta) maybe better?
limitations: The work is only evaluated on simplified canonical space.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents ULNef, which leverages layered neural fields to create collision-free clothing surfaces for mix-and-match virtual try-on. 
strengths_and_weaknesses: *Strength*

- This paper aims to resolve an interesting problem that how to fit multi-layer garments to a target body shape while avoiding colliding among garment layers. The authors propose a novel model that untangles the layered neural fields with neural implicit representations (i.e., a signed distance field and a covariant field). This potentially introduces a new paradigm for solving related problems.

- The visual results clearly verify the effectiveness of the proposed method.

- The way of obtaining SDF from open surfaces is interesting and has proven to be effective.

- I like the pipeline of solving layered VTO and the results are promising.

*Weaknesses*

- Only a quantitative ablation study is given in Table 1. A visual comparison should be given for understanding how the Fourier features and gradient supervision contribute to reducing the garment collision qualitatively.

- The authors briefly discussed the limitations without showing visual failure examples. In my opinion, illustrating some failure cases (for example, how does the method work on a posed body, or extreme body shapes say very large or small $\beta$? Will the results look bad?) can help to guide future research directions.

- Can the mix-and-match VTO be directly achieved by using cloth simulation? Is it possible to compare the proposed method with its simulation counterparts?

- Minor: Figure 1 could be more informative. For now, it is hard to make sense out of this simple diagram.


questions: - I encourage the authors to add visual results of ablations and failures cases.

- Comparing with pure cloth simulation methods would be better provided.
limitations: The authors have discussed several limitations of the proposed method. I am interested in looking at some visual failure cases and more in-depth analysis.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work presents a novel approach to mix-and-match 3D virtual try-on.  The proposed model is able to untangle layered neural fields (how garments are represented) by introducing a projection operator that works directly on neural implicit representations. The composition of the layered neural field with the neural projection is what the authors refer to as a ULNeF.  At runtime, the model is able to take as input a representation of the body shape and optimize for a layering of the garments by using ULNeFs to resolve collisions.
strengths_and_weaknesses: Strengths:
* Mix-and-match is a very challenging problem in VTO, and the authors demonstrate results on up to 7 layers of garments, which is quite impressive.

* ULNeFs could be applicable to other object-object interaction problems, as the authors point out.

* The neural model introduced that learns the untangling operation results in significant efficiency gains (“up to two orders of magnitude faster” [258-259]).

* The authors provide sufficient details about implementation, such that this work would be reproducible by others.

Weaknesses:
* No comprehensive quantitative evaluation is performed to measure the quality of the mix-and-match results, besides the ablation study of terms and encodings used for training the implicit surface model.  Only a handful of qualitative examples are shown, though the video in the supplemental material shows a compelling interactive demo.

* The authors acknowledge that they have only validated their work on a single pose, using simulated models and clothing.

questions: Are there any failure cases with the ULNeF model? For example, are there any types of clothing with certain features that would prove problematic for this approach?

What would happen if the ordering of the garments did not make sense? For example, if one were to layer the pants on top of the dress?

[159-161] Have the authors performed any experiments with parameterizing additional codes, other than body shape?

limitations: The authors have acknowledged that the proposed method has only been demonstrated on a single pose (T-pose).  The work is based on simulated models, and does not involve real humans or garments, so potential negative societal impact is lessened..
review_title: null

Meta-review: metareview: The reviewers found this paper novel and ambitious (neural virtual try-ons), well written, with good qualitative results, and thought the mix-n-match is a good addition. 
They were concerned about certain exposition parts (section 4.2), and the experimental setup that lacked quantitative evaluation. 
I encourage the authors to take into heart these comments while preparing their revision. 

confidence: Certain
Decision: Accept