Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a GAN model for virtual try-on trained from unpaired image data. The main network is based on StyleGAN2. During the training stage, for each image, the extracted 2D skeleton pose is used to convert the parsed garment region into a set of rectified texture patches with the proposed Patch-routed disentanglement Module. These rectified patches are encoded to be a single vector fed into every stage of the generator. The head region and the 2D skeleton pose are encoded to be identity features as input to the generator. Because the detailed texture information is not well preserved in style patches, the generator first synthesize coarse-level image and garment mask. The garment region is further fused in later stage of the generator to synthesize the final output. The model is trained in a reconstruction way without using paired person images. During the inference stage, the garment region is warped by the target pose as input into the texture synthesis stage of the generator. The key idea is its patch-based garment disentangling. The part-wise patch rectification removes pose and shape related information but only keeps low-level texture information. Results are reported on DeepFashion and MPV and self-collected UPT datasets with comparisons to state-of-the-art methods. 
main_review: Pros:
1) It is an interesting idea of garment disentangling by rectifying the corresponding garment region on each skeleton bone into a standard square patch. The rectification process makes sure the resulting patches are normalized in terms of pose and shape. It is also the key to enable unpaired learning.
2) The StyleGAN2 based generator is well constructed. Having a separate texture synthesis branch with warped garment as input makes sure the generator can synthesis high-quality texture with the aid of spatially-adaptive residual module. 
3) It presents impressive visual results. Its high-quality synthesis is also verified with significant improvements in FID scores and user preferences compared to the related works. 

Cons:
1) The proposed patch-routed disentanglement module is based on 2D skeleton pose, which may be only feasible to normalize/rectify garment w.r.t. in-plane pose changes. The presented visual results are all about near frontal views, which seem to agree with this observation. The visual results sometimes look 3D implausible, especially when large patterns/logos/designs are present in the garment. In a recent work by Kripasindhu Sarkar et al. (StylePoseGAN), the DensePose is used to extract UV texture map, which is meant to be 3D pose invariant. How would this paper compare with DensePose-based disentangling?
2) In Line 140, it'd be better to explain how to come up with the numbers of valid patches for different kinds of garments. 
3) It'd be also desirable to generate higher-resolution results, 512x512. 
limitations_and_societal_impact: Limitations and societal impact are mostly addressed. It'd be better to provide additional results for viewpoint limitation resulted from 2D skeleton pose as mentioned above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on virtual try-on based on unpaired images. The authors propose to disentangle the style and spatial information of each garment, and generate it under the condition of homography transformed normalized patches. The authors also introduce a new unpaired dataset. Compared with the paired and unpaired state-of-the-art methods, both quantitative and qualitative results prove the effectiveness of the proposed approach.
main_review: The proposed pose-guided patch-routed disentanglement module is interesting, and the method is technically reasonable. Detailed ablation studies have proven the effectiveness of each design choice. The authors also propose a new dataset that can facilitate further research. Both quantitative and qualitative results are promising. I also appreciate the detailed descriptions in the supplemental material. In general, I am inclined to accept. I have some concerns about the experiment and the dataset, and I want to hear the authors' response to re-evaluate my score.
limitations_and_societal_impact: - In Table 1, I am surprised that the proposed method performs very well on the UPT dataset (~7.8 FID, compared to 21.58 in DeepFashion). Other methods show similar performance on the two datasets. It is better to discuss the root cause here. I am a little worried that the proposed design choices are particularly suitable for UPT, but they may not always be very helpful in other datasets.
- It is better to have an apple-to-apple comparison to further demonstrate the advantages of the proposed dataset. Figure 1(b) in the supplement is a good example. I want to know how the 25 clothing categories are distributed in the new dataset compared to the other two. Since the authors have proposed a new dataset, it is better to display detailed statistics to convince readers of the benefits of the dataset.
- Figure 5, 1st row, left example. It seems that the proposed method smoothed out the black belt. Is it because of the error in the predicted garment mask? It's better to discuss.
- It is better to show the running time compared with other methods, because the speed of inference is also a consideration for virtual try-on.
- Minor issue: L40, the main text is too close to the caption of the figure 1.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper creates realistic virtual try-on results by modifying the StyleGAN2 architecture and using the garment appearance as the style modulation. In particular, to preserve the details of the target garment, the garment patches are warped and used as reference for the output image. The proposed method significantly outperforms the existing baselines thanks to the photorealism brought by preserving the input patches. The improved quality is backed by FID, user study, and analysis via ablations. 
main_review: I am generally satisfied with the clarity of writing, several insights and conceptual contributions, practicality of the application, and evaluation. 

While the synthesis models based on neural networks produce nice results, it is very challenging to use it for retaining the precise details and patterns of the input image in the process of image manipulation, such as logos on t-shirts. For example, many GAN inversion works (Image2StyleGAN Abdal et al, 2020, StyleGAN encoder Richardson et al., CVPR 2021) can't embed complex patterns on hats (see Figure 11 of https://arxiv.org/pdf/2008.00951.pdf). The proposed method nicely handles this by first warping the garment to retain the details, and then using deep networks to refine the final output. I think this combination of non-parametric sampling and parametric approach will inspire future works in image manipulation using deep networks. 

The writing is clear, and the authors comprehensively listed the details of the method, including the precise network architecture, insights (L175-181), details about implementing baselines (L271-281), failure cases and ethical concern. 

The evaluation clearly supports the improved quality of the proposed method, both qualitatively and quantitatively. The FID is significantly better, likely because the input patches are recycled in the output. 

Having said that, here are some weaknesses and suggestions.

Resolution is 256px. Can it be higher?

The quality of the final output depends on the quality of the warped garment. Therefore, when the pose change is too drastic to find a nice warp, the quality of the output will degrade as in Figure 3 of Supp Mat. Likely because of this reason, most results of the submitted paper are frontal-facing pose. Still, I think this limitation is not important, as it is reasonable to assume that the target garment will contain at least one frontal facing picture in the virtual try-on task. 

The human evaluation protocol could be performed at larger scale, because 30 volunteers may not represent unbiased population. 

The labeling of ablated models was too difficult to follow. Could you use more intuitive names? Such as “-TSB, intact-garment”.  

It dismisses a line of work [19, 27, 26, 25] because their methods require images of the same person in different poses. However, this may not be a harsh requirement on the fashion dataset, as these types of images can be readily found on the Internet. In particular, the authors already evaluate their method on the DeepFasion dataset, and StylePoseGAN[25] shows nice results on this dataset. Ideally, some side-by-side comparison to the SOTA methods such as [25] would be nice, while clearly stating that the training requires different amount of supervision. This is merely a suggestion for future revisions, since [25] is still only on arxiv. 

In particular, it does share some similarity to StylePoseGAN[25]. Both inherit StyleGAN2 and modify it to a conditional version. The garment style is used as style modulation. The garment style information is normalized to some canonical pose in both works. Considering StylePoseGAN is a recent work and still only on Arxiv, similarity to StylePoseGAN does not affect my rating, but it would be nice to further discuss similarities and differences, because the authors already cited the work. 

limitations_and_societal_impact: The authors discussed both positive and negative implications of the work in Conclusion.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: This paper presents a human garment try-on pipeline. The key point of the proposed approach is to normalize each body part into a regular patch according to the body skeleton, and use this patch representation to extract the style and disentangle the spatial variation due to different body poses. The network is based on StyleGAN architecture, and adapted to accept style and pose conditions. Furthermore, to overcome the loss of the texture details, it also proposes to use the first-stage network to generate the clothed mask, and use a refining network to synthesize/inpaint the texture details based on the deformed patches aligned to the target skeleton.
main_review: It is not new to have a canonical representation for body appearance and garment to neutralize the pose variations, including using 3D UV space like DensePose or 2D patches like this paper. This paper does not provide enough insight why this simple patch extracting and normalization is better than previous solutions. Also, the normalization step is confusing, by mapping a rectangle (expanded vector between two joints) to a patch, a simple closed-form rotation+scaling should suffice, why there is a need to do least-squares optimization to get the matrix?

Regarding the network architecture, it looks pretty similar to the StylePoseGAN work (Style and Pose Control for Image Synthesis of Humans from a Single Monocular View). Although I guess it is still an unpublished work, why that method can produce better quality results without the need for the texture synthesis branch as the second stage? And in the texture branch, the feature inpainting method does not sound very convincing to me. What if the warped patches are not accurate, and how would it be to simply ask the network to generate the image with the warped patches as the soft condition input?

And my biggest concern is on the training losses. Why is it reasonable to have reconstruction loss between the source image and the result? They wear the same garment, but the body poses are not aligned.

---------------------

Addition to the final review:

I would like to thanks the authors for participating in the long discussion. While the discussion helped achieve a better understanding of most technical details, I am not fully convinced that this paper is ready for acceptance. My major concerns still exist:
1. The way how self-supervised training is done in the proposed method introduces a huge training-testing gap that is not negligible. The proposed augmentation approach does help improve robustness against occlusion but still suffers from unseen mismatches and other defects that are not seen during training. Please see the detailed comments.
2. The way how the paper handles patch cropping and normalization/deformation is presented as is. Many choices should be justified: Why optimization is needed for normalization? Why the proposed way to generate the cropping patch is the best? Why the patch does not cover the whole garment? But the paper does not provide enough experiments to convince the reader.
3. Lack of significant technical breakthrough. The proposed method is somehow similarly studied by other works on cloth try-on or human retargeting. It does have some interesting ideas, but I am concerned that how these ideas can really inspire follow-up works.
limitations_and_societal_impact: Most of the limitations are listed in the main review. Including unclear technical details and lack of explanation of the proposed network design.
review_title: null

Meta-review: The paper was reviewed by four expert reviewers in the community. Most reviewers appreciate the novelty on unpaired training for virtual try-on (although paired data is not a critical restriction in this problem domain). The conditional extension of StyleGAN2 is somewhat similar to StylePoseGAN. All reviewers are aware that StylePoseGAN is an unpublished work so the reviewer assessment of this paper is not affected by an arXiv paper. There were an extensive discussions between the authors and Reviewer 5Zj3. The AC reads the reviews, authors' rebuttal, and the discussions. While there are still some clarifications required for the method exposition and several other limiting factors (e.g., low resolution results only, similar approach as concurrent work), the AC thinks the paper has sufficient merits and could inspire future research work. The AC thus recommends to accept.
Decision: Accept (Poster)