Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Paper Summary**
The paper proposed a method to learn the geometric details of clothing mesh via perturbing texture coordinates. The paper first trains multiple networks (each network is for one specific camera view) to predict the texture coordinates in the corresponding camera view, then recover 3D shapes from multiple camera views. 

**Strength**
1. Using texture sliding as a proxy to learn 3D geometry is an interesting idea
2. The paper also spends many efforts on designing texture sliding methods for occlusion boundary, which is hard for geometry processing. 

**Weakness**
1. The paper claims to prevent overfitting induced from regularization and claim the proposed strategy can recover high-frequency details via procedurally embedded low-frequency data, however, the proposed method is specifically designed for 3D meshes (texture perturbation), it's not clear how does the method work with other machine learning representations, e.g. 2D image. At least, from my point view, this is not a trivial extension, and the paper didn't show the experiments beyond the 3D clothing mesh. I would suggest the authors turn down the claims in the paper. 

2. The experiments on 3D reconstruction is not well enough. The paper argues to learn the geometric details, thus a careful analysis of the reconstructed 3D geometry (especially emphasizing on the geometric details) would be more convincing. However, the paper only presented one example of the final result (Fig.12/13), it would be better to: a) show more qualitative results, b) show quantitative comparisons with more baselines, e.g. a network that directly predicts the vertex offset in the clothing mesh. c). the quantitative analysis on the geometric details, how good the method can capture the details comparing with the baselines. 

3. The scope of the paper is narrow. It's not trivial to extend the pipeline in the paper for a general machine learning representation. Also, in the paper, recovering 3D shapes requires post-processing and smoothing to make the shape look good.

**Overall**
Overall, Even though the idea of texture sliding is interesting, the experiments in this paper didn't demonstrate the clear improvements in recovering 3D geometric details. so I vote for a reject initially. 
review_title: Interesting idea, but the experiment is not good enough with many limitations

final_score: rating: 3: Clear rejection
review_content: This paper presents a general approach to embed high frequency information into low-frequency data with a particular focus on improving the performance of virtual clothing. To address over-smoothing issues in the predicted meshes, authors proposed the texture sliding method that changes texture coordinates on each camera through the deep networks. The texture sliding neural network (TSNN) is trained using the ground truth offset computed for each camera and pose. 

* Pros
1) The proposed TSNN leads to the performance gain in the virtual clothing

* Cons
1) The current manuscript requires major revisions by addressing the following concerns.
- It is hard to understand what to convey in Section 3.1. Especially, the following sentence just states the problem without how to address it.
‘This assumes linearity, which is only valid when the triangles are small enough to capture the inherent nonlinearities in a piecewise linear sense; moreover, folds and wrinkles can create significant nonlinearity.’
- It is difficult to follow what authors intend to convey in Figure 2.
- Two methods in Section 3.2 are hard to interpret. 
- It is needed to mark both camera view and pose in Figure 4.
- Algorithm stating the overall procedure of the proposed method would improve the readability.
- What is 'UV space' in Section 5?
- How did you estimate the joint angles theta_k?
- What data did you use for training/validating/testing the networks?

2) Comparative study seems to be insufficient. It seems that a more thorough performance analysis is needed.
review_title: It seems that this work needs substantial revisions by providing more details in a concise manner. Experimental results lack thorough comparative study with recent methods.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes an approach to encoding high frequency wrinkles into lower frequency texture coordinates, dubbed texture sliding. 

At its core, is the idea of perturbing a predicted texture so that the rendered cloth mesh appears to more closely match the ground truth from a camera view point. 
This is contrast to previous methods, which were attempting to make the network produce high-frequency components directly.
Once texture perturbations are recovered from at least two unique camera views, 3D geometry can then be reconstructed to recover high-frequency wrinkles.

Positive:

-The approach can be used to improve the quality of different architectures, although it requires laborious data pre-processing and to re-train a network for every specific architecture one wants to refine.

-Experimental results show the benefit of the proposed post-processing technique.


Negative:

-In the manuscript, it is mentioned that is necessary to train a separate network predicting texture perturbation for each camera. This makes the proposed pipeline cumbersome and computationally intensive. Why not conditioning prediction both on human pose and camera view point?

-The approach is claimed to be applicable to a general setting in the manuscript ("We focus on the specific task of adding highfrequency wrinkles to virtual clothing, noting that the idea of learning a low-frequency embedding may be generalized to other tasks”). However, both network architecture and the presented experiments are tailored to solve a very specific problem (getting wrinkles in pre-defined template t-shirt right). Unless other applications are presented, I find the authors claim to be misleading.


Clarifications: 

-Are we assuming to have access to ground truth texture for the experiments? If so, this should be stated more clearly.

-Lifting slid textures to 3D vertex perturbations seems a non-trivial operation to me. Some math would have helped making the paper easier to read and more set-contained.

Justification of recommendation:

Although experimental results clearly show the benefit of the proposed method, I find several components of the pipeline to be to cumbersome to be used in practice (specifically: data re-generation per method, network re-training per method, network training per camera, relying on ground texture). Moreover, the proposed method feels more like a work-around to make things look better with clothing prediction, rather than an attempt to trying to solve the actual problem, i.e. learning high frequency information with neural networks. 

Post author response:
After having carefully read the author's response and additional reviews, I confirm my original recommendation.   
review_title: RECOVERING GEOMETRIC INFORMATION WITH LEARNED TEXTURE PERTURBATIONS

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a method to correct high-frequencies details in the textures of animated clothes. The main idea is to train a network to learn the 2D offset in the UV space for a given pose and view. The paper shows results using a t-shirt, on interpolating to novel views and 3D reconstruction by correcting the output of a state of the art method.

PROS
1) Comparison:
The paper improves a state-of-the-art method (Jin et al. 2020), suggesting that the proposed module could be applied by other methods as post-processing.

2) Applicability:
I think the idea could be easily applicable in other tissues domains (e.g. soft-tissue modeling for humans, animals fur). 

CONS
1) Method and Experiments:

- I think the proposed methodology is not particularly innovative, and it acts in a controlled scenario: the method is a CNN developed to work in a (virtual) setup that requires several prior knowledge: the view (also, it need to be trained for every different one), the pose parameters (making it also dependent on the animation system), the garment, the texture, and the light condition. Finally, the shown examples and experiments are limited to a few setups, without stressing the limits of the method.
I am wondering how does the method performs training a different kind of garments with higher mobility (e.g. skirts) or subject to non-isometric deformations (e.g. zip clothes) or with irregular textures at higher frequencies (e.g. a shirt with a clear logo). 
- I am not surprised that the method improves the starting one, and I would suggest to compare it with others refinement baselines (e.g. PCA, or nearest neighbor in the training set). Also, the proposed subdivision step seems harmful in Table 2 (and not helpful in Sup.Mat. Table 1), and it is unexpected to me: I think would be good including a comment in the text.
- Finally,  I understand that the focus of the paper is in a specific computer graphic domain, but while it is not the precise scope of the paper, I think would be interesting also to show the method limits when some of these priors elements are estimated for real-world cases (e.g. using pose regression techniques to infer the parameters). If this method can be applied also in real scenarios it would be really interesting.

2) Presentation:
I had some problems following the train of thought in some passages. I think it can be clearer by:
- Highlighting the contribution in the introduction, to clearly state the novelty of the work
- Provide a more structured taxonomy of the previous works for the reader. Also, I would suggest remarking the difference between the most important related papers and the proposed method.
- Grouping Data information in a single section (at the moment they are divided into Sections 4 and 6.1)
- Moving hyperparameters details to Supplementary Materials
- Moving the Methods with the Architecture (likely after the data section)

FINAL RATING
I think there is room to make the paper stronger; even if the paper proposes a simple method, with more evaluations, experiments, and analysis on different (and more challenging) examples it could find interest in the graphic community and future works.
review_title: An ok paper, that could be strengthened

Meta-review: The 4 reviewers all had a consistent view of this paper:  concern that the scope of the work was overstated (paper claims, without evidence, to apply in more generality than the 1 example scenario shown); concern about the difficulty of implementing this approach (1 TSNN required for each rendered viewpoint); and lack of examples showing how the method performs under more challenging scenarios.

The AC encourages the authors to revise the work in response to the reviews.  That would involve additional experimentation and examples, and some attention to revising the manuscript.   After two of the reviewers complained of lack of clarity in the algorithm description, the authors replied, "We explain our algorithm in the paper; the reader can refer to our code for implementation details."  I hope the authors can be more responsive to the readers' concerns than that in their revisions.
Decision: Reject