Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents TRITON, a novel algorithm for generating textures for images to aid in sim2real transfer. Unlike previous methods, TRITON claims not only to maintain surface consistency across camera positions, but object positions and deformations. 

The method is as follows: a "neural neural" texture projection, where scenes are mapped to textures that are learned and also represented as neural networks. The textures are applied to image pixels to obtain the projections. These are given to an image-to-image translator to get synthetic but realistic photos. The image translator can also turn real photos into synthetic photos. Losses require that projected surfaces transformed back into the texture space maintain consistency, that projections look like images, and other standards. 

Experiments include testing image translation and sim2real. TRITON outperforms cycleGAN alone and CUT on image translation, and improves performance on a pose imitation sim2real transfer task. 
strengths_and_weaknesses: **Strengths**
- Paper is fairly well-presented. There are many moving parts and it's certainly not a simple method, but each part seems well-motivated and is explained clearly.
- The approach is creative, nonobvious, and novel to my knowledge
- Evaluation is posed well. It is helpful to separate out image translation specifically, even though the primary claim is sim2real, because the algorithm structure is complicated enough to be a bit mystifying if all we learn about is the output. I appreciate the investigation in the middle of the overall pipeline. 

**Weaknesses**
- Explanation of GAN-related losses gets very confusing. It's less problematic because they are partially taken from prior work, but the final paragraph of 4.2 could be much clearer.
- It would be interesting to get more intuition/evidence for why representing the texture as a neural function is beneficial, i.e. why it creates a performance gain. It's not hard to discern but I think intuition into new methods is always worth talking about.
- I realize that sim2real is expensive, but more testing would be more convincing. Baselines, which might be easier to run, would potentailly be even more helpful.  
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think this paper successfully designs and validates a sophisticated method for generating textures that can aid in sim2real transfer. While a more robust experimental/baseline suite would help, the claims w.r.t. novel object positions and deformations seem compelling. Ultimately I don't have much domain knowledge about texture generation, so I find it somewhat hard to gauge this work against its peers, but for sim2real it does seem like a useful result with good execution. 
issues: - More experimentation if possible, along with analysis of what the results say about the intuition behind this approach
- Baselines if possible
- Clarity around some of the equations, especially the losses
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposed a 3D to 2D, learned rendering pipeline that can adapt the textures given a set of images. The learned rendering pipeline can then be used to generate synthetic dataset for robotic sim2real transfer.

strengths_and_weaknesses: 
### Strengths
**Clarity**

I think the figures are very illustrative and beautiful.
 
**Quality**
- The proposed method is technically sound.
- The limitation section is honest and complete.
 
### Weaknesses
 
**Missing Citation**

The following paper uses 3D model + domain randomization to achieve camera-to-robot pose estimation is highly relevant as it has a similar real-world demo.
- [1] Camera-to-Robot Pose Estimation from a Single Image, ICRA 2020
 
**Baselines**

I think it’s unfair to compare the proposed method with CycleGAN and CUT as they don’t have access to 3D scenes. It would be more suitable to evaluate against baselines that actually use the 3D information, e.g. [1] listed above. 

**Significance**

- Since the focus on this paper is sim2real, I think the real-world experiments are very important. However, the real-world experiments presented in this paper have low significance because a) it’s not a real task that roboticists encounter often and b) there are methods which can solve this problem, e.g., [1]. I believe unique, novel robotic abilities enabled by TRITON in the real world are lacking.
- For image translation evaluation, I think the AlphabetCube-3 dataset may be too simple (as the shape is regular and the object itself is rigid). Additional results on non-rigid (e.g., deformable) objects will make the task more challenging. 

**Clarity**
- The name of each method shown at the top of Figure 7 is blurry.
- Line 96, the image caption and main text are too close to each other.
- I am not sure why the proposed method has two Neural in its name.

**Typo**
- Line 34, Important -> Importantly

- Line 108, neural neural networks -> neural networks

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend weak rejection because a) I think comparing to baselines that can only access 2D information is not fair, b) the paper focuses on sim2real but the real-world experiments are not convincing.
issues: - Why do authors name the method “Neural Neural Texture”? Why are there two “Neural”?
- Line 35, *”learning the textures and training a robot policy solely based on the images generated by TRITON.”* 
  Does this mean TRITON learns the textures from images generated by itself, or this is a typo?
- Line 73, *”the positions of objects in the training data can be moved around or deformed as well between data samples”*
  Do authors have results on deformable objects?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose a new method TRITON that can transform textures from real images to synthetic scenes in an unpaired fashion. The approach requires UVL presentations for sim2real texture transfer. The authors conducted experiments to show the strength of the approach in robotic experiments and image translation evaluation. 
strengths_and_weaknesses: Strength
+ The presented approach is technically interesting. The technical solution is clearly well-written.
+ There are clear improvements with the presented approach over other methods such as CycleGAN and CUT.
+ There are hardware experiments to demonstrate the strength of the proposed method. 

Weaknesses
- There is no analysis or experiment about temporal consistency. Although the authors claim that the approach can ensure consistency between frames statelessly.
- I think the presented approach can *not* handle cases with shadows. The inherent representation limits such a possibility. If this is the case, this should be mentioned in the limitation section.
- I think MUNIT is also a baseline for comparisons. Can authors also evaluate this method?
- The approach requires UVL representations for the 3D scenes, which make it less generally applicable.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I think the approach is interesting, but more experiments and discussions are needed. 

Thanks for the rebuttal. It basically addresses my concerns. I will keep my score.
issues: Please address my concerns above. 

Is it a typo in the title? "Neural Neural"?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose a new method called TRITON in the domain of image translation tasks. Given a real-world photograph, the model can learn the texture of each object and translate a simulated scene into a photorealism image. The textures of objects are generated using differentiable rendering parametrized by neural networks (named neural neural texture). Specifically, the input (u,v map coordinates) of neural rendering goes though a fourier feature network in a high dimensional space, which is parametrized afterwards with $L$ MLPs and the outputs are pixel-wise rgb value. Moreover, in order to maintain the temporal consistency of objects between frames, the authors propose a surface consistency loss. Another texture realism loss and adversarial loss are included to close the gap between simulator and real-world. The experiments show that TRITON can surpass prior work CycleGAN and CUT by a large margin.
strengths_and_weaknesses: **Strengths**:
1. The authors combine fourier network and neural rendering, which gives continuous and smooth representations over the UV space for the texture rendering. The authors also claim that it is faster and more stable compared to raster neural textures.
2. To keep the consistency of object surfaces, the authors propose a novel loss, which unprojects surfaces of a batch of fake images to a common texture space. Simply reducing the standard deviation of the pixel-wise $w$ imposes the surface consistency.
3. The authors create two datasets to evaluate their method, namely AlphabetCube-3 and RobotPose-xArm. In particular, the experiments on RobotPose-xArm demonstrate the image translator on unseen camera viewpoint. The generated video looks promising and could potentially help sim2real policy learning and alleviate human efforts in labelling.

**Comments**: 
1. Does $L$ also include the background, e.g. $L$ in AlphabetCube-3 is 3 or 4? 
2. Can you elaborate on more details of the RobotPose-xArm dataset, e.g. how many photographs, simulated scenes, $L$ and training details?
2. I think TRITON is only working on known objects in the training dataset. Is that possible to render textures of unseen objects (e.g. duck which is not shown during training) given an existing label (e.g. the label of "apple") using TRITON? Or it might fail because of different geometric features.
3. In Fig. 1 row 2, we see multiple different objects. I think you already tested on them, why not include their results in the experimental section but only AlphabetCube-3? This is a critical missing point in my opinion.
4. Do you think $L$ could be variable? Currently, the number of MLPs is implied by the number of objects. Therefore, TRITON is not scalable given large numbers of different objects.
5. I think the UVL label along with the value of $L$ are still a strong prior knowledge since they reveal the information between photographs and scenes, which is limited because in practice, predefined object labels sometimes are not accessible. It would be interesting to see an extended work: new images can be generated given UVL scenes by learning the textures from photographs, where the objects in scenes and photographs are different or even the objects in scenes are unknown during training. 

**minor**: 
A margin space is missing between the caption of fig.4 and the following paragraph.


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall I think this work is well-motivated and technically sound. The results look promising which support potential applications for robot learning, e.g., imitation learning or policy learning. Some minor changes could be made to improve the clarity, e.g., more details about the experiments on RobotPose-xArm, an experimental results on various objects like in fig.1 row 2 instead of only cubes.
issues: 1. The experimental results on various shapes of objects are missing, e.g., like the example showed in fig.1 row 2&3.
2. The details of experiments on RobotPose-xArm.
review_title: null

Meta-review: metareview: Scores: z1oF: Weak Accept, agLh: Weak Reject, 9MxW: Weak Accept, uVwa: Weak Accept
 
Quality: The paper provides a novel algorithm for generating temporally consistent sim2real transfers, which is an important contribution. The authors showed  in baseline comparisons and an ablation study that the proposed methods outerperforms existing SoTA algorithms. However, the video seems to show that the learned policy has low precision for the reaching task (see Cons).
 
 Clarity: The paper has been improved in the revision. New experiments with rigid and soft objects and an additional baseline have been included. The term “neural neural texture” has been clarified.
 
 Originality: The paper is interesting, and the proposed method is novel and creative.
 
Significance: Generating temporally consistent sim2real transfers are significant for robot learning, and the presented method provides a technically sound approach to address this important problem.
 
 Pros: 

- The problem is well-motivated, and the approach is novel and creative.
 
 Cons: 

 - The video ((https://youtu.be/IwAs420hY6A) seems to show that the learned policy has low precision for the reaching task (reaching for the apple at the time 0:10, pepper at the time 0:55, and apple again at the time 2:02). Can this be improved?
- The approach requires UVL representations for the 3D scenes, which may make it less applicable.
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)