TRITON: Neural Neural Textures for Better Sim2Real
Authors: Ryan D Burgert
Jinghuan Shang
Xiang Li
Michael S Ryoo
Venue: CoRL 2022 Poster (2022)
Keywords: differentiable rendering
sim2real
image translation

Abstract: Unpaired image translation algorithms can be used for sim2real tasks, but many fail to generate temporally consistent results. We present a new approach that combines differentiable rendering with image translation to achieve temporal consistency over indefinite timescales, using surface consistency losses and neu- ral neural textures. We call this algorithm TRITON (Texture Recovering Image Translation Network): an unsupervised, end-to-end, stateless sim2real algorithm that leverages the underlying 3D geometry of input scenes by generating realistic- looking learnable neural textures. By settling on a particular texture for the objects in a scene, we ensure consistency between frames statelessly. TRITON is not lim- ited to camera movements — it can handle the movement and deformation of ob- jects as well, making it useful for downstream tasks such as robotic manipulation. We demonstrate the superiority of our approach both qualitatively and quantita- tively, using robotic experiments and comparisons to ground truth photographs. We show that TRITON generates more useful images than other algorithms do. Please see our project website: tritonpaper.github.io

Neural Neural Textures Make Sim2Real Consistent
Ryan Burgert
Jinghuan Shang Xiang Michael S. Ryoo
Department of Computer Science
Stony Brook University
Stony Brook. NY 11794
(rburgert, jishang, (ar.aangnlm.mm.m@e@oo.oo pinp wtw.ocsssssooooroooooooo edu
Abstract: Unpaired image translation algorithms can be used for sim2real tasks, but many fail to generate temporally consistent results. We present a new approach that combines differentiable rendering with image translation to achieve temporal consistency over indefinite timescales, using surface consistency losses and neu- ral neural textures. We call this algorithm TRITON (Texture Recovering Image Translation Network): an unsupervised, end-to-end, stateless sim2real algorithm that leverages the underlying 3D geometry of input scenes by generating realistic- looking learnable neural textures. By settling on a particular texture for the objects in scene, we ensure consistency between frames statelessly. TRITON is not lim- ited to camera movements - it can handle the movement and deformation of ob- jects as well, making useful for downstream tasks such as robotic manipulation. We demonstrate the superiority of our approach both qualitatively and quantita- tively, using robotic experiments and comparisons ground truth photographs. We show that TRITON generates more useful images than other algorithms do. Please see our project website: aitonpaper.github.o
Keywords: sim2real, image translation, differentiable rendering
Introduction
Current sim2real image translation algorithms used for robotics [1, 2] often have difficulty generat- ing consistent results across large time-frames particularly when the objects in an environment are allowed to move. This makes training good robotic policies using such sim2real challenging. In this paper, we discuss an algorithm called TRITON (Texture Recovering Image Translation Network) that combines neural rendering, image translation, and two special surface consistency losses cre- ate surface-consistent consistent translations over frames. We use the term surface-consisentl to refer to the desirable quality of preserving the visual appearance of object surfaces they move, viewed from another angles.
TRITON is applicable when we have simulator capable of generating a realistic distribution of geometric data, but when we not have any information about the surfaces of those objects (which need to render realistic images). For example, in one of our experiments we use robotic arm model provided by the manufacturer that contains no material information, and then learn to render the materials of the arm using TRITON. As opposed to requiring els with meaningful textures, TRITON can generate these from scratch using a set of photographs capturing the distribution of states in simulation, without requiring any matches between domains. That is, given a set of unpaired, unannotated real images and geometry images, TRITON learns the underlying texture of objects and surfaces appearing in the scene without any direct supervision.
expert/artist to create 3D mod-
We introduce the concept of neural neural texture, which is an implicit texture representation having the form of neural network function generating RGB values given surface coordinates. Unlike previous raster-based 'neural textures' to create realistic images [3, 4]. we model surface textures as function instead of discretized pixels. It is continuous implicit pixel-less parametric texture represented by a neural network using Fourier features [5] that takes in 2D UV coordinates and outputs colors. This is similar to how neural radiance fields (NeRF) represents 3D scene in the form of a neural function [6]. The difference is that we focus on learning textures from unpaired training images of very different scene configurations, for viewpoint well object motion synthesis.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Simulated Data:
Photos
Unpaired
Maps
Labels
TRITON
Recovered Textures
Fake Photos
+
+
+
+
This is overview of what TRITON accomplishes. TRITON learns the textures of 3D objects to help translate simulated images into photographs. It does this with unpaired data. Each row is different dataset.
Figure 1:
We conduct multiple experiments to confirm TRITON's advantage prior image translation ap- proaches commonly used for sim2real including CycleGAN [7] and CUT [8]. Importantly, we show the advantages of the proposed approach in real-robot sim2real experiments, learning the textures and training robot policy solely based on the images generated by TRITON.
Related Works
Unsupervised Image to Image and Video to Video Translation Generative models have been very successful in creating images unconditionally [9, 10], creating images conditioned on text [11, 12] and creating images when trained paired data [13]. Unpaired image translation algorithms include [14, 7, 15, 16). Of particular interest are sim2real image to image translation algorithms such as [1, 2, 17, 18]. These algorithms aim to translate simulated images into realistic ones, for various purposes such as data augmentation [1]. Rao al. [2] are used to help train robots. Our paper an architecture based on Pfeiffer et al. [17]. which based on MUNIT [15]. Extending to video to video translation, optical flow is often used for stateful translation to improve temporal coherence [19, 20]. But they do not provide temporal coherence over long time periods.
video game enhancement. In particular, Ho et al.
View Synthesis NeRF [6] has spawned a large body of research on end-to-end view synthesis like [21, 22, 23). Related to work, which also uses geometry based backbone [24, 25, 26). However, all of these works only work with static scenes. There are NeRF-based approaches that work dynamic scenes [27, 28, 29] but none of these for sim2real. Menapace et al. [30] is interactive, but its conditioned on learned actions and not compatible with physics simulators that would be used to train robots. To learn the dynamics in the context of robot learning, self-supervised learning over time is an available approach [31, 32, 33, 34]. Differently, we only use the geometry without temporally aligned samples in those works.
interactive, prohibiting their usage
Neural Textures Thies al. [3] introduced the concept of neural textures and image translators in the context of deferred rendering pipeline, used in other works such as [4] which is closely related our project. Rivoir et [4] differs from TRITON biggest being that it only synthesizes new camera views, where the only difference between each scene the placement of the camera. contrast, our algorithm takes in scenes where several objects are placed in different places or deformed, where one static global 3D model of the environment is sufficient to accomplish our tasks. In addition, TRITON introduces new type of neural texture, "neural neural texture", which is represented implicitly using Fourier feature network [5].
few very important aspects though, the

---

3d Scene
UVL Scene
UV Coordinates
Labels
Bendering
A scene obtained by rendering 3D state into an image, where the first two color channels represent u, coordinates and the third channel / represents object labels.
Figure 2:
Formulation
TRITON's goal is to turn 3D simulated images into realistic fake photographs (by training it without any matching pairs), while maintaining high surface consistency. does this by simultaneously learning both an image translator and a set of realistic textures. These translations can be useful for downstream tasks, especially robotic policy learning from camera data, enabling sim2real.
In contrast to previous works involving neural textures [3, 4], TRITON can handle more than just camera movements: the positions of objects in the training data can be moved around deformed as well between data samples. With high surface consistency, surfaces of translated objects will look the same when moved around viewed from different camera angles.
There are two components in every dataset: a set of simulated 3D scenes and a set photographs. Datasets usually have many more scenes than photographs, since scenes can be generated automat- ically. A photograph is an image tensor, having (r.g. b) values between and We denote it as P E [0, 1 XxX x3 where 3 refers to the three (r.9. 6) channels, and H, W are the height and width.
In this work, scene refers to rendered 3D simulation state which includes the position, rotation, and deformation of every object (including the camera). We represent each 3D state with simple (r, g. b) image which we call UVL scene, which simple enough format that any halfway decent simulator should able to provide. A scene has the same dimensions a photograph and is denoted as s E [0, 11 xW x3, However, unlike p, s's three (r", g, b) channels encode special semantic meaning: (u, v. I) depicted in Figure 2. The channels u and e refer to texture coordinate, whereas / refers to label value that identifies every object
scene. In a given dataset, each object gets a different label. Likewise, each object is assigned different texture. We assume that every dataset has L different label values for L different objects, and that we must learn different neural textures.
Unprojection Consistency Loss: Ix..
Neural Neural
Texture Realism Loss: f,p
Textures: y (. v)
Ridirectional Image-tor Image Translator: T
UVL Scenes: gl..8
Projections: N-N
Fake Photos: p
T A
Adversarial Losses: Lat
Fake UVL Scenes and Projections: (31-., 11..9
Photos: p I.x
T TB+
Figure 3: An overview of TRITON. Inputs the left, and outputs the right. The dashed arrows indicate losses. Although 1_...NN and fi...N look similar, they are not identical Ern encourages them to look as similar possible.

---

256 Random Fourier Peatures:
1. Separate Pixel-wise MLPs: Mi....
Neural Neural Textures: 2.4
uand
Figure 4: Details of calculating neural neural textures in Figure 3. They are fully differentiable, and represented continuously along texture's u and v axes using Fourier feature networks.
Method
In this section, describe how TRITON works to translate images from domain A (simulation) to domain B (photos) while maintaining surface consistency (Figure 3). TRITON introduces a learn- able neural neural texture with two novel surface consistency losses to existing image translator. In the end, TRITON able to effectively generate photorealistic images.
4.1 Neural Neural Texture Projection
Instead of feeding UVL scene directly into the image translator TA-B, we first apply learnable textures & to the object surfaces, which is learned jointly with TA-B.
These neural neural textures represented implicitly by neural network that maps UV values to RGB values. For each texture vi* E vhI...,
v²: E (r,g.b) E
Given a UVL scene 8, we obtain projection x by applying a texture ei E w..... to every pixel (u, v. 1) individually, where texture index i = I [() is decided by the label value of that scene pixel. The projection π, which an intermediate image is computed as:
T|a = (f(......1) $1x.y.2))
where r E [1...W1, W]. E (1... H}. Texture index = I(]) where I = str..33 and the function I(-) scales and discretizes into integers. Subscripts mean multidimensional indexing.
We now discuss the implementation of our neural neural networks vI...,, Each neural neural tex- ture vf is a function consisting of two components: a multi-layer perceptron M' and a static set of 256 random spatial frequencies fi...256 E 256xx (Figure 4). Given two-dimensional (u, v) vector, the spatial frequencies are used to generate high dimensional embedding vector of (u, v): (sin(g1). cos(g1). sin( cos(025))) These embeddings are fed into M', where M' is function 5252 - 3 mapping that embedding to an (r,g,) b) color value:
(u, M ((iin(g1)) cos(g1), sin(g2), cos(g2), ... sin(g2se), cos(g256) )
(3)
where gs fa vu. fa (au. 3a) is two dimensional vector, where au, 8u NN 0, 10) and is the dot product operator. The hyperparamters 256 and 10 are selected empirically.
In practice, we found that TRITON learns faster and more stably with our neural neural textures than does with raster neural textures. The generated textures are also smoother and less noisy. See the supplementary material for more details.
4.2 Image Translation
Our image translation module T bidirectional: TA-B translates sim to real (aka domain to domain B), and T TB-A translates real to sim (aka domain B to domain A).
T TA-B(TT, 8)
and (#. 8) TB-
(4)

---

Enpecjettion Mear:
Serees
Proection
Pake Paeto
UVL Securosaaat
Uie Tomuree
Withest Teetor
WES Techure Realisn Loss
Rxepectonn Take Photee:
Mngrojecton Sul e(o)
Ungrajection
Consictency Laws: p(o(s))-))
.02184
Figure The neural neural texture looks more realistic with texture real- ism loss enabled. Note that the blocks show different letters because of differ- ent initializations and the symmetry of cube both solutions are valid texture assignments.
Figure 5: This is the unprojection consistency loss used in Figure 3. The Unprojection Mean is not used in any losses, but help to illustrate the content in Section 4.4.
is based on the image translation module used in Rivoir et al. [4], which is a based on a variant [17] of MUNIT [15]. T uses the same network architectures MUNIT, and also inherits its adver- sarial losses fadv. cycle consistency losses Leye and reconstruction losses free. The main differences between our image translation model and MUNIT are that we use a different image similarity loss Ω. Like Pfeiffer et al. [17] and Rivoir et al. [4] our style code is fixed (making it unimodal) and noise injected into the latent codes during training prevent overfitting, During inference however, this intermediate noise is removed and our image translation module is deterministic.
MUNIT translates images by encoding both domains A and B into common latent space, then decoding them into their respective domains B and A. In our translation module (Figure 3), we define fake photos p TA, 8) = (TT,)) and fake projection/UVL scenes (ft, (p) = (A(E((()) where EA, En, GA, Gn are encoders and generators for domains A and B respectively.
We define an image similarity loss between two images x, y that returns a score between -1 and 1, where 0 means perfect similarity:
Qz v) L2(xx,)) msssim v)
(5)
This function combines mean pixel-wise L2 distance (used in MUNIT) with multi-scale structural image similarity msssim, introduced in [35]. Note that this loss can also be applied the latent rep- resentations obtained by E and Eg, because like images those tensors are also three dimensional.
We have cycle consistency loss lege Z2 ((T,, 8), TB-A (p)) + Ω (p, TA-N (ft, 8)) similarity loss free = Ω ((T,, s),G,, GA (FA(T. s))) + Ω (p. GB (En (p))) (which is effectively an autoencoder loss), and content similarity loss leon = 52 (EA (T,, 8) EB (f)) + 12 (FA (p), FA adversarial losses ladv that come from two discriminators DA and DB. targeting domains A and B respectively, using the LS-GAN loss introduced by Mao al. [36]. In total, our image translator loss is l = Leye - Eyoc + Loon + Ladu-
FA (#, 8)). We also have
4.3 Unprojection Consistency Loss
To keep the object surfaces consistent, we impose a pixel-wise "unprojection consistency loss" by unprojecting surfaces in fake photos back into the common texture space. Given a UVL scene and its respective fake photo P. the unprojection is obtained from assigning (r, g. b) values at each pixel location (z. y) of P to its corresponding (u, v, 1) coords given by For simplicity, we note
w [u..o]
E P(za)
s.t. (u, v. $[r.v)
where the expectation E means we aggregate multiple (r.gg, b) vectors being assigned the same (u, v, coordinate by averaging them. In practice, (u, v, are real numbers between [0, 1], where as each have to be rasterized, i.e., represented by L images with a size of (D D 3), where is the number of labels and D x D is the resolution of the unprojection. Therefore, we discretize and scale corresponding (u, v, 1) to integers so that each (r,g, g, b) vector will assigned to a pixel on w. For the exact implementation, please see our supplementary material.
We obtain N unprojections {a/ IN from a batch of N UVL scenes and fake photos. The un- projection consistency loss is defined as the per pixel-channel standard deviation of w over the batch
5

---

(input)
(Gurs)
Only
w.duca Fre
Cyc
Figure 7: Image translation comparisons showing how TRITON has better temporal consistency. The first column is the UVL map as the input. The other columns are results using different image translation methods. Each row of images shows different random placement of objects in the scene. TRITON consistently outputs high quality results over different object arrangements. "TRITON Texture-Only" stands for TRITON without two surface consistency losses lvc and lTR-
1 luc LXDxDX3
L,,,00
where o--) stands for the standard deviation function and E (1...3} is the channel index of w. We minimize tye encourage unprojections be consistent across the batch. Intuitively, if tve were O, it would mean the object surfaces in translations p appear exactly the same in every scene. In addition, we call the mean of unprojections p( p((w>")) }==1) as "Recovered Textures", visualzed in Figure and Section 5.
Texture Realism Loss
To encourage the neural textures look realistic possible, we try to make the projections look like the final output by introducing "texture realism" loss ÉTR. fTR is an image similarity loss that makes look like its translation p.
LTR = (
(8)
is the image similarity from Equation 5. Without ÊTR, & can look wildly different each time you train it. As seen in the top of Figure 6, the neural texture looks very unrealistic the colors completely arbitrary. By adding texture realism loss εyR, we make the textures 1 more realistic. In practice, this makes TRITON less likely to mismatch the identity of translated objects.
Experiments
In this section, we perform two quantitative experiments: the first measures TRITON's accuracy directly, and the second measures it's sim2real capability real-world setting. Please see the appendix for more interesting experiments!
5.1 Datasets
We constructed two datasets AlphabetCube-3 and RobotPose-xAmm to benchmark different image translators in many perspectives. In our setting, each dataset is composed of two sets of unpaired images: UVL scenes from a simulator and real photographs. Note that the images all datasets are unpaired and UVL scenes only rely rough 3D models of the objects in the scene without need of precisely aligning objects the simulator to the real world. We use AlphabetCuhe-3 for image translation quality evaluation, and RobotPose-xAL for sim2real policy learning evaluation.

---

5.2 Image Translation Evaluation on AlphabetCube-3
AlphabetCube-3 dataset features a table with three different alphabet blocks on The dataset con- tains 300 real photos and 2000 UVL scenes from a simulator. Each photo features these three cubes with a random position and rotation. In this section, we evaluate the image translation accuracy of TRITON and other image translation methods using AlphahetCube-3. In this dataset, 7. = because there are four different textures: three textures the three cubes and one for the table.
Figure shows qualitative results on the dataset. From Figure we find that TRITON consistently outputs high quality results over different object arrangements. The textures keep aligned even when the position of blocks change dramatically over multiple scenes (See examples in green rectangles in Figure 7). In contrast, though MUNIT [15] and CycleGAN [7] manage generate realistic images, the surfaces of the cubes are either consistent over scenes (See examples in the red rectangle) or replicated by mistake (See examples in the orange rectangle). Also note that the floor background of the outputs from CycleGAN are quite different from the ground truth. CUT [8] fails to generate meaningful images regarding both the foreground blocks and the background.
In quantitative experiments shown Table 1, we manually align the simulator with 14 photos of different real world scenes and generate the UVL maps for translation. We measured the LPIPS [37] and 12-norm between the translated images and the real images using multiple contigurations. 'Masked' means we mask the background (which is the floor in AlphabetCube-3 dataset) and only measure the translation quality w.r.t three foreground blocks. For the 'unmasked' tests, we compare the whole image without any masking instead, and the background contributes much more to the losses due to its larger area. Similar to the qualitative results, TRITON consistently outper- forms other methods under different metrics and configurations. Further ablations TRITON w/o sb, &uc,, f and TRITON w/o Luc, f show the importance of the new losses we introduce.
Table 1: Quantitative results on AlphabetCube-3. LPIPS [37] and 12-norm between the translated images and the real images are reported. We exclude backgrounds in the 'Masked' configuration. TRITON consistently outperforms other methods in different metrics and configurations
Masked
Unmasked
LPIPS (x10-1) L2 (x10-
LPIPS (x10-1) L2 (x1002)
CycleGAN CUT
0.450
0.559
6.02
4.25
0.469
0.553
5.40
6.37
TRITON w/o bs Evc, frn
0,437
0.500
4.34
4.25
TRITON w/o Epc, FTR
0.442
0.586
2.85
3.64
TRITON
0.286
0.479
1.17
1.23
5.3 Sim2real Transfer for Robot Learning
In this section demonstrate how TRITON improves sim2real transfer for robot policy learning. To this end, we first train TRITON on a dataset consist of unpaired UVL scenes from a robot simu- lator (Gazebo) and photos from the real robot. Once TRITON is trained, learn the robot policy while only utilizing the simulator. The input of the policy is the fake photo ý translated from the sim-
Trained and Fixed
Seen:
TA-B
Train
Policy
Evaluate
Simulator
TRITON
Fake Photos
Real World
Train
Evaluste
TA-B
Policy
Simulator New Viewpoint
World New Viewpoint
Figure 8: Sim2real framework of using TRITON. We evaluate in two settings: Seen and Unseen. In Unseen, we use new camera viewpoint to train and evaluate the policy. This new viewpoint is not used for training TRITON.

---

TRITON TRITON wo y. (UC, MR CycleGAN CUT
TRITON TRITON wio EJC, (TR
CycleGAN CUT
Arguler (dea)
Anguler Ener (deg)
25
10
20
84x84
168x168
336x536
Seen
Unsoon
(8) Comparison across baselines and resolutions
(b) Transfer to unseen viewpoints
Figure 9: Evaluation results of sim2real robot learning. We compare TRITON against two baselines and one TRITON variant w/o v, &EFC, LTR across different input image resolutions. TRITON outperforms CycleGAN [7] and CUT [8] consistently across (a) all resolutions and (b) unseen view- points. Results in (a) are from Seen setting only. Results in (b) are from 336x336 resolution.
ulated UVL scene using TRITON's translator TA-BB- Finally, the trained policy is directly evaluated on the real robot, taking real photos p as input.
Ideally, a good image translation model makes ý similar to p, so that the policy trained on synthesized photos will transfer to real domain seamlessly and will have better performance. All our policies are learned with zero real-world robot interaction with the environment.
Task We use robot pose imitation task for our experiments. Given the input of photo of real robot pose, the policy outputs robot controls replicate that target pose. We measure the angular error between the replicated and target poses our evaluation metrie: v E,(ā, a,)², where âj and aj are replicated and target joint angles respectively, and j is the joint index. We use an xArm robot which has seven joints. We also attached patterns and tapes on the robot to make its texture more challenging to model. Robot policy and Baselines Since the sim2real formulation allows benefiting from abundant training data using the simulator, we use behavioral cloning to train a good robot policy. The policy network is CNN similar to [38]. We compare TRITON against two baseline image translation methods: CycleGAN [7] and CUT [8], by replacing TRITON with each baseline method respectively in the above pipeline. The robot policy learning method is same for all the methods.
Evaluation Settings We introduce two main evaluation settings, Seen and Unseen. In the Seen setting, the policy is trained and tested using the same camera viewpoint that is used during TRITON (or baseline) training. Whereas in Unseen setting, we introduce camera at a new viewpoint for training and testing the policy. That is, the image translator has to generate the fake photos out of its training distribution (seen viewpoints), which becomes a challenging task for the translator. We also vary input image resolutions to show the power of photo-realistic images in higher resolutions.
Results Figure 9 shows the evaluation results of the sim2real transfer. Both Seen and Unseen evalu- ations show that TRITON outperforms baselines consistently. In Figure 9(),, find that TRITON continuously improves from low to high resolutions due to its ability to generate more photo-realistic images compared to the baselines. From Figure 9(),, we confirm that TRITON outperforms the oth- ers also in the Unseen setting, showing that TRITON learns image translations that better generalize to new viewpoints. TRITON outperforms TRITON w/o th, Luc, PTR consistently except at the low- est resolution. This again shows the effectiveness of v. LvC and ęTR to generate photo-realistic images, and such high quality images are important in higher resolutions for sim2real transfer.
In Appendix in the supplementary material, we provide more experimental results.
Limitations
TRITON relies on 3D models of objects in order to generate photo-realistic images, making it less applicable when the geometry of an environment is unknown. Acquiring such 3D models could be challenging the wild, especially in robots allowed to roam the world. TRITON not able to handle transparent objects, because the UVL format is opaque. Generated shadows, though visually realistic, are not physically accurate. Moreover, the probability of incorrectly matching textures to objects increases as the number of object classes increases.
8

---

Acknowledgments
We thank Srijan Das and Kanchana Ranasinghe for valuable discussions. This work is supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Ministry of Science and ICT (No.201 8-0-0020,, Development of Core Technology of Robot Task-Intelligence for Improvement of Labor Condition. This work also supported by the National Science Foundation (IIS-2104404 and CNS-2104416).

---

References
[1] D. Ho, K. Rao. Z. Xu, E. Jang, M. Khansari, and Y. Bai. RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer. arXiv, 2020.
[2] K. Rao. C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari. RL-CycleGAN: Rein-
forcement Learning Aware Simulation- To-Real. 2020 JEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), 00: 00:11154-11163 2020. 100:100.10900cy222600000000 01117. RL-CycleGAN.
[3] J. Thies, M. Zollhöfer, and M. Nießner. Deferred Neural Rendering: Image Synthesis using Neural Textures. arXiv, 2019.
[4] D. Rivoir, M. Pfeiffer, R. Docea, F. Kolbinger, C. Riediger, J. Weitz, and S. Speidel. Long- Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data. arXiv, 2021.
[5] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil. N. Raghavan, U. Singhal, R. Ra- mamoorthi, J. T. Barron, and R. Ng. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. arXiv, 2020.
[6] B. Mildenhall, P. Srinivasan, M. Tancik, J. '. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the Euro- pean Conference Computer Vision (ECCV), 2020.
[7] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. arXiv, 2017.
[8] T. Park, A. A. Efros, R. Zhang, and J. Zhu, Contrastive learning for unpaired image-to-image translation. CoRR, abs/2007. 2020. URL http:p////axxivoor///aaaaa
15651.
[9] I. J. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks, 2014. URL https://arxiv org/abs/ /bs//
1406. 2661.
[10] Karras, S. Laine, and Aila. A style-based generator architecture for generative adversarial networks, 2018. URL https://aaxta org/abs/ 1112/11111.1 1812.04944..
[11] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation, 2021. URL https: / wwwpriiiv..//raeee org/abs/212.. 12092.
[12] C. Saharia, W. Chan, S. Saxena, L. Li, Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes. T. Salimans, J. Ho, J. Fleet, and M. Norouzi. Photorealistic text- to-image diffusion models with deep language understanding, 2022. URL https://arxx. org/ abs/ /205. 11487.
[13] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks, 2016. URL pattps://arxxs http:///aaxvv.oo/aa///////////// wtt./or/////////////////// 07004.
[14] X. Su, Song, C. Meng, and Ermon. Dual Diffusion Implicit Bridges for Image-to-Image Translation. arXiv, 2022.
[15] X. Huang, M. Y. Liu, S. Belongie, and J. Kautz. Multimodal Unsupervised Image-to-Image Translation. arXiv, 2018
[16] M.-Y. Liu, T. Breuel, and Kautz. Unsupervised Image-to-Image Translation Networks. arXiv, 2017. UNIT.
[17) M. Pfeiffer, I. Funke, M. R. Robu, S. Bodenstedt, I.. Strenger, S. Engelhardt, T. RoB, M. J. Clarkson, K. Gurusamy, B. R. Davidson, L.. Maier-Hein, Riediger, T. Welsch, Weitz, and S. Speidel. Generating large labeled data sets for laparoscopic image processing tasks using unpaired image-to-image translation. arXiv, 2019.
[18) S. R. Richter, H. A. AlHaija, and Koltun. Enhancing photorealism enhancement. arXiv:2105.0461. 2021.
10

---

[19] M. Chu, Y. Xie, J. Mayer, L. Leal- Taixé. and N. Thuerey. Learning temporal coherence via self-supervision for GAN-based video generation. ACM Transactions on Graphics (TOG). 39 (4):75:1-75:13, 2020. ISSN 0730-0301. doi: 10.1 141143336559933392555
[20] L. Amsaleg, B. Huet, M. Larson, G. Gravier, H. Hung, C.-W. Ngo, W. T. Ooi, Y. Chen, Y. Pan, T. Yao, X. Tian, and T. Mei. Mocycle-GAN. Proceedings of the 27th ACM International Conference Multimedia, pages 647-655, 2019. doi:10.1 14511101145/333311..3333 133559977.
[21] P. P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and T. Barron. NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis. arXiv, 2020.
[22] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin. FastNeRF: High-Fidelity Neural Rendering 200FPS. arXiv,, 2021.
[23] C. Lin, W. Ma, A. Torralba, and S. Lucey. BARF: bundle-adjusting neural radiance fields. CoRR, abs/2104.06400, 2021. URL hwtth:t//aaxiiiiii http::///arxi..or//bbbs...
wwtpg/abs/210..0000
[24] G. Riegler and V. Koltun. Free View Synthesis. arXiv, 2020.
[25] G. Riegler and V. Koltun. Stable View Synthesis. arXiv, 2020.
[26] J. Y. Zhang, G. Yang, S. Tulsiani, and D. Ramanan. NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild. arXiv, 2021.
[27] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[28] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla. Nerfies: Deformable neural radiance fields. ICCV, 2021.
[29] Y. Du, Y. Zhang, H.-X. Yu, J. B. Tenenbaum, and J. Wu. Neural radiance flow for 4d view synthesis and video processing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[30] W. Menapace, S. Lathuilière, A. Siarohin, C. Theobalt, S. Tulyakov, V. Golyanik, and E. Ricci. Playable Environments: Video Manipulation in Space and Time, arXiv, 2022.
[31] Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. Time- contrastive networks: Self-supervised learning from video. In IEEE International Conference on Robotics and Automation (ICRA), pages 1 114-1144. IEEE, 2018.
[32] J. Shang and M. S. Ryoo. Self-supervised disentangled representation learning for third-person imitation learning. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 214-221, 2021.
[33] J. Shang, S. Das, and M. S. Ryoo. Learning viewpoint-agnosti visual representations by recovering tokens in 3d space. In Advances in Neural Information Processing Systems, 2022.
[34] Y. Li, S. Li, V. Sitzmann, Agrawal, and A. Torralba. 3d neural scene representations for visuomotor control. In Conference on Robot Learning, pages 1 112-123. PMLR, 2022.
[35] J. Snell, K. Ridgeway, R. Liao, B. D. Roads, M. C. Mozer, and R. S. Zemel. Learning to Generate Images with Perceptual Similarity Metrics. arXiv, 2015.
[36] X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley. Least Squares Generative Adversarial Networks. arXiv, 2016.
[37] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The Unreasonable Effectiveness of Deep Features as Perceptual Metric. arXiv, 2018
[38] V. Mnih, K. Kavukeuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning, 2013. arXiv:1312.5602.
11


Reviews:
[{"review_id": "x1Hydu-mo4", "reviewer": "Reviewer_agLh", "summary": "This paper proposed a 3D to 2D, learned rendering pipeline that can adapt the textures given a set of images. The learned rendering pipeline can then be used to generate synthetic dataset for robotic sim2real transfer.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n### Strengths\n**Clarity**\n\nI think the figures are very illustrative and beautiful.\n \n**Quality**\n- The proposed method is technically sound.\n- The limitation section is honest and complete.\n \n### Weaknesses\n \n**Missing Citation**\n\nThe following paper uses 3D model + domain randomization to achieve camera-to-robot pose estimation is highly relevant as it has a similar real-world demo.\n- [1] Camera-to-Robot Pose Estimation from a Single Image, ICRA 2020\n \n**Baselines**\n\nI think it’s unfair to compare the proposed method with CycleGAN and CUT as they don’t have access to 3D scenes. It would be more suitable to evaluate against baselines that actually use the 3D information, e.g. [1] listed above. \n\n**Significance**\n\n- Since the focus on this paper is sim2real, I think the real-world experiments are very important. However, the real-world experiments presented in this paper have low significance because a) it’s not a real task that roboticists encounter often and b) there are methods which can solve this problem, e.g., [1]. I believe unique, novel robotic abilities enabled by TRITON in the real world are lacking.\n- For image translation evaluation, I think the AlphabetCube-3 dataset may be too simple (as the shape is regular and the object itself is rigid). Additional results on non-rigid (e.g., deformable) objects will make the task more challenging. \n\n**Clarity**\n- The name of each method shown at the top of Figure 7 is blurry.\n- Line 96, the image caption and main text are too close to each other.\n- I am not sure why the proposed method has two Neural in its name.\n\n**Typo**\n- Line 34, Important -> Importantly\n\n- Line 108, neural neural networks -> neural networks\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend weak rejection because a) I think comparing to baselines that can only access 2D information is not fair, b) the paper focuses on sim2real but the real-world experiments are not convincing.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- Why do authors name the method “Neural Neural Texture”? Why are there two “Neural”?\n- Line 35, *”learning the textures and training a robot policy solely based on the images generated by TRITON.”* \n  Does this mean TRITON learns the textures from images generated by itself, or this is a typo?\n- Line 73, *”the positions of objects in the training data can be moved around or deformed as well between data samples”*\n  Do authors have results on deformable objects?\n"}}, {"review_id": "rPv8IDRwFV4", "reviewer": "Reviewer_9MxW", "summary": "The authors propose a new method TRITON that can transform textures from real images to synthetic scenes in an unpaired fashion. The approach requires UVL presentations for sim2real texture transfer. The authors conducted experiments to show the strength of the approach in robotic experiments and image translation evaluation. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n+ The presented approach is technically interesting. The technical solution is clearly well-written.\n+ There are clear improvements with the presented approach over other methods such as CycleGAN and CUT.\n+ There are hardware experiments to demonstrate the strength of the proposed method. \n\nWeaknesses\n- There is no analysis or experiment about temporal consistency. Although the authors claim that the approach can ensure consistency between frames statelessly.\n- I think the presented approach can *not* handle cases with shadows. The inherent representation limits such a possibility. If this is the case, this should be mentioned in the limitation section.\n- I think MUNIT is also a baseline for comparisons. Can authors also evaluate this method?\n- The approach requires UVL representations for the 3D scenes, which make it less generally applicable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the approach is interesting, but more experiments and discussions are needed. \n\nThanks for the rebuttal. It basically addresses my concerns. I will keep my score.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please address my concerns above. \n\nIs it a typo in the title? \"Neural Neural\"?", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "Un0vW47XrN1", "reviewer": "Reviewer_z1oF", "summary": "This paper presents TRITON, a novel algorithm for generating textures for images to aid in sim2real transfer. Unlike previous methods, TRITON claims not only to maintain surface consistency across camera positions, but object positions and deformations. \n\nThe method is as follows: a \"neural neural\" texture projection, where scenes are mapped to textures that are learned and also represented as neural networks. The textures are applied to image pixels to obtain the projections. These are given to an image-to-image translator to get synthetic but realistic photos. The image translator can also turn real photos into synthetic photos. Losses require that projected surfaces transformed back into the texture space maintain consistency, that projections look like images, and other standards. \n\nExperiments include testing image translation and sim2real. TRITON outperforms cycleGAN alone and CUT on image translation, and improves performance on a pose imitation sim2real transfer task. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Paper is fairly well-presented. There are many moving parts and it's certainly not a simple method, but each part seems well-motivated and is explained clearly.\n- The approach is creative, nonobvious, and novel to my knowledge\n- Evaluation is posed well. It is helpful to separate out image translation specifically, even though the primary claim is sim2real, because the algorithm structure is complicated enough to be a bit mystifying if all we learn about is the output. I appreciate the investigation in the middle of the overall pipeline. \n\n**Weaknesses**\n- Explanation of GAN-related losses gets very confusing. It's less problematic because they are partially taken from prior work, but the final paragraph of 4.2 could be much clearer.\n- It would be interesting to get more intuition/evidence for why representing the texture as a neural function is beneficial, i.e. why it creates a performance gain. It's not hard to discern but I think intuition into new methods is always worth talking about.\n- I realize that sim2real is expensive, but more testing would be more convincing. Baselines, which might be easier to run, would potentailly be even more helpful.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this paper successfully designs and validates a sophisticated method for generating textures that can aid in sim2real transfer. While a more robust experimental/baseline suite would help, the claims w.r.t. novel object positions and deformations seem compelling. Ultimately I don't have much domain knowledge about texture generation, so I find it somewhat hard to gauge this work against its peers, but for sim2real it does seem like a useful result with good execution. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {"issues": "- More experimentation if possible, along with analysis of what the results say about the intuition behind this approach\n- Baselines if possible\n- Clarity around some of the equations, especially the losses"}}, {"review_id": "MUHNdIMr1L", "reviewer": "Reviewer_uVwa", "summary": "The authors propose a new method called TRITON in the domain of image translation tasks. Given a real-world photograph, the model can learn the texture of each object and translate a simulated scene into a photorealism image. The textures of objects are generated using differentiable rendering parametrized by neural networks (named neural neural texture). Specifically, the input (u,v map coordinates) of neural rendering goes though a fourier feature network in a high dimensional space, which is parametrized afterwards with $L$ MLPs and the outputs are pixel-wise rgb value. Moreover, in order to maintain the temporal consistency of objects between frames, the authors propose a surface consistency loss. Another texture realism loss and adversarial loss are included to close the gap between simulator and real-world. The experiments show that TRITON can surpass prior work CycleGAN and CUT by a large margin.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n1. The authors combine fourier network and neural rendering, which gives continuous and smooth representations over the UV space for the texture rendering. The authors also claim that it is faster and more stable compared to raster neural textures.\n2. To keep the consistency of object surfaces, the authors propose a novel loss, which unprojects surfaces of a batch of fake images to a common texture space. Simply reducing the standard deviation of the pixel-wise $w$ imposes the surface consistency.\n3. The authors create two datasets to evaluate their method, namely AlphabetCube-3 and RobotPose-xArm. In particular, the experiments on RobotPose-xArm demonstrate the image translator on unseen camera viewpoint. The generated video looks promising and could potentially help sim2real policy learning and alleviate human efforts in labelling.\n\n**Comments**: \n1. Does $L$ also include the background, e.g. $L$ in AlphabetCube-3 is 3 or 4? \n2. Can you elaborate on more details of the RobotPose-xArm dataset, e.g. how many photographs, simulated scenes, $L$ and training details?\n2. I think TRITON is only working on known objects in the training dataset. Is that possible to render textures of unseen objects (e.g. duck which is not shown during training) given an existing label (e.g. the label of \"apple\") using TRITON? Or it might fail because of different geometric features.\n3. In Fig. 1 row 2, we see multiple different objects. I think you already tested on them, why not include their results in the experimental section but only AlphabetCube-3? This is a critical missing point in my opinion.\n4. Do you think $L$ could be variable? Currently, the number of MLPs is implied by the number of objects. Therefore, TRITON is not scalable given large numbers of different objects.\n5. I think the UVL label along with the value of $L$ are still a strong prior knowledge since they reveal the information between photographs and scenes, which is limited because in practice, predefined object labels sometimes are not accessible. It would be interesting to see an extended work: new images can be generated given UVL scenes by learning the textures from photographs, where the objects in scenes and photographs are different or even the objects in scenes are unknown during training. \n\n**minor**: \nA margin space is missing between the caption of fig.4 and the following paragraph.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall I think this work is well-motivated and technically sound. The results look promising which support potential applications for robot learning, e.g., imitation learning or policy learning. Some minor changes could be made to improve the clarity, e.g., more details about the experiments on RobotPose-xArm, an experimental results on various objects like in fig.1 row 2 instead of only cubes.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "1. The experimental results on various shapes of objects are missing, e.g., like the example showed in fig.1 row 2&3.\n2. The details of experiments on RobotPose-xArm. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)