TUVF: Learning Generalizable Texture UV Radiance Fields
Authors: An-Chieh Cheng
Xueting Li
Sifei Liu
Xiaolong Wang
Venue: ICLR 2024 poster (2024)
Keywords: texture synthesis
generative model
NeRF
3D AIGC
Primary area: generative models
TL;DR: Given a 3D shape, TUVF can synthesis realistic, high-fidelity, diverse, and 3D consistent textures.

Abstract: Textures are a vital aspect of creating visually appealing and realistic 3D models. In this paper, we study the problem of generating high-fidelity texture given shapes of 3D assets, which has been relatively less explored compared with generic 3D shape modeling. Our goal is to facilitate a controllable texture generation process, such that one texture code can correspond to a particular appearance style independent of any input shapes from a category. We introduce Texture UV Radiance Fields (TUVF) that generate textures in a learnable UV sphere space rather than directly on the 3D shape. This allows the texture to be disentangled from the underlying shape and transferable to other shapes that share the same UV space, i.e., from the same category. We integrate the UV sphere space with the radiance field, which provides a more efficient and accurate representation of textures than traditional texture maps. We perform our experiments on synthetic and real-world object datasets where we achieve not only realistic synthesis but also substantial improvements over state-of-the-arts on texture controlling and editing.

Published as a conference paper at ICLR 2024
TUVF: LEARNING GENERALIZABLE TEXTURE UV RADIANCE FIELDS
An-Chieh Cheng' Xueting Li Sifei Liult Xiaolong Wang't UC San Diego "NVIDIA
Disentangled Texture Synthesis with Diverse Samples
Texture Editing
Texture Transfer through Dense Correspondence
We propose Texture UV Radiance Fields (TUVF) to render 3D consistent texture given a 3D object shape input. TUVF provides a category-level texture representation disentangled from 3D shapes. Top three rows: TUVF can synthesize realistic textures by training from collection of single-view images; Fourth row: Given a 3D shape input, we can render different textures on top by using different texture codes; Bottom row: We perform editing on given texture (adding flag of France) and directly apply the same texture on different 3D shapes without further fine-tuning. Note that all samples rendered under 1024 x 1024 resolution; zoom-in recommended.
Figure 1:
ABSTRACT
Textures vital aspect of creating visually appealing and realistic 3D models. In this paper, we study the problem of generating high-fidelity texture given shapes 3D assets, which has been relatively less explored compared with generic 3D shape modeling, Our goal is to facilitate controllable texture generation process, such that one texture code can correspond a particular appearance style independent of any input shapes from category. We introduce Texture UV Radiance Fields (TUVF) that generate textures a learnable UV sphere space rather than directly on the 3D shape. This allows the texture to be disentangled from the underlying shape and transferable to other shapes that share the same UV space, i.e., from the same category. We integrate the UV sphere space with the radiance field, which provides a more efficient and accurate representation of textures than traditional texture maps. We perform our experiments where we achieve not only realistic synthesis but also substantial improvements over ofate-oftte the-arts texture controlling and editing. We show more qualitative results at https:/ / //www/ iechenn me /TUV..
synthetic and real-world object datasets
Equal advising. Codes and trained models are publicly available

---

Published as a conference paper at ICLR 2024
INTRODUCTION
3D content creation has attracted much attention given its wide applications in mixed reality, digital twins, filming, and robotics. However, while most efforts in computer vision and graphics focus 3D shape modeling (Chabra et al., 2020; Zeng et al., 2022; Cheng et al., 2023), there is less emphasis on generating realistic textures (Siddiqui et al., 2022). Textures play a crucial role in enhancing the immersive experiences in virtual and augmented reality. While 3D shape models are abundant in simulators, animations, video games, industry manufacturing, synthetic architectures, etc., rendering realistic and 3D consistent texture on these shape models without human efforts (Figure first three rows) will fundamentally advance the visual quality, functionalities. and experiences.
Given instances of one category, ideally. their textures should be disentangled from their shapes. This can be particularly useful in scenarios where the appearance of an object needs to be altered frequently, but the shape remains the same. For example, it is common in video games to have multiple variations of the same object with different textures to provide visual variety without creating entirely new 3D models. Thus, the synthesis process should also be controllable, i.e., we can apply different textures to the exact shape (Figure 1 fourth row) or use the same texture code for different shapes and even edit part of the texture (Figure 1 bottom row). Recently, the wide utilization of GANs (Goodfellow et al., 2014) allows training on 3D content creation with only 2D supervision (Nguyen- Phuuc et al., 2019; Siddiqui et al., 2022; Skorokhodov et al., 2022; Chan al., 2022). While this alleviates the data and supervision problem, the learned texture representation often highly depends on the input geometry, making the synthesis process less controllable: With the same texture code or specifications, the appearance style of the generated contents changes based on the geometric inputs.
We propose novel texture representation, Texture UV Radiance Fields (TUVF), for high-quality and disentangled texture generation on given 3D shape, i.e., sampled texture code represents a particular appearance style adaptable to different shapes. The key to disentangling the texture from geometry is to generate the texture in a canonical UV sphere space instead of directly on the shape. We train the canonical UV space for each category via a Canonical Surface Auto-encoder in self-super manner so that the correspondence between the UV space and the 3D shape is automatically established during training. Unlike traditional UV mesh representation, TUVF does not suffer from topology constraints and can easily adapt to continuous radiance field.
Given a texture code, we first encode with a texture mapping network to a style embedding, which is then projected onto the canonical UV sphere as textured UV sphere. Using correspondence, can assign textures to arbitrary 3D shapes and construct a point- based radiance field. Consequently, we sample the points along the ray and around the object shape surface and render the RGB image. In contrast to volumetric rendering (Drebin et al., 1988; Mildenhall et al., 2020), our Texture UV Radiance Field allows efficient rendering and disentangles the texture from the 3D surface. Finally, we apply an adversarial loss using high-quality images from the same category.
We train our model on two real-world datasets (Yang et al., 2015; Park al., 2018a), along with synthetic dataset generated by our dataset pipeline. Figure 1 visualizes the results of synthesizing 3D consistent texture given a 3D shape. Our method can provide realistic texture synthesis. More impor- tantly, our method allows complete texture disentanglement from geometry, enabling controllable synthesis and editing (Figure 1 bottom two rows). With the same shape, we evaluate how diverse the textures can be synthesized. With the same texture, we evaluate how consistently it can be applied across shapes. Our method outperforms previous state-of-the-art significantly on both metrics.
2 RELATED WORK
Neural Radiance Fields. Neural Radiance Fields (NeRFs) have been widely studied on broad applications such as high fidelity novel view synthesis (Mildenhall al., 2020; Barron al., 2021; 2023) and 3D reconstruction (Wang et al., 2021; Yariv et al., 2021; Zhang et al., 2021). Following this line of research, the generalizable versions of NeRF are proposed for faster optimization and few-view synthesis (Schwarz et al., 2020; Trevithick & Yang, 2021; Li et al., 2021a; Chen et al., 2021; Wang et al., 2022: Venkat et al., 2023). Similarly, TUVF is trained in category-level and learn across instances. However, instead of learning from reconstruction with multi-view datasets (Yu et al., 2021a; Chen et al., 2021), I), our method leverages GANs for learning from 2D single-view image collections. From the rendering perspective, instead of performing volumetric rendering (Drebin et al., 1988), more efficient rendering techniques have been applied recently, including surface rendering (Niemeyer et al., 2020; Yariv et al., 2020) and rendering with point clouds (Xu et al., 2022;
2

---

Published as a conference paper at ICLR 2024
Yang et al., 2022; Zhang et al.,, 2023b). Our work relates to the point-based paradigm: Point-Nerf (Xu et al., 2022) models a volumetric radiance field using a neural point cloud; Neu-Mest (Yang et al., 2022) proposes point-based radiance field using mesh vertices. However, these approaches typically require densely sampled points and are optimized for each scene. In contrast, TUVF only requires sparse points for rendering and is generalizable across scenes.
Texture Synthesis on 3D Shapes. Texture synthesis has been an active research area in computer vision and graphies for a long time, with early works focusing on 2D image textures (Cross & Jain, 1983; Taubin, 1995; Efros & Leung, 1999) and subsequently expanding to 3D texture synthesis (Turk, 2001; Bhat et al., 2004; Kopf et al., 2007). Recently, learning-based methods (Raj et al., 2019; Siddiqui et al., 2022; Foti et al., 2022) combined with differentiable rendering techniques (Liu et al., 2019; Mildenhall et al., 2020) have shown promising results in texture synthesis on 3D shapes by leveraging generative adversarial networks (GANs) (Goodfellow et al., 2014) and variational autoencoders (VAEs) (Kingma & Welling, 2013). These paradigms have been applied to textured shape synthesis (Pavllo al., 2020; Gao et al., 2022; Chan et al., 2022) and scene completion (Dai et al.. 2021; Azinović et al., 2022). Motivated by these works, we also adopt GANs to supervise a novel representation for 3D texture synthesis. This allows our model to train from a collection of single-view images instead of using multi-view images for training.
Texture Representations. Several mesh-baset methods (Oechsle et al., 2019; Dai et al., 2021; Yu et al., 2021b; Chen et al., 2022: Siddiqui et al., 2022: Chen et al., 2023; Yu et al., 2023) have been pro- posed. AUV-Net (Chen et al., 2022) embed 3D surfaces into a 2D aligned UV space using traditional UV mesh; however, they requires shape-image pairs as supervision. Texturify (Siddiqui al., 2022) use 4-RoSy fields (Palacios & Zhang, 2007) to generate textures on a given mesh. However, the texture representation entangled with the input shape, and the style can change when given different shape inputs. Our approach falls into the NeRF-based methods (Chan et al., 2022; Skorokhodov et al., 2022). The tri-plane representation has been widely used in these methods. However, these methods often face similar problem in structure and style entanglement. NeuTeex (Xiang et al., 2021) provides an explicit disentangled representation. However, the representation designed for single scene. Our TUVF representation disentangles texture from geometry and is generalizable across instances, which allows transferring the same texture from one shape to another.
Disentanglement of Structure and Style. The disentanglement of structure and style in generative models allows better control and manipulation in the synthesis process. Common approaches to achieve disentanglement include using Autoencoders (Kingma & Welling, 2013; Kulkarni et al.. 2015; Jha et al., 2018; Mathicu et al., 2019; Liu et al., 2020; Park et al., 2020; Pidhorskyi et al., 2020) and GANs (Chen et al., 2016; Huang et al., 2018; Karras et al., 2019; Singh et al., 2019; Nguyen- Phuuce et al., 2019; Chan et al., 2021). For example, the Swapping Autoencoder (Park et al., 2020) learns disentanglement by leveraging network architecture bias and enforcing the texture branch of the network to encode co-occurrent patch statistics across different parts of the image. However, these inductive biases do not ensure full disentanglement, and the definition of disentanglement itself is not clearly defined. In the second paradigm with adversarial learning. StyleGAN (Karras et al., 2019) learns separate mappings for the structure and style of images, allowing for high-quality image synthesis with fine-gra fineggainedd control over image attributes. Recently, CoordGAN (Mu et al., 2022) shows that it is possible to train GANs and pixel-wise dense correspondence can automatically emerge. Our work leverages GANs to provide supervision training, but instead of disentangling texture from 2D structures, we are learning the texture for 3D object shapes.
3 TEXTURE UV RADIANCE FIELDS
We introduce Texture UV Radiance Fields (TUVF) that generate a plausible texture UV representa- tion conditioned on the shape of given 3D object. Semantically corresponding points on different instances across the category are mapped to the same locations on the texture UV, which inherently enables applications such as texture transfer during inference. As shown in Figure 2, our texture synthesis pipeline begins with a canonical surface auto-encoder (Section 3.1) that builds dense corre- spondence between canonical UV sphere and all instances in a category. Such dense correspondence allows us to synthesize textures on shared canonical UV space using a coordinate- based generator (Section 3.2). Finally, since we do not assume known object poses for each instance, we render the generated radiance field (Section 3.3) and train the framework with adversarial learning (Section 3.4).
3

---

Published as a conference paper at ICLR 2024
Canonical Surface Auto-encoder
Texture UV Radiance Fields Rendering
(Section 3.1)
(Section 3.3)
Volume Rendering
Surface Points
Indicator Function
c
Surface Normals
Texture Feature Generator
UA N Ponuulo
Real Fake?
(Section
Real Sämple Rendend Sample
Patch-based Adversarial Learning
W
Texture Feature
Weoe
(Section 3.4)
Figure 2: Method overview. We perform two-stage training: (i) We first train the Canonical Surface Auto-encoder (Equation 6), which learns decoders fe and ge (8) predicting the coordinates and normals for each point on the UV sphere, given an encoded shape. (ii) We then train the Texture Feature Generator he which outputs textured UV sphere. We can construct a Texture UV Radiance Field with the outputs from fe. ga, and hg, and render an RGB image as the output. We perform patch-based generative adversarial learning (Equation 7) supervise hig
3.1 CANONICAL SURFACE AUTO-ENCODER
The key intuition of this work is to generate texture shape-independent space, where we resort to a learnable UV space containing dense correspondences across different instances in a category. To this end, we learn canonical surface auto-encode, that maps any point a canonical UV sphere to a point on an object's 's surface (Cheng et al., 2021; 2022). Specifically, given a 3D object with point O, we first encode its shape into a geometry code Zgea E R by encoder & (Cheng et al., 2021). For point p on the canonical UV sphere, we feed the concatenation of its coordinates Xp and the geometry code into implicit function fe (Figure 2
predict the coordinates of the mapped point p'. denoted as Xp,, on the given object's 's surface. We further predict the normal Np at p' with a separate implicit function go (Figure 2 D). The overall process can be denoted as follows:
Zgeo
{())
Xp = Jo =geo),
Np go Xxrr Ageo)
(2)
The coordinates and normal of p' are then used for the rendering process discussed in Section 3.3.
We use a graph-based point encoder following DGCNN (Wang et al., 2019) and decoder architecture following (Cheng et al., 2022) for fo and go. As proved by (Cheng et al., 2021), correspondences emerge naturally during training, and fo and ge are trained end-to-end using Chamfer Distance (Borge- fors, 1988) on the surface points and the L2 losses the indicator grid discussed in Section 3.4.
3.2 TEXTURE FEATURE GENERATOR
The canonical UV sphere defines dense correspondences associated with all instances in a category. Thus, shape-independent textures can be formulated as generating texture features on top of this sphere space. To this end, we introduce CIPS-UV, an implicit architecture for texture mapping function he (Figure 2 Specifically, CIPS-UV takes 3D point on the canonical sphere X,, and a randomly sampled texture style vector žter ~ (0, 1) as inputs and generates the texture feature vector сソ Rd at point p, which are further used for rendering discussed in Section 3.3. The style latent is injected via weight modulation, similar to StyleGAN (Karras et al., 2019). We design ho based on the CIPS generator (Anokhin et al., 2021). where the style vector žtez is used to modulate features at each layer. This design brings two desired properties. First, combined with the canonical UV sphere, we do not require explicit parameterization, such as unwrapping 2D. Second, it does not include operators (e.g., spatial convolutions (Schmidhuber, 2015), up/downsampling, or self-atte atteeetttiin (Zhang al., 2019)) that bring interactions between pixels. This important because nearby UV coordinates may not correspond to exact neighboring surface points in the 3D space. As result, our generator can better preserve the 3D semantic information and produce realistic and diverse textures on the UV sphere. Please refer to Appendix I. for implementation details.
4

---

Published as a conference paper at ICLR 2024
3.3 RENDERING FROM UV SPHERE
Efficient Ray Sampling. Surface rendering is known for its speed, while volume rendering is known for its better visual quality (Oechsle et al., 2021). Similar to (Oechsle et al., 2021; Yariv et al., 2021; Wang al., 2021), we take advantage of both to speed up rendering while preserving the visual quality, i.c., we only render the color of a ray on points near the object's 's surface. To identify valid points near the object's surface, we start by uniformly sampling 256 points along ray between the near and far planes and computing the density value oy (discussed below) for each position 24. We then compute the contribution (denoted as w;) of x; to the ray radiance as
{-1
W/= = Tí, a = - (r T; = =eppp- Eaydr)
(3)
where ō the distance between adjacent samples. If w. = 0, then x; is an invalid sample (Hu et al., 2022) and will not contribute to the final ray radiance computation. Empirically, we found that sampling only three points for volume rendering is sufficient. It is worth noting that sampling o alone is also fast since the geometry is known in our setting.
Volume Density from Point Clouds. We discuss how to derive continuous volume density from the Canonical Surface Auto-encoder (Section 3.1), which was designed to manipulate discrete points. Given a set of spatial coordinates and their corresponding normal derived from fø and ga. we use the Poisson Surface Reconstruction algorithm (Peng et al., 2021) to obtain indicator function values over the 3D grid. We then retrieve the corresponding indicator value dpsr(z,) for each location x; via trilinear interpolation. dpsr(z;) is numerically similar to the signed distance to the surface and can serve proxy for density in volume rendering. We adopt the formulation from VoISDF (Yariv et al., 2021) 1) to transform the indicator value into density fields a by:
(xx)) Sigmoid( dpsr(r:))
(4)
Note that the parameter "y controls the tightness of the density around the surface boundary and is learnable parameter in VoISDF. However, since our geometry remains fixed during training, we used a fixed value of y = -4.
Point-based Radiance Field. To compute the radiance for a shading point 2;, we query the K nearest surface points D'JENx in the output space of the Canonical Surface Auto-encoder and obtain their corresponding feature vector CjeNK by hgg We then use an MLPp, following (Xu et al., 2022), to process pair-wise feature between the shading point 2; and each nearby neighboring point, expressed as Cj.z. = M P/j x;2). Next, we apply inverse distance weighting to normalize and fuse these K features into one feature vector Cr. for shading point 2;₁:
Co2 Σ JENK
Pj Σej
where Pj -
(5)
1p'
Then another MLPC to output final color value for point x; based Cj,z, and optional viewing direction d, denoted c, = LLP((c,, θ d). We design MLP, and MLPo to be shared across all points, i.e., as implicit functions, that they do not encode local geometry information. Finally, following quadrature rules (Mildenhall et al., 2020; Max, 1995), we render the pixel C(r) with points {x,li- 1, ,.... N} along the ray C(r) = E. Tiacc,
3.4 GENERATIVE ADVERSARTAL LEARNING
Patch-based Discriminator. NeRF rendering is expressive but can be computationally expensive when synthesizing nigh-resolution images. For GAN- generative NeRFs, using 2D convolutional discriminators that require entire images as inputs further exacerbates this challenge. Thus, in our work, we adopt the efficient and stable patch-discriminator proposed in EpiGRAF (Skorokhodov et al., 2022). During training, we sample patches starting from a minimal scale, covering the entire image in low resolution. As the scale gradually grows, the patch becomes high-resolution image crops. As our rendering process is relatively lightweight (see Section 3.3), we use larger patches (128 x 128) than those used in EpiGRAF (64x64), which brings better quality.
Training Objectives. We train the Canonical Surface Auto-encoder (Section 3.1) and the Texture Generator (Section 3.2) in separate stages. In stage-1, we adopt a Chamfer Distance between the output and input point sets, and a L2 loss to learn the mapping dpsr(.) between points and volume density, aforementioned:
LOSAE LeD(p. vh) IDrsR x1x - x
(6)
5

---

Published as a conference paper at ICLR 2024
Texturify
EpiGRAF
TUVF
Figure 3: Qualitative comparison. TUVF achieves much more realistic, high-fidelity, and diverse 3D consistent textures compared previous approaches. Each column also presents results generated using the same texture code. Texturify and EpiGRAF both have entangled texture and geometry representations, which occasionally result in identical global colors or similar local details despite having different texture codes. Visualized under 1024x 1024 resolution; zoom-in is recommended.
where (x', x) denotes the predicted and ground truth indicator function (see details in (Peng et al., 2021)). In stage-2, with I denotes rendering, enforce non-saturating GAN loss with R1 regularization to train the texture generator (Karras et al., 2019; Skorokhodov et al., 2022):
LGAN (L Xp)))) + Es (f(ea((reat)) + AlI (reat))))))))) [[3] (7) where f(u) llog( + exp (-u)).
4 EXPERIMENTS
4.1 DATASETS
CompCars & Photoshape. We used 3D shapes from ShapeNet's "chair" and "car" cate- gories (Chang et al., 2015). For the 2D datasets. we employed Compcars (Yang et al., 2015) for cars and Photoshape (Park et al., 2018b) for chairs. Notably, the chair category includes subsets with significantly different topologies, such as lounges and bean chairs, where finding a clear cor- respondence may be challenging baselines' performance on the "straight chair" category, one the largest subsets the chair dataset. For fair comparisons, we follow Texturify, splitting the 1,256 car shapes into 956 for training and 300 for testing. We apply the same split within the subset for the chair experiment, yielding 450 training and 150 testing shapes. We also provide the evaluation of full Photoshape in Appendix F.
for humans. Consequently, we evaluate model and the

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative Results on CompCars. The symbol "f" denotes an instance-specifie approach,
whereas the remaining methods employ category-wis training. Our method significantly improves over all previous methods on all metries. KID is multiplied by 10?.
Method
IPIPS, f LPIPS:
FID 4
KID +
TexFields (Oechsle et al., 2019)
177.15
17.14
LTG (Yu et al., 2021b)
70.06
5.72
RG3D-Mesh (Chan al., 2022)
83.11
5.95
Text2Tex (Chen et al., 2023)*
46.91
4.35
Texturify (Siddiqui et al., 2022)
9.75
2,46
59.55
4.97
EpiGRAF (Skorokhodov al., 2022)
4.26
2.34
89.64
6.73
TUVE
15.87
1.95
41.79
2.95
DiffusionCats. The above real-world dataset assumes known camera pose distributions, such as hemispheres. However, aligning in-the-wild objects into these specific poses can be time-consuming and prone to inaccuracies. Therefore, we introduce a data generation pipeline that directly synthesizes realistic texture images. We render depth maps from 3D shapes and convert these depth maps into images using pre-trained diffusion models (Rombach et al., 2022; Zhang et al., 2023a). Next, we determine the bounding box based on the depth map and feed this into off-the-shel segmentation model (Kirillov et al., 2023) to isolate the target object in the foreground. This pipeline eliminates the need for TUVF to depend on real-world image datasets, making it adaptable to other categories with controllable prompts. For quantitative evaluation, we use 250 shapes of cats from SMAL (Zuffi al., 2017), which includes appearance variations and deformations, to create new 2D-3D dataset for texture synthesis through our data generation pipeline. We discuss the details of this pipeline in Appendix P and samples of the generated dataset Appendix Q
4.2 BASELINES
Mesh-based Approaches. Texturfiy (Siddiqui et al., 2022) is a state-of-the-art prior work texture synthesis. They proposed using a 4-rosy field as better representation for meshes. TexFields (Oechsle et al., 2019), SPSG (Dai et al., 2021). ). LTG (Yu et al., 2021b), and EG3D-Mesh (Chan et al., 2022) are all bassed baselines. These baselines follow a similar framework, where the texture generators are conditioned on certain shape geometry condition. The biggest difference among these methods is that they use different representations. Specifically, the TexFields (Oechsle et al., 2019) uses global implicit function to predict texture for mesh, and SPSG (Dai et al., 2021) uses 3D convolution networks to predict voxels for textures. EG3D-Mesh (Chan et al., 2022) uses the triplane representation in EG3D (Chan al., 2022) to predict the face colors for given mesh. Note that all baselines require explicit geometry encoding for texture synthesis. On the other hand, our method relies on correspondence and does not directly condition texture on given geometry. Furthermore, our learned dense surface correspondence allows for direct texture transfer. We also compare with a concurrent work, Text2Tex (Chen al., 2023), which proposes an instance-specific approach for texture synthesis using pre-trained diffusion model.
NeRF-based Approach. We also evaluate our method against state-of-the-art NeRF-based approach, EpiGRAF (Skorokhodov et al., 2022), which employs a tri tri-plane representation and patch-based discriminator. To modify EpiGRAF into texture generator for radiance fields, we follow TexFields (Oechsle et al., 2019) and use point cloud encoder to encode geometry information into EpiGRAF's style- triplane generator, For fair comparison, we employ the same discriminator and training hyper-parametert for EpiGRAF (Skorokhodov et al., 2022) and our method.
4.3 EVALUATION METRICS
LPIPS, and LPIPS,, We introduce two metrics to evaluate our model's ability to disentangle geometry and texture. The first metric, LPIPS is calculated by generating ten random latent codes for each shape in the test set and measuring the diversity of the synthesized samples. the model struggles to disentangle, the generated samples may appear similar, leading to lower LPIPS score. For the second metric, L.PISS, we measure the semantic consistency after texture swapping. Specifically, we randomly sample four latent codes and transfer them among 100 test shapes. If model successfully disentangled the geometry and texture, all samples with the same texture code should look semantically similar, leading to a lower LPIPS score.
FID and KID, In addition to LPIPS, and LPIPS,, we employ two standard GAN image quality and diversity metrics, specifically the Frechet Inception Distance (FID) and Kernel Inception Distance
7

---

Published as a conference paper at ICLR 2024
Table Quantitative Results on Photoshape and DiffusionCats. While our method has slightly larger FID and KID than Texturify on Photoshape, we achieve significantly better results in control- lable synthesis. On the other hand, our method achieves better results in both visual quality and controllable synthesis on DiffusionCats. KID is multiplied by 10².
Photoshape
DiffusionCaas
LPIPS,
LPIPS:
FID 4
KID
LPIPS, 1
LPIPS,
EpGGRAF (Skorokhodov et al., 2022)
7,00
3.14
65.62
4,20
5.37
1.98
196.01
19.10
Texturify (Sidkiqui et al., 2022)
6.74
2.89
45.92
2.61
4.77
2.61
72.43
5.61
TUVE
14.93
2.55
51.29
11.99
1.50
64.13
3.56
Texturff
EpiGRAF
TUVF
Figure 4: Texture Transfer Comparison. Each approach applies the same texture code to synthesize
textures on different shapes. TUVF can obtain consistent textures across all shapes, while previous approaches output different styles on different object shapes when using the same texture code.
(KID) scores. We follow Texturiy'ss setup in all experiments, training on 512x512 resolution images and rendering images at a resolution of 512x512 and subsequently downsampling to 256x256 for evaluation. We employ four random views and four random texture codes for all evaluations and incorporate all available images in the FID/KID calculations.
4.4 RESULTS
Canonical Surface Auto-encoder. To the best of our knowledge, our work represents the first attempt to explore joint end-to-end canonical point auto-encoder (Cheng et al., 2021) and surface learning (Peng et al., 2021). A key concern is the smoothness of the learned cor- respondence and the reconstructed surface. We construct and visualize the mesh using the pre- dicted indicator function X', and without requir- ing any proxy function (such nearest neighbor search), dense surface correspondence is readily obtained. As result of the Poisson surface reconstruction, Pj, which holds the correspondence, naturally lies on the surface. In Figure 5 and Figure 7, we showcase that our reconstructed surface is indeed smooth and that the correspondence is both dense and smooth well.
Figure 5: Surface reconstruction with dense cor- respondence. The color map indicates the cor- respondence between each instance and the UV. Please refer to Appendix A for further results.
Quantitative Texture Synthesis Results. We show the quantitative results on CompCars in Table I, the results Photoshape and DiffusionCats in Table 2. For the CompCars and DiffusionCats datasets, we achieve significant improvements over all the metrics. For the Photoshape dataset, while our approach is slightly worse than Texturify in FID and KID, as for the fidelity metrics, we obtain much better results on controllable synthesis. We further conduct a user study to evaluate the texture quality. Two metrics are considered: (I) General: The users compare random renders from baselines and our method, choosing the most realistic and high-fidelity method. (2) Transfer: The users compare three random renders with the same texture code, selecting the most consistent across shapes. We use Amazon Turk to collect 125 user feedback; the results are shown in Table 3.
Qualitative Texture Synthesis Results. We show our qualitative results for texture synthesis in Figure 3, which confirms that textures generated by our approach are more visually appealing, realistic, and diverse. EpiGRAF suffers from the redundancy of tri-plane representation, leading to less sharp results. We also observe that the tri- tri-plane representation fails when objects are thin (e.g., cats). Our proposed method also shows better diversity and disentanglement than Texturify
8

---

Published as a conference paper at ICLR 2024
Table 3: User study. Percentage of users who favored our method over the baselines in a user study with 125 responses.
Table 4: Ablation with different architecture designs for texture mapping function Evaluated on CompCars. KID is multiplied by 102.
Dataset
Metric
Texturify 1
EpiGRAF
Architocture
FID 4
KID
CompCars
General
82.40
85.60
CTPS-2D (Anokhin et al., 2021)
148.09
13.38
Transfer
75.20
StyleGAN2 (Karrns et al,, 2020b)
103.62
7.89
General
74.40
80.00
Photoshape
Transfer
70.40
75.20
CIPS-UV [ours)
41.79
2.95
and EpiGRAF. We show texture transfer results in Figure 4, where Texturify and EpiGRAF failed to transfer the texture on some samples. Please refer to Appendix B for more texture synthesis results.
Ablation study. We conducted an ablation study on different texture generator architectures using the CompCars dataset. Two architectures were considered: CIPS-2D 2D and StyleGAN2, the former being the same as the one proposed in (Anokhin er al., 2021), and the latter being a popular choice for both 21) and 3D GANs. Since the input to both generators is in 3D (i.e., sphere coordinates), an equirectangular projection was first performed to transform the coordinates into 2D. We show the results in Table where CIPS-2D suffers from the explicit parameterization of unwrapping 3D 2D. Similarly, StyleGAN2 suffers from pixel-wise interaction operators that degrade its performance as well. In contrast, our proposed generator design avoids explicit parameterization and operators that bring interactions between pixels. By preserving 3D semantic information, our generator produces realistic and diverse textures on the UV sphere. Please refer to Appendix for more ablation studies.
Texture Editing. Our method en- ables texture editing (Yang et al., 2022; Bao et al., 2023) by allowing di- rect modification of rendered images, such as drawing or painting. Given a synthesized texture, one directly operate on the rendered view to edit the texture. By fine-tuning the edited image through back-pre pack-propagation to the texture feature, we can obtain an edited texture that is 3D consistent across different views. As shown in Figure 6, after editing an image, we can fine-tune its texture feature and transfer it to different shapes.
Original
Editing
Results
Transfer
Figure 6: Editing and Transfer Results. The disentangle- ment ensures that the radiance field is independent of density conditions, enabling to fine-tune UV texture features with sparse views. Please refer to Appendix C for more samples.
CONCLUSION, LIMITATIONS, AND FUTURE WORK
In this paper, introduce Texture UV Radiance Fields for generating versatile, high-quality textures applicable to a given object shape. The key idea is to generate textures in a learnable UV sphere space independent of shape geometry and compact and efficient as a surface representation. Specifically, we leverage the UV sphere space with a continuous radiance field so that an adversarial loss on top of rendered images can supervise the entire pipeline. We achieve high-quality and realistic texture synthesis and substantial improvements over state-of-the-an approaches to texture swapping and editing applications. We are able to generate consistent textures over different object shapes while previous approaches fail. Furthermore, we can generate more diverse textures with the same object shape compared to previous state-of-the-arts
Despite its merits, our method has inherent limitations. Our current correspondence assumes one-to- one dense mapping. However, this assumption does not always hold in real-world scenarios. See Appendix T for more discussion regarding the limitations. To further achieve more photorealistic textures, one option is to incorporate advanced data-driven priors, such as diffusion models, which can help mitigate the distortions and improve the quality of the generated textures. Utilizing more sophisticated neural rendering architectures, such as ray transformers, can also enhance the results.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
This work was supported, in part, by NSF CAREER Award IIS- 2220014, the Amazon Research Award, and the CISCO Faculty Award.
ETHICS STATEMENT
Our work follows the General Ethical Principles listed in ICLR Code of Ethics (https:/ /iclr. cc/publ i c/ CodeofEthi cs).
We share similar concerns as other generative models (Poole et al., 2023). Our work may raise ethical concerns related to the improper use of content manipulation or the generation of deceptive information.
Our data generation pipeline relies on pre-trained diffusion (Rombach et al., 2022) model, which can inherit biases in its training data. The dataset (Schuhmann et al., 2021) used to train StableDiffusion may include undesirable images. Additionally, the pre-trained diffusion model is conditioned on features from another pre-trained large language model, which could introduce unintended biases. As result, generated texture may also include unintended biases.
REFERENCES
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov. Image generators with conditionalll-indeeeddde pixel synthesis. IEEE Conference Computer Vision and Pattern Recognition, pp. 14278-14287, 2021. 4, 36
Dejan Azinović, Ricardo rgb-d surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 6290-6301, 2022.
Dan B Goldman, Matthias Nießner, and Justus Thies. Neural
Chong Bao, Yinda Zhang, Bangbang Yang, Tianxing Fan, Zesong Yang. Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Sine: Semantic-driven image-based nerf editing with prior prior-guided editing field. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 20919-20922 2023. 9
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Bertin-Buuulla,, and Pratul P Srinivasan. Mip-nerf: multiscale representation for anti-aliasing neural radiance fields. In IEEE International Conference on Computer Vision, pp. 5855-5864, 2021. 2
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In IEEE International Conference on Computer Vision, 2023. 2
Pravin Bhat, Stephen Ingram, and Greg Turk. Geometric texture synthesis by example. In SGP, pp. 41-44, 2004.3
Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael Black. Dynamic faust: Registering human bodies in motion. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 6233-6242, 2017. 17
Gunilla Borgefors. Hierarchical chamfer matching: A parametric edge matching algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(6):899-855, 1988. 4
Rohan Chabra, Jan E Lenssen, Eddy 1lg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction.
In European Conference on Computer Vision, pp. 608-625. Springer, 2020. 2
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 17799-5809 2021.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-ave 3d generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 16123-16133, 2022. 2, 3, 7,37
10

---

Published as a conference paper at ICLR 2024
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-ne 3d model repository. arXiv preprint, 2015. 6, 17.
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In IEEE International Conference on Computer Vision, pp. 14124-14133, 2021.
Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint, 2023. 3, 40
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Neural Information Processing Systems, 29, 2016.
Zhiqin Chen, Kangxue Yin, and Sanja Fidler. Auv-net: Learning aligned uv maps for texture transfer and synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, 2022. 3, 40
An-Chieh Cheng, Xueting Li, Min Sun, Ming-Hsuan Yang, and Sifei Liu. Learning 3d dense correspondence via canonical point autoencoder. Neural Information Processing Systems, 34: 6608-6620, 2021. 4, 8, 32, 35, 40
An-Chich Cheng. Xueting Li, Sifei Liu, Min Sun, and Ming- Hsuan Yang. Autoregressive 3d shape generation via canonical mapping. In European Conference on Computer Vision, pp. 89-104. Springer, 2022, 4, 32, 35
Yen-Chi Cheng, Hsin-Ying
Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yor Yan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In IEEE Conference on Computer Vision and Pattern Recognition, pp.
2023. 2
George Cross and Anil K Jain. Markov random field texture models. IEEE Transactions on Pattern
Analysis and Machine Intelligence, pp. 25-39, 1983. 3
Angela Dai, Yawar Siddiqui, Justus Thies, Julien Valentin, and Matthias Nießner. Spsg: Self- supervised photometric scene generation from rgb-d scans. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1747-1756, 2021. 3,
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: universe of annotated 3d objects. In IEEE Conference Computer Vision and Pattern Recognition, pp. 13142-13153, 2023. 40
Robert A Drebin, Loren Carpenter, and Pat Hanrahan. Volume rendering. ACM SIGGRAPH, 22(4): 65-74, 1988.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint, 2016.
Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In IEEE International Conference on Computer Vision, volume 2, pp. 1033-1038. IEEE, 1999.
Simone Foti, Bongjin Koo, Danail Stoyanov, and Matthew J Clarkson. 3d shape variational au- toencoder latent disentanglement via mini-batch feature swapping for bodies and faces. In IEEE Conference Computer Vision and Pattern Recognition, pp. 18739, 2022. 3
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiging Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. In Neural Information Processing Systems, 2022. 3
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Neural Information Processing Systems, volume 27, 2014. 2,

---

Published as a conference paper at ICLR 2024
Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. Efficientnerf efficient neural radiance fields. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 12902-12999 12911, 2022. 5
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In European Conference on Computer Vision, pp. 172-189, 2018. 3
Ananya Harsh Jha, Saket Anand, Maneesh Singh, and VSR Veeravasarapu. Disentangling factors of variation with cycle-consistent variational auto-encoders. In European Conference on Computer Vision, pp. 005-820, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019. 3, 4, 6, 36
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Neural Information Processing Systems, 33: 1104,, 2020a. 37
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 81 10-81119 19, 2020b. 9, 36
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint, 2013.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In JEEE International Conference on Computer Vision, 2023. 7,38
Johannes Kopf, Chi-Win Wing Fu, Daniel Cohen-Or, Oliver Deussen, Dani Lischinski, and Tien-Tsin Wong. Solid texture synthesis from 2d exemplars. In ACM SIGGRAPHI, 2007.
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. Neural Information Processing Systems, 28, 2015.
Jiaxin Li, Zijian Feng, Qi She, Henghui Ding, Changhu Wang, and Gim Hee Lec. Mine: Towards continuous depth mpi with nerf for novel view synthesis. In IEEE International Conference Computer Vision, pp. 12578-12555 12588, 2021a.
Ruihui Li, Xianzhi Li, Ka-Hei Hui, and Chi- Chi-Wing Fu. Sp-gan: and manipulation. ACM Transactions Graphics, 40(4):1-12, 2021b. 36
3d shape generation
Andrew Liu, Shiry Ginosar, Tinghui Zhou, Alexei A Efros, and Noah Snavely. Learning to factorize and relight a city. In European Conference on Computer Vision, pp. 544-561. Springer, 2020. 3
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A diffcrentiable renderer for imsee-based 3d reasoning. IEEE International Conference on Computer Vision, Oct 2019. 3
Emile Mathieu, Tom Rainforth, Nana Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. In ICML, pp. 4402-4412. PMLR, 2019.
Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, 1(2):99-108, 1995.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference Computer Vision, 2020. 3,
Jiteng Mu, Shalini De Mello, Zhiding Yu, Nuno Vasconcelos, Xiaolong Wang, Jan Kautz, and Sifei Liu. Coordgan: Self-supervised dense correspondences emerge from gans. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1001 100220,, 2022. 3
Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsupervised learning of 3d representations from natural images. In /EEE International Conference Computer Vision, pp. 7588-7597, 2019. 2,
12

---

Published as a conference paper at ICLR 2024
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1504-3515, 2020. 2
Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In IEEE International Conference on Computer Vision. pp. 4531-4540, 2019. 3,7
Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In IEEE International Conference on Computer Vision, pp. 5589-5599, 2021. 5
Jonathan Palacios and Eugene Zhang. Rotational symmetry field design on surfaces. ACM Transac- fions Graphics, 26(3):55-es, 2007.
Keunhong Park, Konstantinos Rematas, Ali Farhadi, and Steven M. Seitz. Photoshape: Photorealistic materials for large-scale shape collections. ACM Transactions on Graphics, 37(6), November
2018a. 2
Keunhong Park, Konstantinos Rematas, Ali Farhadi, and Steven M. Seitz. Photoshape: Photorealistic materials for large-scale shape collections. ACM Transactions on Graphics, 2018b. 6
Taesung Park, Jun- Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. Neural Information Processing Systems, 33:7198 2020 3
Dario Pavllo, Graham Spinks, Thomas Hofmann, Convolutional generation of textured 3d meshes. Neural Information Processing Systems, 33: 870-882, 2020.
Moens, and Aurelien Lucchi.
Songyou Peng, Chiyu Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, and Andreas Geiger. Shape as points: A differentiable poisson solver. Neural Information Processing Systems, 34: 13032-13044, 2021. 5,
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 14104-14113, 2020. 3
Ben Poole. Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d
diffusion. In International Conference on Learning Representations, 2023. 10
Amit Raj, Cusuh Ham, Connelly Barnes, Vladimir Kim, Jingwan Lu, and James Hays. Learn- ing to generate textures on 3d meshes. In IEEE Conference on Computer Vision and Pattern Recognition pp. 32-38, 2019. 3
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text- fext-guided texturing 3d shapes. ACM SIGGRAPH, pp. 1-11, 2023. 40
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022. 7, 10, 38
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85-117, 2015. 4
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint, 2021. 10
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Neural Information Processing Systems, 33:20154-20166, 2020.
Yawar Siddiqui. Justus Thies, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Texturify: Generating textures on 3d shape surfaces. European Conference on Computer Vision, pp. 72-88. Springer, 2022. 2, 3, 7, 8, 20, 23, 24, 27, 28, 29, 30, 31, 32, 40
13

---

Published as a conference paper at ICLR 2024
Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Finegan: Unsupervised hierarchical disentanglement for fine- pee-grained object generation and discovery. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 6490-6499, 2019. 3
Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka. Epigraf: Rethinking training of 3d gans. Neural Information Processing Systems, 2022. 2, 3, 5, 6, 7, 8, 27, 28, 29, 30, 31, 32, 33, 36, 37
Gabriel Taubin. A signal processing approach to fair surface design. In ACM SIGGRAPH, PP. 351-358, 1995.
Alex Trevithick and Bo Yang. Grf: Learning a general radiance field for 3d representation and rendering. In IEEE International Conference on Computer Vision, pp. 15182-15192, 2021. 2
Greg Turk. Texture synthesis on surfaces. ACM SIGGRAPH, pp. 347-354, 2001.
Naveen Venkat, Mayank Agarwal, Maneesh Singh, and Shubham Tulsiani. Geometry- biisset trans- formers for novel view synthesis. arXiv preprint, 2023. 2
Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang, et al. Is attention all nerf needs? arXiv preprint, 2022. 2
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Neural Information Processing Systems, 2021. 2,5
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics, 1-12. 2019. 4, 36
Fanbo Xiang, Zexiang Xu, Milos Hasan, Yannick Hold-Geoffroy, Kalyan Sunkavalli, and Hao Su. Neutex: Neural texture mapping for volumetric neural rendering. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1119-7128, 2021.
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-bascd neural radiance fields. In IEEE Conference Computer Vision and Pattern Recognition, pp. 5438-5444 2022. 2, 3,5
Banghang Yang, Chong Bao, Junyi Zeng. Hujun Bao, Yinda Zhang, Zhaopeng Cui, and Guofeng Zhang. Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. In European Conference Computer Vision, pp. 597-614. Springer, 2022. 3,9
Linjie Yang. Ping Luo, Chen Change Loy, and Xiaoou Tang. large-scale car dataset for fine-graine ane-grained categorization and verification. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3973973-33 3973-3981, 2015. 2,
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Neural Information Processing Systems, 33:2492-2502, 2020. 2
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces.
Neural Information Processing Systems, 34:4805 2021. 2,5
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In IEEE Conference on Computer Vision and Pattern Recognition, pp-
2021a. 2
Rui Yu, Yue Dong, Pieter Peers, and Xin Tong. Learning texture generators for 3d shape collections from internet photo sets. In BMVC, 2021b. 3.
Xin Yu, Peng Dai, Wenbo Li, Lan Ma. Zhengzhe Liu, and Xiaojuan Qi. Texture generation 3d meshes with point-uv diffusion. In IEEE International Conference on Computer Vision, 2023. 3, 40
14

---

Published as a conference paper at ICLR 2024
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In Neural Information Processing Systems, 2022.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In ICML, pp. 7354-7363. PMLR, 2019.
Jason Zhang, Gengshan Yang. Shubham Tulsiani, and Deva Ramanan. Ners: neural reflectance surfaces for sparse-view 3d reconstruction in the wild. Neural Information Processing Systems, 34:
29835-29847, 2021. 2
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. IEEE International Conference on Computer Vision, 2023a. 7, 38
Yanshu Zhang, Shichong Peng, Alireza Moazeni, and Ke Li. Papr: Proximity attention point rendering. In Neural Information Processing Systems, 2023b. 3
Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael Black. 3d menagerie: Modeling the 3d shape and pose of animals. In JEEE Conference on Computer Vision and Pattern Recognition, pp. 6365-6373, 2017. 7, 17
15

---

Appendix Table of Contents
A More Qualitative Results of Canonical Surface Auto-encoder.
17
More Qualitative Results using our Data Generation Pipeline
18
More Qualitative Results of Texture Editing
19
D More Qualitative Results of Resolutono Synthesis
20
More Qualitative Results of Texture Transfer
27
More Quantitative Results on full Photoshape
32
More Comparisons with EpiGRAF using Ground truth Geometry
33
H Ablation Study Canonical Surface Auto-encoder.
34
Ablation Study on Different UV Resolution
34
J Implementation Details of TUVF Rendering
35
K Implementation Details of Canonical Surface Auto-encoder
35
L Implementation Details of Texture Generator ho CIPS-UV
36
M Implementation Details of Patch-based Discriminator
36
Training Details and Hyper-parameters.
36
Computational Time and Model Size
37
Implementation Details of Data Generation Pipeline...
38
Samples of Our Generated Dataset
38
R Motivation for using the Neural Radiance Field...
39
Disscusion on More Related Work
40
Limitations
40
16

---

Published as a conference paper at ICLR 2024
A MORE QUALITATIVE RESULTS OF CANONICAL SURFACE AUTO-ENCODER
Can our Canonical Surface Auto-encoder handle different topologies? We align with prior works by evaluating our approach on two categories within ShapeNet (e-g.- cars, chairs) for com- parisons. While using UV sphere adds constraints representing shape varieties, can still model reasonably diverse topologies. We additionally include the canonical surface auto- encoding results on three different shape collections in Figure 7. Specifically, we include results ShapeNet (Chang et al., 2015) Airplanes (top), DFAUST (Bogo et al., 2017) (middle, scanned human subjects and motions), and SMAL Zuffi et al. (2017) (bottom, articulated animals e.g., lions, tigers, and horses). Our method consistently produces high-quality shape reconstructions with dense and smooth learned correspondence, even for non-genus zero shapes such airplanes with holes. This is not achievable for traditional mesh-based UV deformation.
method
ShapeNet (airplanes)
D-FAUST (dynamic human scans)
SMAL (articulated animals)
Figure Surface Reconstruction Results on Different Datasets.
17

---

Published as a conference paper at ICLR 2024
B MORE QUALITATIVE RESULTS USING OUR DATA GENERATION PIPELINE
We provide our texture synthesis results using Canonical Surface Auto-encoder trained on SMAL (Appendix A) and 2D datasets generated by our data pipeline(Appendix P). As shown in Figure we demonstrate that our model can generate photorealistic textures for various animal categories. With the pipeline, can also control the style with different prompts. Note that for simplicity, some results in the figure are trained on single instance or view, indicated by grey text.
(multi-view/single instance)
A DLSR image of a horse, pony, Gaited high quality, high res
viin/le-viewmuulii instances)
Texture Transfer (horse - cow)
A DLSR image of a dairy cow, Holstein, high quality, high res
A DLSR image of cat, tabby cat, high quality, high res
<singlevveew/snglle instance)
A DLSR image of a dog, shiba, Shiba Inu, high quality, high res
(single-view/siiglle instance)
Figure 8: Texture Synthesis Results on SMAL. We include results using horses, cows, cats, and dogs. We also transfer a texture from horse to a cow using correspondence learned from Appendix A.
18

---

Published as a conference paper at ICLR 2024
C
MORE QUALITATIVE RESULTS OF TEXTURE EDITING
Direct Editing
Target Shape
Transfer to other Shapes
Figure 9: Texture Editing and Transfer. Our approach offers exceptional flexibility when comes to texture editing. We support range of texture editing techniques, including texture swapping, filling, and painting operations. Given a synthesized texture, one can directly operate on the rendered view to edit the texture (as illustrated by the green box By fine-tuning the edited image using the back- prekkppppagation to the texture feature, we can obtain an edited texture that is 3D consistent across different views (as shown in the blue box). Moreover, this edited texture feature can also be transferred among different shapes (as demonstrated by the red box).
19

---

Published as a conference paper at ICLR 2024
D MORE QUALITATIVE RESULTS OF HIGH-RESOLUTION SYNTHESIS
Figure 10:
Our results on Compcars dataset. The model is trained with 512x512 resolution, images shown are rendered with 1024x 1024 resolution. Compared to Texturify Siddiqui al. (2022) (results shown in Figure 13 and Figure 14), our texture synthesis approach produces textures with superior detail. Notably, our generator is capable of synthesizing intricate features such logos, door handles, car wipers, and wheel frames. Zoom in for the best viewing.
20

---

Published as a conference paper at ICLR 2024
Our results Compcars dataset. The model is trained with 512x512 resolution, images shown are rendered with 1024x 1024 resolution. Note that all the images are rendered from the same instance, including images in Figure 10 and Figure 12. This highlights the effectiveness of our proposed method in synthesizing photo-realistic textures while maintaining 3D consistency. Zoom in for the best viewing.
Figure 11:
21

---

Published as a conference paper at ICLR 2024
Our results Compcars dataset. The model is trained with 512x512 resolution, images shown are rendered with 1024x 1024 resolution. In addition to generating different global colors, our proposed method can generate diverse textures by including intricate local details. For example, the generated textures may include unique logos (different from those shown in Figure 11) or distinct tail light styles (different from Figure 10). Zoom in for the best viewing.
Figure 12:
22

---

Published as a conference paper at ICLR 2024
Figure 13: Texturify Siddiqui et al. (2022) results Compears dataset. The model is trained with 512x512 resolution, images shown are rendered with 1024x 1024 resolution. The sample shown in this figure was generated using the pre-trained model provided by the authors. Notably, all the images in this figure depict different render angles of the same instance.
23

---

Published as a conference paper at ICLR 2024
Figure 14: Texturify Siddiqui et al. (2022) results Compears dataset. The model is trained with 512x512 resolution, images shown are rendered with 1024x 1024 resolution. The sample shown in this figure was generated using the pre-trained model provided by the authors. Notably, all the images in this figure depict different render angles of the same instance.
24

---

Published as a conference paper at ICLR 2024
Our results on Photoshape dataset. The model is trained with 512x512 resolution, images shown are rendered with 1024x 1024 resolution. Our model is highly effective in synthesizing top-quality textures for chairs. Interestingly, the generated textures may even feature a variety of material styles, such as black leather, suede fabric, or flannel (see Figure 16), adding an extra level of realism to the textures. Zoom in for the best viewing.
Figure 15:
25

---

Published as a conference paper at ICLR 2024
Our results on Photoshape dataset. The model is trained with 512x512 resolution, images shown are rendered with 1024x 1024 resolution. Thanks to the correspondence learned from our Canonical Surface Auto-encoder, textures can be generated without interference from geometry information. Furthermore, the model can predict accurate textures for different parts of the object. For instance, the legs of the chair may have distinct textures from the seats, and the boundary between these two parts is clearly defined. This demonstrates the importance of the correspondence learned from the Canonical Surface Auto-encoder. Zoom in for the best viewing.
Figure 16:
26

---

Published as a conference paper at ICLR 2024
E
MORE QUALITATIVE RESULTS OF TEXTURE TRANSFER
Texturify
EpiGRAF
Ours
Texturify
EpiGRAF
Ours
Texturify
EpiGRAF
Ours
Texturify
EpiGRAF
Ours
Texture Transfer Results on CompCars dataset. Each approach (in the same row) ap- plies the same texture code to synthesize textures on different input shapes. Our method can generate textures that exhibit consistency across all shapes, unlike other approaches (e.g.. Texturify Siddiqui et al. (2022) and EpiGRAF Skorokhodov al. (2022)), which may produce different styles or local details on different object shapes even when using the same texture code.
Figure 17:
27

---

Published as a conference paper at ICLR 2024
Texturity
EpiGRAF
Ours
Texturity
EpiGRAF
Ours
Texturify
EpiGRAF
Ours
Texturify
EpiGRAF
Ours
Figure 18: Texture Transfer Results on CompCars dataset. Each approach (in the same row) ap- plies the same texture code to synthesize textures on different input shapes. Our method can generate textures that exhibit consistency across all shapes, unlike other approaches (e.g.. Texturify Siddiqui al. (2022) and EpiGRAF Skorokhodov al. (2022)), which may produce different styles local details on different object shapes even when using the same texture code. Consider the results shown in row 4 of the figure. While the samples generated by the Texturify Siddiqui et al. (2022) method exhibit consistency in global color (i.c., all the cars are red), the same texture code may result in different window styles (i.e., number of windows).
28

---

Published as a conference paper at ICLR 2024
Texturity
EpiGRAF
Ours
Texturily
EpiGRAF
Ours
Texturify
EpiGRAF
Ours
Texturify
EpiGRAF
Ours
Figure 19: Texture Transfer Results on CompCars dataset. Each approach (in the same row) ap- plies the same texture code to synthesize textures on different input shapes. Our method can generate textures that exhibit consistency across all shapes, unlike other approaches (e.g.. Texturify Siddiqui et al. (2022) and EpiGRAF Skorokhodov al. (2022)), which may produce different styles local details on different object shapes even when using the same texture code.
29

---

Published as a conference paper at ICLR 2024
Texturity
EpiGRAF
Ours
Texturity
EpiGRAF
Ours
Texturify
EpiGRAF
Ours
Texturify
EpiGRAF
Ours
Figure 20: Texture Transfer Results on Photoshape dataset. Each approach (in the same row) ap- plies the same texture code to synthesize textures on different input shapes. Our method can generate textures that exhibit consistency across all shapes, unlike other approaches (e.g.. Texturify Siddiqui et al. (2022) and EpiGRAF Skorokhodov et al. (2022)), which may produce different styles or local details on different object shapes even when using the same texture code.
30

---

Published as a conference paper at ICLR 2024
Texturity
EpiGRAF
Ours
Texturity
EpiGRAF
Ours
Texturify
EpiGRAF
Ours
Texturify
EpiGRAF
Ours
Figure 21: Texture Transfer Results on Photoshape dataset. Each approach (in the same row) ap- plies the same texture code to synthesize textures on different input shapes. Our method can generate textures that exhibit consistency across all shapes, unlike other approaches (e.g.. Texturify Siddiqui et al. (2022) and EpiGRAF Skorokhodov et al. (2022)), which may produce different styles or local details on different object shapes even when using the same texture code.
31

---

Published as a conference paper at ICLR 2024
F MORE QUANTITATIVE RESULTS ON FULL PHOTOSHAPE
We report results on the fill Phioshape dataset in Table 5, showcasing superior controllable synthesis but higher FID and KID values compared to Texturify. Similar to prior works (Cheng et al., 2021; 2022), our method builds upon learnable UV maps, assuming one-to-one dense correspondence between instances of a category. However, in real-world scenarios, this assumption does not always hold: There may be variations in shape (e.g., armchairs and straight chairs) structure (e.g.. lounges and bean chairs) across different instances. This introduces challenges in modeling high-fidelity shapes and detailed correspondences the same time, Despite these challenges, our method produces reasonable results, as depicted in Figure 22.
Table 5: Quanitative Results on Full Photoshape. While our method has slightly larger FID and KID than Texturify on full Photoshape, we achieve significantly better results in controllable synthesis. KID is multiplied by 102.
Method
FID
KID 4
LPIPS, t
LPIPS: +
EpiGRAF (Skorokhodov et al., 2022)
89.74
6.28
3.63
6.51
Texturify (Siddiqui et al., 2022)
26.17
1.54
8.86
3.46
TUVF (ours)
57.56
3.74
16.43
2.72
Figure 22: Qualatative Results on Full Photoshape. Although finding correspondence for shapes with large structural differences is challenging, our method produces reasonable results.
32

---

Published as a conference paper at ICLR 2024
(a)
(b)
(c)
(d)
Figure 23: Visualization of the ablation study over the Canonical Surface Auto-encoder. The four sub-figures correspond to the four settings introduced in Section H. Zoom in for the best viewing.
G MORE COMPARISONS WITH EPIGRAF USING GROUND TRUTH GEOMETRY
For fair comparison, rokhodov et al., 2022) and our method. This ensures that both approaches utilize the same geometry. Below, we provide the results comparing method to EpiGRAF while employing the ground-truth SDF for EpiGRAF.
the auto-encoded geometry the input for both EpiGRAF (Sko-
Table 6: Comparisons with EpiGRAF using Groundtruth Geometry. KID is multiplied by 102.
Dataset
Method
FID
KID
LPIPS, t
LPIPS,
EpiGRAF (Skorokhodov al., 2022)
88.37
6.46
4.23
2.31
CompCars
TUVF (ours)
41.79
2.95
15.87
1.95
EpiGRAF (Skorokhodov al., 2022)
55.31
3.23
7.34
2.61
Photoshape
TUVF (ours)
51.29
2.98
14.93
2.55
33

---

Published as a conference paper at ICLR 2024
Table 7: Ablation Study over Canonical Surface Auto-encoder. Evaluated on CompCars.
Method
Mapping Direction
GT Geometry
Proxyless Surface
Smootheness
FID
KID L
(a)
(base)
Surface
41.79
2.95
(b)
UV Surface
61.81
4.08
(C)
UV > Surface
79.43
6.16
(d)
Surface UV
139.19
12.92
H ABLATION STUDY ON CANONICAL SURFACE AUTO-ENCODER
One drawback of our framework is that the auto-encoded indicator grid may not be perfect. As a result, we investigated several different network designs for the stage-1 geometry pre-training, which enabled us learn texture synthesis using the ground-truth indicator function. We considered comparing four settings in this study:
(a) Our Canonical Surface Auto-encoder: The geometry network takes UV points as inputs and maps them to the surface. An additional function go is learned to predict the surface normal for each point, and an auto-encoded indicator function is obtained. Texture synthesis is performed using the auto-encoded indicator function.
(b) The geometry network takes UV points as inputs and maps them to the surface. No go is used. Texture synthesis is performed using the ground-truth indicator function.
(c) The geometry network takes UV points as inputs and maps them to the surface. No ga is used. Texture synthesis uses the ground-truth indicator function, while points are warped to the ground truth surface via the nearest neighbor.
(d) The geometry network takes surface points as inputs and maps them to the UV. In this case, there is no need for gø. and texture synthesis is learned using the ground-truth indicator function.
Two important factors may affect the quality of synthesis. First, the surface points should lie as close to the exact surface of the indicator function possible. This is because our MLPp takes the nearest neighbor feature and the distance between the query point and the nearest neighbor as inputs. there gap between the points and the surface of the indicator function, it can confuse MLP, and harm the performance. Secondly, the surface points should as smooth as possible, i.e., evenly distributed among the surface. This ensures that each surface point contributes The results of our ablation study on the car category can be found in Table We also show samples from each setting in Figure 23. The results obtained for settings without smooth correspondence (e-g., setting 3 and 4) show that the textures are more blurry and tend to have distortions. On the other hand, our method produces sharper details compared setting 2, which is trained on proxy surface points. This study demonstrates the unique advantage of our Canonical Surface Auto-encoder design, in which we can learn UV-to-surface mapping with smooth correspondence. Therefore, learning an additional function to predict point normal and obtain an auto-encoded indicator function is necessary to obtain high- fideliyy textures.
similar amount of surface area.
I ABLATION STUDY ON DIFFERENT UV RESOLUTION
Table 8: Ablation Study over different UV resolution. Evaluated on the CompCars dataset.
UV Resolution
FID 4
KID 1
2K (base)
41.79
2.95
1K
43.65
3.01
We investigated the effect of UV resolution the quality of our method. To achieve this, compared our base method with different numbers UV resolution (1K and 2K). The results in Table 8 showed that increasing the UV resolution leads improved performance in terms of
34

---

Published as a conference paper at ICLR 2024
producing higher-quality fine-scale details. However, we found that using level 4 ico-sphere vertices (i.e., 2K points) is sufficient to achieve high-quality results. Further increasing the resolution would result in prohibitively long training times due to the K nearest neighbor search. For example, using level 5 ico-sphere vertices would result in 10242 points, which would significantly slow down the training speed.
IMPLEMENTATION DETAILS OF TUVF RENDERING
To obtain samples from the UV sphere, with 2562 coordinates. After passing these coordinates through the mapping functions fa, ga. and he, obtain 2562 surface points (Xp'). 2562 surface normal (N₂)). ). 128 indication function grid (x). and 2562 32-d 22-dimensional texture feature vectors. To compute the final color of ray, we first sample 256 shading points and identify the valid points using the indication function grid X'. Next, we sample three valid shading points around the surface, and for each valid shading location 24, conduct a K-nearest neighbor search on the surface points Xp- We perform spatial interpolation of the texture feature on the K-nearest surface points obtain the texture feature Ca, for the current shading points z,. our experiment, we set 4, which is not computationally expensive since we only deal with 2562 surface points.
the vertices of level 4 ico-sphere, which provides us
K IMPLEMENTATION DETAILS OF CANONICAL SURFACE AUTO-ENCODER
muax-pool
EdgeCCnn
EdgeC onn
EdgeConv
Linear
Linear
Figure 24: Architecture of our Shape Encoder E.
Shape Encoder 3. Given 3D object O, we first normalize the object to unit cube and sample 4096 points on the surface as inputs to the Shape Encoder & Cheng et al. (2021). The encoder structure is adopted from DGCNN Wang et al. (2019), which contains 3 EdgeConv layers using neighborhood size 20. The output of the encoder is global shape latent Zgen Rd where d 256.
Px (3++ +1)
Sym
Feature Embedding
max-pool /
Linear
Style Embadding
Style Embedding
diplicate limes
EdgeCorv (w/ Attention)
Adaptive Instance Normalization
EdgeConv (w/ Attention)
Adaptive Instance Normalization
X
Linear
X
concutentate
Figure 25: Architecture of Surface Points Decoder fo and Surface Normal Decoder ge-
Surface Points Decoder fo and go. Both the surface points decoder fo and surface normal decoder Go share the same decoder architecture, which is adapted from Cheng et al. (2022). We show the detailed architecture of our decoder in Figure 25. The decoder architecture takes set of point coordinates X,n and geometry feature zgeo as input and learns to output a set point coordinates Xous in wiint-wise manner. To process a set of point coordinates Xin and geometry feature Zgeo- the decoder first creates a matrix by duplicating Zgen for each coordinate X,, and concatenating it each coordinate. This matrix includes both the point coordinates and geometry features. The
35

---

Published as a conference paper at ICLR 2024
decoder has two branches: one branch uses an EdgeConv Wang al. (2019) with attention module to extract point-wise spatial features from the point coordinates. The attention module is adopted from Li al. (2021b), which regresses additional weights among the K point neighbors' features as attentions. The other branch employs a nonlinear feature embedding technique to extract style features from the geometry feature. The local styles adaptive instance normalization Dumoulin et al. (2016) to create fused features. The style embedding and fusion process repeated, and finally, the fused feature used predict the final output Xout- It is worth noting that both the surface points decoder fo and surface normal decoder go use the same geometric features as input. However, they differ in their coordinate input. Specifically. fe takes 2562 UV coordinates as input, while go uses the 2562 output coordinates from So input.
then combined with the spatial features using
L IMPLEMENTATION DETAILS OF TEXTURE GENERATOR ha CIPS-UV
tFeut
tFeat
TexFeature
Pos. Enceding
ModFC
ModFC
ModFC
2x ModEC
2x ModFC
(u, v. w)
A
Mapping Network
Architecture our UV texture feature generator hg. denotes the Affine Transfor- mation module Karras et al. (2019), ModFC denotes modulated fully connected layers, and tFeat denotes temporary features.
Figure 26:
Our generator network, which has multi-layer perceptron-type architecture Anokhin et al. (2021), is capable of synthesizing texture features on a UV sphere. To achieve this, we use a random texture latent vector Stex that shared across all UV coordinates, as well as the UV coordinates (u, v, w) as input. The generator then returns the 32-dim texture feature vector value c for that particular UV coordinate. Thus, to compute the entire UV sphere, the generator is evaluated at every pair of coordinates (u, v, w) while keeping the texture latent vector Zter fixed. Specifically, we utilize a mapping network convert the random texture latent vector Stea into a style vector with the same dimension as Stea- This vector injects style into the generation process through weight modulation. We follow the Fourier positional encoding method outlined in Anokhin al. (2021) to encode the input UV coordinates. The resulting coordinate features pass through the modulated fully connected layers (ModFC), which are controlled by the style vector mentioned above. Finally, we obtain 32- 22-dimensional texture feature for the input coordinate.
M IMPLEMENTATION DETAILS OF PATCH-BASED DISCRIMINATOR
Our discriminator is based on EpiGRAF Skorokhodov et al. (2022), which is similar to the one used in StyleGAN2 Karras et al. (2020b), but modulated by the patch location and scale parameters. We follow the patch-wise optimization approach for training, along with using Beta distribution Skorokhodov al. (2022) for sampling the scale. We use an initial beta value of -4 and gradually anneal it to 0.8 after processing Ie? images.
N TRAINING DETAILS AND HYPER- PARAMETRSS
To demonstrate the training pipeline, we use the car category in ShapeNet and the CompCars dataset as examples. All experiments are performed on workstation equipped with an AMD EPYC 7542 32-Core Processor (2.90GHz) and 8 Nvidia RTX 3090 TI GPUs (24GB each). We implement framework using Py Torch 1.10. For further details and training time for each stage, please refer to Algorithm 1.
36

---

Published as a conference paper at ICLR 2024
Algorithm 1 The training phase of our approach consists of two stages: (1) Canonical Surface Auto-encoder (2) Texture Feature Generator using adversarial objectives
(A) CANONICAL SURFACE AUTO-ENCODER
12 hours ShapeNet Car dataset
Sub-sample points from the imput point clouds as a and the canonical UV sphere n
2: Compute ground-truth indicator function grid X;
3: Initialize weights of the encoder &, decoder fo and go:
4: while not converged do
5: foreach iteration do
6:
Zgeo f- {(x);
7:
Φ So([T,, Egeol), where I, E m
8:
â (e( Zgeal): where f; E 2;:
9:
x' + dpsr (ât ft);
10:
Obtain reconstruction loss (cD((t x)) and Lprsa(x. x):
11:
Update weight;
(B) TEXTURE FEATURE GENERATOR
36 hours on CompCars dataset
1:
Sample points from the canonical sphere m
2:
Random sample shapes with point cloud x and images from dataset Treat:
3: Load pre-trained encoder &, fo and go:
4: Initialize weights of the texture feature generator he and patch-based discriminator D;
5:
while not converged do
6: foreach iteration do
7:
Obtain a. n. and x' with encoder &, fo and go: Sample Ztew from multivariate normal distribution;
8:
9:
+ ho( ztex), where E
Take +- R( G, X. xd), where R denotes renderer and are camera angles; Obtain loss LGAN(L Ireai);
11:
12: Update weight;
N.1 DATA AUGMENTATIONS AND BLUR.
Direct applying the discriminator fails synthesize reasonable textures since there exists geometric distribution shift exists bet collection and rendered 2D images. Therefore, following (Chan et al., 2022: Skorokhodov et al., 2022), we apply Adpative Discriminator Augmentation (ADA) (Karras et al., 2020a) to transform both real and fake image crops before they enter the discriminator. Specifically, we use geometric transformations, such as random translation, random scaling, and random anisotropic filtering. However, disable color transforms in ADA as they harm the generation process and result in undesired textures. In addition to ADA, we also blur the image crops, following (Chan et al., 2022; Skorokhodov al., 2022). However, since we employ a stronger initial blur sigma (i.e., 60) and slower decay schedule, where the image stops blurring after the discriminator has seen 10 images.
larger patch sizes,
O COMPUTATIONAL TIME AND MODEL SIZE
Table 9: The parameter size and inference time for different models. Inference time is measured in seconds.
Method
Representation
Feature Parameterization
Model Size |
Inference Time J
Texturify
Mesh NeRF
24K Faces
52M
0.2039
EpiGRAF Ours
128 x 128 Triplanes
31M
0.2537
NeRF
2K Point Clouds
9M
0.3806
We provide a comparison the inference time and model size of different models Table Specifically, measure the inference time and size of each model based on the time and number
37

---

Published as a conference paper at ICLR 2024
of parameters required to generate texture for given shape instance and render image of resolution 1024. All experiments are conducted on a workstation with an Intel(R) Core(TM) i7- 12700K (5.00GHz) processor and single NVIDIA RTX 3090 TI GPU (24GB). Texturify is mesh-based approach and more efficient in terms rendering compared to NeRF-hased methods. However, its feature space is heavily parameterized the faces, which makes it memory inefficient. Similarly, EpiGRAF requires computing high-resolution triplanes, making it memory-intensine contrast, we only parametrize on 2K point clouds throughout all the experiments and can achieve comparable or even better fidelity. Note that we use the same rendering approach for both TUVF and EpiGRAF: therefore, EpiGRAF has lower inference time than TUVF because it does not require KNN computation.
In
P IMPLEMENTATION DETAILS OF DATA GENERATION PIPELINE
We utilized Stable Diffusion models Rombach et al. (2022) generate realistic texture images, which were subsequently used as training data for TUVE. We start by rendering depth maps from synthetic objects using Blender and converting these depth maps into images using depth-conditioned Controinet Zhang et al. (2023a). If the 3D shape contains object descriptions in its metadata (e.g.. ShapeNet (Chang et al., 2015)), we use the description as text prompt guidance. After generating the image, determine the bounding box based on the depth map and feed this into the Segment Anything Model (SAM) Kirillov al. (2023) to mask the target object in the foreground. This results in realistic textures for synthetic renders. Our pipeline eliminates the need for perfectly aligned cameras and mitigates differences between 3D) synthetic objects and 2D image sets.
Figure 27: Examples of 2D Images Generated by our dataset pipeline. Zoom-in is recommended.
Q SAMPLES OF OUR GENERATED DATASET
We show samples of the depth map and its corresponding 2D images that Controlnet generated on four categories (e.g., cats, horses, dogs, airplanes, and cars). Our pipeline can automatically generate realistic and high-quality (1024x 1024) 2D textured images for 3D models. For the DiffusionCat dataset, we use 250 shapes from SMAL, and split them into 200 for training and 50 for testing. We all 250 shapes to generate textured images. Specifically, for each shape, which results in 4000 images. We use 20 denoise steps for Controlnet, and the entire process takes less than 12 hours for single Nvidia GeForce RTX 3090.
generate 2 samples for eight views
38

---

Published as a conference paper at ICLR 2024
R MOTIVATION FOR USING THE NEURAL RADIANCE FIELD
Our use of NeRF representation highly motivated and intuitive rather than just a trendy choice. provides necessary mechanism for learning surface representation from pixels without any paired (mesh, textured mesh) training data. To understand why we opted for NeRF, we discuss three other options using mesh representation and their limitations. We find that without the help of NeRF, there no straightforward way to maintain correspondence on the faces while ensuring good geometry.
Target
Vanilla (w/o reg)
Moderate Regularized
Over Regularized
Figure 28:
Results using Direct Mesh Deformation.
Mas
Reconst Pointclouc
Recurnl-ucted Mesh tw/ sphere faces)
Figure 29:
Results using Canonical Point Auto-encoder + Sphere Faces.
1. Direct Mesh Deformation. One way build dense correspondence using mesh represen- tation is through direct deformation. This approach involves deforming a template mesh with the offset of each vertex while maintaining the mesh's ''s topology. We show the results in Figure 28 using direct mesh deformation targeting this chair. The main limitation of this approach is that it struggles with non con-genus-zero shapes and tends to produce non-smooth results, even with various shape regularizers. We tested the method with multiple shape regularizers, including Ledges Lnormatt and Liaplacion. with different balancing weights. We also included images of the results when regularizer
used (noted Vanilla).
2. Canonical Point Auto-encoder Sphere Faces. Canonical Point Auto-encoder handle non-genus-zero shapes by allowing complex transformations like folding, cutting, squeezing, and stretching. Therefore, using the same faces as the points on the canonical sphere would lead to incorrect meshes. We show the results in Figure 29,
3. Canonical Surface Autoencoder + Marching Cubes. While this method maintains corre- spondence for points and can reconstruct shapes that are not genus-zero, the faces generated by marching cubes lack correspondence. Therefore, these faces cannot be used for our purpose.
These challenges led us to choose the neural radiance field over the mesh representation for TUVE The neural radiance field offers continuous representation that aligns naturally with our design goals. Given that our canonical point auto-encoder already provides continuity by mapping every point the sphere to the shape's 's surface, using continuous representation is logical and intuitive choice. To further ensure the continuity, we introduce an implicit CIPS-UV generator to achieve high-resolution and high-fidelity output.
39

---

Published as a conference paper at ICLR 2024
S DISSCUSION ON MORE RELATED WORK
Texture synthesis is popular field, with several recent works dedicated to it. While they all aim to synthesize high-quality textures for given shapes, they each have different setups. Some prioritize per-object optimization, some lack correspondence capabilities and others demand 3D supervision. TUVF stands out by focusing on learning generalizable, disentangled, and realistic textures for untextured collections of 3D objects in unsupervised manner (without paired training data for supervised learning). What sets TUVF apart is its reliance on easily accessible images and untextured shape datasets, climinating the need for explicit 3D color supervision. This is crucial because, to knowledge, no large-scale realistic textured 3D dataset is currently available. Even recent efforts like Objaverse (Deitke et al., 2023), while providing substantial 3D data, have limited inter-class sample numbers (in the hundreds), and most of the textures are not photorealistic (compared to natural images). TUVF also enables direct texture transfer through dense shape correspondence. Such property is also significant to the field because post-processing texture transfer often requires user-provided correspondence (which is extremely expensive) ad hoc assumptions about the input geometry.
We summarize the differences in Table 10 below clarify the distinctions between TUVF and recent works. To assess the realism of the generated textures compared to natural images, we follow Texturify's ' setup and evaluate them on the CompCars dataset, which contains real car photos from various angles. It is important to note that this is not an apples-to-apple comparison for baselines that are not generalizable or require 3D supervision. This comparison doesn't aim to claim superior texture quality from our method. Instead, it highlights the differing approaches of these works and TUVF, given their distinct setups. The results also underscore the challenge of learning realistic texture representations due to the absence of large-scale realistic texture-shape shape datasets.
Table 10: Comparison to recent state-of-the-arts texture synthesis. G denotes Generalizable, indicating whether a method is encoder-based or iemiiizaaion-bbseed U denotes Unsupervised, indicating whether the method is free from 3D paired data supervision. C denotes Correspondence, indicating whether the method provides dense correspondence for texture transfer. The inference time is measured in minutes.
Method
G
U
C
FID
KID
Inference Time!
AUV-Net (Chen et al., 2022)
/
/
N/A
N/A
N/A
Texturify (Siddiqui et al., 2022)
/
V
59.55
4.97
0.01
TEXTure (Richardson al., 2023)
/
181.67
14.09
Point-U' UV Diffusion (Yu et al., 2023)
/
101.31
8.61
3.5
Text2Tex (Chen et al., 2023)
/
46.91
4.35
15
TUVF
/
/
41.79
2.95
0.01
T
LIMITATIONS
Geometry. Our work has some limitations inherited from Cheng et al. (2021) since our Canonical Surface Auto-encoder follows similar principles. Specifically, encoding the shape information of point cloud in global vector may cause fine details, such as corners and edges, to be blurred or holes to disappear after reconstruction. Similar to Cheng et al. (2021), we also observed that the correspondences predicted near holes the boundaries between parts might be incorrect, possibly due to the sparsity nature of point clouds and the limitations of the Chamfer distance. Future research should address these limitations.
Characteristic Seams. Seams are barely noticeable in our results. There are three reasons. Firstly, unlike prior works (Chen et al., 2022), we avoid cutting the shape into pieces and instead use a unified UV across all parts, resulting in seamless appearance without any distinct boundaries. Secondly, our UV mapper employs a non-linear mapping function trained with Chanfer loss, seamlessly connecting the UV coordinates without explicit stitching lines. Thirdly, unlike prior works that directly regress RGB values using UV features or RGB information alone, our MLP, F also takes the local coordinate additional input, representing local radiance field that effectively reduces the seams. However,
40

---

Published as a conference paper at ICLR 2024
Fail reconstruct fine details with complex topology.
(ii) Incorrect correspondences part boundaries.
Figure 30: Visualization of Failure Cases.
these design choices do not completely solve the seam issue. As illustrated in Figure 23, unsmooth correspondence can still result visible seams.
41


Reviews:
[{"review_id": "U3SWCvdvXH", "reviewer": "Reviewer_9F5H", "summary": "This paper proposes a novel approach for generating novel texture given a 3D shape. The key idea of this paper is to learn correspondence between UV coordinates and 3D points and apply a generative model in the UV space. The correspondence associates the generated texture features with the actual 3D location on the surface. It enables differential volume rendering wrt. the texture features. The authors use an adversarial learning setup on respective renderings.\nFor the geometry, they used the 3D cars and chairs from the ShapeNet dataset and photoshapes and CompCars datasets are used as 2D GT. A qualitative and quantitative comparison to Texturify, EpiGraf and more baselines show a decent performance.", "questions": "Since the proposed method uses a shape encoder decoder part, I’m wondering if this could be directly used to build a generative model for both shapes and textures.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and easy to understand. The method sections constraints useful figures and clear structure.\nThe research problem is important since a general formulation of UV mapping is an open topic.\nThe experimental sections contain many insights and support claims, e.g.Table 4 the ablation on the texture mapping network.", "weaknesses": "Even though the method requires a GT shape as input, the rendered shapes appear to have over-smoothed regions, e.g. the mirrors of the cars. \n\nIt is unclear how well the learned UV correspondence preserves surface areas in the UV space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aPG6ro6ABs", "reviewer": "Reviewer_KY39", "summary": "This paper proposes a textured shape generation method based on learned UV mapper and neural texture features. \nThe training is supervised by images rendered on 2D space. \nAfter training the method can support random textured generation for a given shape, texture editing, and texture transfer. \nThough the proposed pipeline is supported with a large mount of qualitative and quantitative results. \nI find is that the paper's technical contribution is little and the results shown in the paper are OK but not exciting(not comparable to recent diffusion based method). \nMost aspects have already been explored by previous papers(some important references are missing). \nIn addition, the intuition of using neural radiance field is really unclear. \nI do not feel using neural radiance field can bring any benefit since every property before the rendering is on the surface. \nThus, I am leaning towards rejection but also listen suggestions from other reviewers.", "questions": "See the weakness section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is overall clear with a good structure. \nReaders can follow the text easily.\nThe paper shows a lot of results and comparisons, which makes the pipeline more convincing. \nDetails of the network architecture are given in the supp, making reproduction easier.", "weaknesses": "The biggest issue is that the paper is not novel. \nMost part in the paper has been explored in previous papers. \nThough well combined, it only produces OK results instead of exciting results. \nFor example, recent PointUVDiffusion (Texture Generation on 3D Meshes with Point-UV Diffusion) can generate very realistic textures for a 3D shape.\nThough this paper can support more things like texture transfer via its UV mapper. \nBut actually, correspondence between shapes can also be obtained by postprocessing.\nSo I am wondering how the method compares to PointUVDiffusion.\n\nAnother issue is the intuition of using NeRF. \nThough NeRF is hot and can reconstruct 3D scenes very well and provide vivid results, for this given task, I strongly feel that NeRF is not necessary because everything before the rendering process is defined on the surface. \nWhy not use surface-based rendering?\nIs there any benefit to using NeRF (for example, view-dependent texture)? I do not see such results.\nFor example, AUV-Net learns an aligned UV space, which is pretty similar to the proposed pipeline, though it does not use a neural radiance field. \nSo I think comparing it to AUV-Net should be necessary to prove the advantage of NeRF. \n\nA formulation of the rendering process would be better in Sec. 3.2 or Sec3.3.\n\nReference:\nSINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aQk5y2iyJ0", "reviewer": "Reviewer_VxE5", "summary": "This paper proposes a method to learn a category-level latent space of both canonical surface and UV texture fields, from a 3D shape dataset and an unpaired collection of 2D images depicting this category of shapes. There are mostly three model components: (1) a canonical surface autoencoder that encodes the ground-truth shape into a latent code, which then gets decoded into points on a sphere, (2) a generative model that produces texture features conditioned on the style code, and (3) a differentiable radiance field rendering module that renders the generated texture onto the autoencoded shape, producing an image that can be compared against real images of the same shape category.\n\nFor the canonical surface autoencoder, Chamfer distance is used in this autoencoding task, and the latent code learned should ideally \"instruct\" us where on the canonical sphere each surface point falls onto, therefore providing a common space to anchor all shapes within the class (i.e., dense correspondence). The authors opted to turn point clouds into density volumes with differentiable Poisson surface reconstruction.\n\nFor the texture feature generator, the authors adopted StyleGAN-like style injection and made sure there's no interaction between neighboring pixels since vicinity in the UV space is often not physically meaning. RGB colors are not explicitly decoded from these features until the final rendering.\n\nFor the differentiable rendering module, the authors chose to adopt volume rendering even if the native shape representation is mesh from ShapeNet. Because of that, the authors discussed how to efficiently sample rays (rendering only near the object surface), convert point clouds into volume density (with differentiable Poisson surface reconstruction), and define radiance fields for points (via interpolating nearest surface points).\n\nThe authors show reasonable qualitative results where they can transfer texture to another shape in the category, make 3D-consistent texture edits, and find dense correspondence among shapes in the same class.", "questions": "As mentioned in \"Weaknesses,\" can we produce a mapping without learning by just raycasting from an enclosing sphere or something similar? I can see self-occlusion might be a blocker, but the learned mapping is not perfect either; this simpler alternative seems to deserve a try. \n\nWhy do we go through the complicated pipeline to make the shape volumetric? This looks like an even more severe issue than the first question. I hope we have a good justification (or I'm misunderstanding the paper); our community would hate to see \"volumetric == cool\" as the motivation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper does a good job presenting what is done with helpful visuals and is easy to follow.\n\nIt also tackles an interesting problem where one needs to learn a canonical space for a category shapes and simultaneously put textures onto the shapes, without paired 3D-2D data. By leveraging autoencoding and adversarial learning, the model learns meaningful patterns/correlations without explicit, direct supervision.\n\nDense correspondence emerging from autoencoding is also interesting and makes sense.", "weaknesses": "I have two significant concerns that need addressing before I can consider raising my ratings.\n\nWhile I understand how canonical surface autoencoding eventually leads to a mapping between a given shape and the canonical sphere, for rigid shapes like cars and airplanes, one can project the shape onto an enclosing sphere, e.g., via raycasting from the sphere to the shape, to obtain similar mappings -- \"similar\" as in all cars' front bumpers mostly map to the same location on the canonical sphere. If this simple approach produces similar results, the whole canonical surface autoencoding part becomes invalid. \n\nI understand the authors also show mappings learned for non-rigid objects like humans and animals, but these are all parametric models (SMPL or SMAL or whatever), so the canonical space for them is by definition well established and doesn't benefit from this work.\n\nThe second major concern is the whole deal of converting meshes or point clouds into density volumes. Volumetric approaches like NeRF are cool but I don't think everything needs to or should be volumetric. If we already have meshes, why throw away the face information, go into the point cloud regime, and then make the points volumetric? Each of these steps is lossy and complicates the method in an unnecessary way in my opinion. A concrete, much simpler alternative is retaining the face information for the points on the canonical sphere and use the same faces in the decoded shape. Then, we don't need differentiable surface reconstruction to return to the mesh domain in a suboptimal manner. What confuses me further is the adoption of volume rendering. With this alternative I proposed, one simply renders the mesh, which will be much more computationally efficient.\n\nIf these two concerns/confusions don't get cleared up, I view this paper as over-engineering a problem that could be tackled in a cleaner and simpler way, possibly also compromising the final quality given the extra lossy steps taken.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KIWYOgCHCJ", "reviewer": "Reviewer_K7dY", "summary": "This work studies generating high-fidelity textures of 3D  shapes. \n\nIt introduces TUVF that generates textures in a learnable UV sphere space, which allows the texture to be disentangled from the underlying shape and transferable to other shapes from the same category. \n\nIt uses a sampled texture code that represents a particular appearance style adaptable to different shapes and generates the texture in a canonical UV sphere space. It learns a canonical surface auto-encoder that maps any point on a canonical UV sphere to a point and normal on an object’s surface, which is transformed to indicator function values using the Poisson Surface Reconstruction algorithm and further transformed to density. This step is learned by Chamfer Distance on the surface points and the L2 losses on the indicator grid. Finally, the texture is learned by neural rendering with a patch-based discriminator.\n\nThe correspondence between the UV space and the 3D shape is automatically established during training.", "questions": "My main concern is about the contribution. The contribution of this work versus the work of text-driven texture synthesis should be further clarified.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "TUVF achieves much more realistic, high-fidelity, and diverse 3D consistent textures compared to previous approaches.\n\nIt achieves state-of-the-art results in both the experiments on synthetic and real-world object datasets.\n\nIt improves texture control and editing.\n\nThe use of the Poisson Surface Reconstruction algorithm is very interesting and contributes insights for the community.", "weaknesses": "There still exist distortions in texture.\n\nThe contribution of texture generation is limited due to recent text-driven texture synthesis work, such as Text2Tex and TEXTure: Text-Guided Texturing of 3D Shapes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)