Learning to Transfer Heterogeneous Translucent Materials from a 2D Image to 3D Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-14 10:26:30

Abstract: Great progress has been made in rendering translucent materials in recent years, but automatically estimating parameters for heterogeneous materials such as jade and human skin remains a challenging task, often requiring specialized and expensive physical measurement devices. In this paper, we present a novel approach for estimating and transferring the parameters of heterogeneous translucent materials from a single 2D image to 3D models. Our method consists of four key steps: (1) An efficient viewpoint selection algorithm to minimize redundancy and ensure comprehensive coverage of the model. (2) Initializing a homogeneous translucent material to render initial images for translucent dataset. (3) Edit the rendered translucent images to update the translucent dataset. (4) Optimize the edited translucent results onto material parameters using inverse rendering techniques. Our approach offers a practical and accessible solution that overcomes the limitations of existing methods, which often rely on complex and costly specialized devices. We demonstrate the effectiveness and superiority of our proposed method through extensive experiments, showcasing its ability to transfer and edit high-quality heterogeneous translucent materials on 3D models, surpassing the results achieved by previous techniques in 3D scene editing.

Main paper: 
 introduction: 
 Translucent materials, exhibiting high scattering characteristics, are ubiquitous in our daily lives, ranging from precious jewels to the intricate cellular structures found in biological organisms. These materials play vital roles across a multitude of applications. The rendering of such translucent materials has witnessed remarkable progress through physically based rendering techniques [33] , including path tracing [20] and volumetric path tracing [32] models, seamlessly integrated into rendering engines like Mitsuba3 [18] , among others.While rendering materials have matured, parameter estimation for these materials has historically involved manual adjustments [18, 31] , consuming substantial time and financial resources. Efforts to automate this process have been ongoing [6, 8, 13, 24, 25] , but they are limited to homogeneous materials and struggle to extend to heterogeneous ones like jade, marble, or human skin. The latter is characterized by heterogeneity, surface microstructure, and short scattering mean free paths, making manual estimation impractical.Addressing the challenge of parameter estimation for heterogeneous translucent materials, some researchers resort to physical measurement instruments [9, 14, 37, 41, 43, 44] . However, this approach is specialized and costly, limiting its accessibility for ordinary consumers or general computer graphics applications.Recently, InverseTranslucent [5] introduced a 3D reconstruction method for translucent objects using low-cost handheld acquisition setups. This approach effectively estimates material parameters through integrating multi-view images and a translucent differentiable renderer. Nevertheless, limitations exist, including the need for fixed lighting conditions during image acquisition and the manual collection of multi-view images, restricting their applicability.In this paper, we propose a novel method for transferring heterogeneous translucent materials from a single image. As illustrated in Figure 1 , our approach enables the direct estimation of relevant parameters under natural lighting conditions and seamlessly transfers the material onto a 3D model. To begin, we strategically select sparse viewpoints that minimize redundancy while ensuring comprehensive coverage of the model. Utilizing a translucent renderer, we render translucent initial images from the selected sparse viewpoints. Subsequently, we perform material transfer sequentially on single-view rendered images, effectively transferring the heterogeneous translucent material to the initial image. This approach successfully preserving the structure and light properties of the initial image while accurately transferring the heterogeneous translucent materials. Inspired by the work of Instruct-NeRF2NeRF [15] , we introduce an innovative iterative editing-optimization strategy. By iteratively editing and performing inverse rendering, we update the edited translucent results onto the 3D model, ensuring consistency and coherence in our multi-view material editing results. In sum, our contributions include:• Proposal of a method for transferring heterogeneous translucent materials based on a single image. • Design of a translucent material transfer editor preserving both lighting and structure, enabling high-quality material transfer for 3D models with initialized translucent materials.• Introduction of an iterative editing-optimization strategy ensuring consistency in material editing for heterogeneous translucent materials across multiple viewpoints. 
related work 2.1 translucent rendering: 
 Achieving realistic translucent rendering involves simulating the scattering and transmission of light through the material. The Bidirectional scattering surface reflectance distribution function (BSS-RDF) is generally used to simulate subsurface scattering effects. Jensen et al. [19] proposed a practical dipole model that can be effectively used in sampling techniques for conventional ray tracers to represent materials scattered by homogeneous subsurfaces. Donner et al. [8] combines photon tracing with diffusion to efficiently render highly scattering translucent materials, and also accounting for internal blockers, complex geometry, translucent inter-scattering, and transmission and refraction of light at the boundary causing internal caustics. d'Eon et al. [6] presents a new analytic BSSRDF for scattering within multilayer translucent materials with arbitrary levels of absorption and under all-frequency illumination which creates accurate results under high-frequency illumination. Vicini et al. [42] proposed a new shape-adaptive BSS-RDF model that retains the efficiency of prior analytic methods while greatly improving overall accuracy. InverseTranslucent [5] accounts for both surface reflection and subsurface scattering to represent translucency using a BSSRDF model. Neural rendering is gaining increasing popularity due to its exceptional fitting capabilities. Suhail et al. [39] combines the strengths of classical light field rendering and geometric reconstruction methods, learning to accurately represent view-dependent effects like translucency from a sparse set of views by operating on a 4D light field representation. Yu et al. [45] proposed Object-Centric Neural Scattering Functions (OSFs) for learning to reconstruct the appearance of opaque and translucent objects.Simulating the appearance of translucent materials requires accurate physical parameters. However, obtaining physically accurate parameters for scattering materials remains a challenging task. Differentiable rendering algorithms strive to estimate partial derivatives of pixels in a rendered image with respect to scene parameters, which is difficult because visibility changes are inherently non-differentiable. Certain methods [16, 29, 47] employ path tracing based on Monte Carlo estimation, and by enabling differentiable rendering, they obtain physically-based material estimates from real images. Neural-PBIR [40] introduce a neural material and lighting distillation stage to achieve high-quality predictions for material and illumination and perform physics-based inverse rendering (PBIR) to refine the initial results and obtain the final high-quality reconstruction. For translucent materials, Gkioulekas et al. [13] combine stochastic gradient descent with Monte Carlo rendering and a material dictionary to invert the radiative transfer equation and measure scattering properties. Then, Gkioulekas et al. [12] tackling the problem of heterogeneous inverse scattering from simulated measurements of different computational imaging configurations. InverseTranslucent [5] uses a differentiable subsurface scattering renderer to represent translucency with a heterogeneous BSSRDF, leveraging low-cost handheld acquisition setups. Li et al.[25] used a physically-based renderer and a neural renderer to estimates homogeneous subsurface scattering parameters from only a pair of captured images of a translucent object.Automated manipulation of surface materials in 3D models plays a vital role in 3D modeling, enabling users to modify and control the appearance of the models' surfaces. Khan et al. [22] present a method for automatically replacing one material with another, completely different material with only a single high dynamic range image as input.The transformations range from applying a texture to the surface of an object, to the application of any arbitrary BRDF. Subedit [38] decouples the BSSRDF non-local scattering effect into the product of two local scattering profiles, enabling a method for editing heterogeneous subsurface scattering materials obtained from real-world samples. Liu et al. [26] proposes an end-to-end network for image-based material editing, replicating the forward image formation process. Diffusion models are becoming increasingly popular in 3D generation. By combining diffusion models and differentiable rendering, text-guided material generation and editing have become highly effective [27, 35] . Unlike the explicit representation of the surface in 3D models, NeRF can implicitly represent models or scenes. Instruct-Nerf2Nerf [15] propose a method for editing NeRF scenes using text instructions.Transferring materials using images is a challenging task. For image-based material editing, editing a single image becomes a crucial step. Given a reference image, convolutional neural networks (CNNs) can be used to transfer the style from the reference image to a content image [11, 30] . Inspired by style transfer, StyleGAN [21] generates images by manipulating latent vectors. Hyperstyle [1] inverts real images into the latent space, enabling real image editing. Recently, large-scale text-driven generative models have received widespread attention for their ability to generate highly diverse images based on given text prompts. Prompt-to-Prompt [17] controls editing solely through text, enabling local edits by replacing words. VCT [4] utilizes inversion techniques with a reference image to translate visual concepts while preserving the source image's content. InstructPix2Pix [2] is a method for instructional image editing that can apply specific styles to the edited image. 
methods: 
 The methodological overview is presented in Figure 2 . First, the Efficient Viewpoints Selection (Section 3.2) method is employed to select multiple viewpoints that capture images of the 3D model, ensuring the selected viewpoints minimize redundancy and cover the entire surface of the model. Subsequently, a translucent material is initialized homogeneously for the 3D model. We use translucent differentiable renderer (Section 3.3) to render initial images from the selected viewpoints, and these images constitute the initial translucent dataset. After initialization, an iterative process is then undertaken, involving an image editing process and an inverse rendering process, to update the translucent dataset and the translucent materials: The image editing process (Section 3.4) utilizes the translucent initial images as the structure and lighting conditions, and a style image as the style and texture conditions. A proposed structurepreserving translucent editor is employed to edit the current material renderings, and the edited translucent results are used to update the translucent dataset. The preliminaries (Section 3.1) provide the techniques used in the editing model.The inverse rendering process randomly selects images from the translucent dataset as the supervision, and then optimizes the material parameters by performing inverse rendering through the translucent differentiable renderer.Through the iterative process of our consistent translucent material update (Sec.3.5), we alternately editing and inverse rendering to update our translucent materials. The iterative process preserves the structure and lighting by using the initial image and the current image as conditions, allowing for the consistent transfer of the style image to the translucent material.DDIM sampling: Text-guided diffusion models have beeb a widely researched area. The primary objective of text-guided diffusion models is to, given a random noise vector 𝑧 𝑇 , denoise 𝑧 𝑇 conditioned on a given text 𝑃, until obtaining an output image latent 𝑧 0 that is close to the description in 𝑃. To this end, the network 𝜖 𝜃 is trained to predict the added noise, following the optimization objective:EQUATIONwhere C represents the conditional embedding of 𝑃, and 𝑧 𝑡 is the noisy sample with noise added according to the timestamp 𝑡 on the original sample 𝑧 0 . During inference, for a given noise vector 𝑧 𝑇 , the pre-trained network is used to sequentially predict the noise 𝜖 𝜃 and remove it through 𝑇 steps, ultimately generating a clear image. The Diffusion Denoise Implicit Model (DDIM) [36] sampling is employed to generate images, with the computation formula:𝑧 𝑡 -1 = √︂ 𝛼 𝑡 -1 𝛼 𝑡 𝑧 𝑡 + √︂ 1 𝛼 𝑡 -1 -1 - √︂ 1 𝛼 𝑡 -1 • 𝜀 𝜃 (𝑧 𝑡 , 𝑡, C), (2)where 𝛼 𝑡 is the noise scaling factor sequence defined by the diffusion process, which describes the relative strength of the noise at each step as it transitions from a completely noisy state to the original data distribution.Classifier-free Guidance: Classifier-free Guidance (CFG) is a key technique to control the influence of the text condition on the image generation process, without relying on a separate text classifier model. CFG introduces an additional unconditional prediction 𝜀 𝜃 (𝑧 𝑡 , 𝑡, ∅), which ∅ represents an empty text embedding. The final noise prediction used for denoising is then a weighted average of these two predictions:EQUATIONThe weighting factor 𝜔, known as the guidance scale, determines how much the final prediction is influenced by the text condition versus the unconditional generation. DDIM Inversion: DDIM Inversion is a complementary technique that allows for the reconstruction of the initial noise latent from an existing image. This capability enables the editing of generated images by manipulating the recovered noise latent.DDIM Inversion takes the final noisy image encoding 𝑧 𝑇 and works backwards through the denoising steps to recover the initial noise encoding 𝑧 0 that was used to generate that image, which can be formulated as:𝑧 𝑡 +1 = √︂ 𝛼 𝑡 +1 𝛼 𝑡 𝑧 𝑡 + √︂ 1 𝛼 𝑡 +1 -1 - √︂ 1 𝛼 𝑡 -1 • 𝜀 𝜃 (𝑧 𝑡 , 𝑡, C). (4)Here, 𝑧 𝑡 represents the noise encoding at each step 𝑡, 𝛼 𝑡 is a noise scaling factor, and 𝜀 𝜃 (𝑧 𝑡 , 𝑡, C) is the noise prediction made by the diffusion model, conditioned on the current encoding 𝑧 𝑡 , timestep 𝑡, and any relevant conditioning information C (e.g. a text prompt). Attention Control The attention control mechanism, proposed by Prompt-to-Prompt [17] , aims to replace the attention maps in the diffusion process using the following formula:EQUATIONwhere 𝑀 𝑡 is the original attention map, 𝑀 * 𝑡 is the edited attention map, and 𝜏 is a timestamp parameter that determines the step until which the attention map replacement is applied. This soft attention constraint allows the method to preserve the original composition in the diffusion steps, while enabling more targeted editing.Existing methods for generating geometric and material properties based on diffusion models typically use either a large range of random viewpoints (e.g., DreamFusion [34] , Fantasia3D [3] ) or fixed viewpoints (e.g., TEXTure [35] , Instruct-Nerf2Nerf [15] ). While effective, these approaches suffer from either excessive redundancy (random viewpoints) or lack of adaptability and reliability (fixed viewpoints) for our translucent material transferring task. To address these issues and satisfy our requirement of covering the vast majority of the surface with only a few viewpoints. we propose a method called Efficient Viewpoints Selection (EVS), which consists of two steps: Viewpoint Sampling: We first normalize the geometry of the mesh, then sample points on the mesh surface. For each sampled point, we define a corresponding viewpoint as the intersection between the surface normal at that point and a bounding sphere of radius 2 around the model. We set the camera orientation to point from each viewpoint towards its associated surface sample point and get sampled viewpoints 𝑉 𝑠𝑎𝑚𝑝𝑙𝑒𝑑 . Viewpoint Selection: For each candidate viewpoint 𝑣 𝑖 in 𝑉 𝑠𝑎𝑚𝑝𝑙𝑒𝑑 , we devide the surface of mesh into 𝑆 𝑛 𝑒𝑤 (new captured surface) and 𝑆 𝑜 𝑡ℎ𝑒𝑟𝑠 (uncaptured or already captured surface). Then we render an image which the the pixel of 𝑆 𝑛𝑒𝑤 set to 1 and 𝑆 𝑜𝑡ℎ𝑒𝑟𝑠 set to 0. We then evaluate the quality of each viewpoint using a scoring function that aims to maximize the coverage of surface at a fine angle. The scoring function is defined as:EQUATIONwhere 𝑆𝑐𝑜𝑟𝑒 (𝑣 𝑖 ) is the score of the 𝑖-th viewpoint, 𝑃 (𝑣𝑎𝑙𝑢𝑒=1) is pixel with a value of 1 in rendered image (representing new captured surface), and 𝐴(𝑆 new ) is the total area of the new captured surface from the current viewpoint. By calculating this scoring function for each candidate viewpoint and iteratively selecting the viewpoint with the highest score, the algorithm can efficiently choose a set of viewpoints 𝑉 𝑠𝑒𝑙𝑒𝑐𝑡𝑒𝑑 that cover nearly the entire surface of the 3D model, as demonstrated in Figure 3 , the selected viewpoints after several iterations can cover nearly the entire surface of the bunny, demonstrating the effectiveness of our algorithm.InverseTranslucent [5] introduced an end-to-end approach to simultaneously estimate the complex geometry and heterogeneous translucent properties of translucent objects from photographs using inverse rendering techniques. This method uses a heterogeneous BSSRDF to represent translucency, and extends the framework of Path-Space Differentiable Rendering (PSDR) [46] to accommodate both surface reflection and subsurface scattering. The following integral formula is used to simulate the subsurface light transport problem:EQUATIONHere, the outgoing radiance 𝐿 𝑜 is the combined result of subsurface scattering and surface reflection. 𝜌 𝑟 represents the BRDF model, calculating the surface reflection contribution at x 𝑜 . 𝜌 𝑠 represents the BSSRDF model, calculating the subsurface light transport contribution from incident light at x 𝑖 in direction 𝜔 i to the outgoing direction at x 𝑜 . The BRDF model describes the scattering behavior of light on rough surfaces, parameterized by the GGX distribution with surface roughness 𝛽 and refractive index 𝜂. For the BSSRDF model, the practical dipole model proposed in [19] is used to compactly represent homogeneous subsurface scattering materials. Spatially-varying parameters following [38] are introduced to model heterogeneity, with parameters including scattering albedo 𝛼 and extinction coefficient 𝜎 t . These physical parameters can be represented either as single values or as textures, collectively referred to as the parameter vector 𝜋.To optimize the parameter vector 𝜋, the loss function 𝑔(I(𝜋)) is minimized. To effectively optimize the model in the presence of Monte Carlo noise introduced by the BSSRDF integral, a dual-buffer method is employed to evaluate L2 image loss:EQUATIONThis provides an unbiased estimate of the difference between the rendered image I(𝜋) generated based on parameters 𝜋 and the reference image I ref .It effectively computes the loss value based on two Monte Carlo estimates, helping to reduce potential gradient estimation bias caused by correlations between a single rendering and its derivatives, even when using low sampling rates to ensure correct convergence of the optimizer. In contrast to InverseTranslucent [5] , our optimization process fixed the 3D model's geometric parameters and updated only the material-related parameters. We employed the translucent differentiable renderer to perform rendering and inverse rendering.Pivot Turning Inversion: A key challenge with standard DDIM inversion is the accumulated error when using CFG in text-guided diffusion models. The CFG guidance scale 𝜔 amplifies this error, leading to visual artifacts. The key idea of Pivot Turning Inversion (PTI) is to modify the unconditional embedding 𝜑 𝑡 associated with each timestamp 𝑡 to better match the given image and reduce error.PTI inverts the initial image 𝑥 𝑖𝑛𝑖 𝑣 𝑖 and the current rendered image 𝑥 𝑐𝑢𝑟 𝑣 𝑖 in viewpoint 𝑣 𝑖 . Taking the inversion of 𝑥 𝑐𝑢𝑟 𝑣 𝑖 as an example: PTI optimizes the unconditional embedding 𝑣 𝑐𝑢𝑟 𝑡 at each timestamp 𝑡 to minimize the distance between the ground-truth noise-free latent 𝑧 𝑐𝑢𝑟 0 and the denoised latent ẑ0 (𝑧 𝑐𝑢𝑟 𝑡 , 𝑣 𝑐𝑢𝑟 𝑡 ) estimated by the pre-trained model: minEQUATIONHere, 𝑧 𝑐𝑢𝑟 0 is the noise-free latent of the editing image 𝑥 𝑐𝑢𝑟 𝑣 𝑖 , and ẑ0 (𝑧 𝑐𝑢𝑟 𝑡 , 𝑣 𝑐𝑢𝑟 𝑡 ) is the denoised latent estimated by the pre-trained model. The same process is applied to 𝑥 𝑖𝑛𝑖 𝑣 𝑖 . This approach learns an unconditional embedding that can perfectly reconstruct the inverted image with the initial noise latent. Multi-concept Inversion: Multi-concept Inversion (MCI) focuses on learning a conditional embedding 𝜑 𝑠𝑡 𝑦 that extracts rich semantic information from the style image 𝑥 𝑠𝑡 𝑦 . However, the negative embedding used in prior methods like Textual Inversion [10] and DreamArtist [7] are not necessary in our case, as they may conflict with the unconditional embedding learned through PTI.Therefore, we adopt a single positive multi-concept embedding approach used by VCT [4] . We fix the parameters of the pre-trained diffusion model and optimize the style embedding 𝜑 𝑠𝑡 𝑦 to minimize the following objective:EQUATION)Here, the style embedding 𝜑 𝑠𝑡 𝑦 represents the embedding of the style image 𝑥 𝑠𝑡 𝑦 , and 𝑧is the noisy latent of the style image at each timestamp 𝑡. The goal of this objective is to learn the style embedding that best predicts the noise residual 𝜀, effectively capturing the essential visual concepts in the style image, which can aid our transfer process. Structure-preserving Translucent Editor: The proposed structurepreserving translucent editor aims to transfer the style image to the current rendered image while preserving the structure and lighting conditions of the translucent initial image. The dual stream denoising architecture employed by VCT [4] , which, although effective in first editing, resulted in unstable style transfer and loss of translucency and structural information during iterative editing.To address these issues, we propose a structure-preserving translucent editor illuminated in Figure 4 . The editor utilizes two branches:Structure preserving Branch: We use PTI to invert the initial translucent image 𝑥 𝑖𝑛𝑖 𝑣 𝑖 of viewpoint 𝑣 𝑖 , obtaining its text embedding 𝜑 𝑖𝑛𝑖 𝑣 𝑖 and initial noise latent 𝑧 𝑖𝑛𝑖 𝑇 , which are used to perfectly reconstruct the translucent initial image. During the reconstruction ) note that the 𝜔 in the above two formulas is the same.During the simultaneous denoising process described above, we use the attention control mechanism in Eq.5, extracting 𝑀 * 𝑡 from 𝐵 * to replace 𝑀 𝑡 of 𝐵. This ensures that the structure and light conditions of the initial image are perfectly preserved during the style transfer.The editing process is as follows:EQUATIONWe continuously render-edit-optimize, and during the editing process, the structure preserving branch always uses the initial image 𝑥 𝑖𝑛𝑖 𝑣 𝑖 , while the style transfer branch's input image iteratively changes from current rendered 𝑥 𝑐𝑢𝑟 𝑣 𝑖 , and the edited result 𝑥 𝑡𝑔𝑡 𝑣 𝑖 , gradually achieving global consistency with the style image across multiple viewpoints.Editing the rendered translucent images only once may generate good results, but could lead to inconsistencies across multiple viewpoints, where some edited images do not perfectly transfer the style and remain close to the originals. Instruct-Nerf2Nerf [15] adopts an iterative editing approach, alternately updating the dataset to achieve consistent NeRF scene editing. Inspired by this alternating update strategy, our method uses iterative editing to update the translucent dataset. Our structure-preserving translucent editor ensures stability across multiple edits, preventing "drift" and loss of structure and translucency information during the iterative material updates. Figure 5 shows our iterative editing-optimization strategy for heterogeneous translucent materials across multiple viewpoints. Translucent Material Initialization: Material initialization is crucial as it determines the optical information. As most semantic information is concentrated in non-black areas, and higher color values are more easily perceived by the attention mechanisms of diffusion models, our translucent initialization meets the requirement of preserving only structure and lighting for material transfer. Opaque materials are unable to capture translucency details due to the lighting model's inability to describe subsurface scattering. As demonstrated in Figure 6 , without translucent initialization, edited results clearly lose translucency details under the same lighting and viewing conditions compared to renderings with translucent initialization, which achieve high-quality texture transfer while preserving initial lighting. The detail of our translucent initialization is in Section 4.1.The translucent dataset, denoted as 𝐷 𝑇 , is designed to facilitate the transfer of materials from 2D to 3D. Initial rendered images 𝑥 0 𝑣 𝑖 are rendered for each viewpoint, and this collection forms the initial translucent dataset 𝐷 𝑇 . Iterative Translucent Dataset Update: After initializing the translucent dataset 𝐷 𝑇 , an iterative update process is performed, alternating between the translucent style image style transfer 𝐸 and the translucent material parameters optimization 𝑂.In process 𝐸, the current image 𝑥 𝑗 𝑣 𝑖 in 𝑗-th iteration is rendered and edited to obtain 𝑥 𝑗 𝑡𝑔𝑡 , which updated the 𝐷 𝑇 . During the process, 𝐷 𝑇 transitions from the old state to the new state. Next editing process will edit current image rendered in next view (𝑣 𝑖 → 𝑣 𝑖+1 ).In process 𝑂, an image from 𝐷 𝑇 is randomly selected for supervision, and the translucent parameters are optimized through a differentiable renderer using inverse rendering.The combination of the 𝐸 and 𝑂 allows for the gradual refinement of 𝐷 𝑇 . By alternating between these processes, the material properties are iteratively updated to match the desired style while maintaining consistency across different viewpoints. This iterative editing-optimization strategy ensures consistent material editing for heterogeneous translucent materials across multiple viewpoints. 
results: 
 Figure 1 shows the results of our method of editing various standard models with different style images. The selected results can be found in the appendix. Our method accomplishes numerous challenging edits, including those involving semitransparent materials like jade and marble found in nature. We elaborate on the implementation details of our method in Section 4.1. In Section 4.2, we qualitatively compare our approach with renowned works such as 𝑆𝑡𝑦𝑇 𝑅 2 [5] , Artflow [23] , and InstructPix2Pix [2] in image-to-image (I2I) tasks. To validate our method, we conducted ablation experiments against a set of ablative baselines in Section 4.3.Our work was conducted with the following specifications: The translucent renderer was performed at a resolution of [512, 512] pixels, and the field of view (FOV) was set to 45 degrees. In the rendered scene, a pinhole camera was placed on a sphere with a radius of 2, and a point light was utilized, positioned around the camera at a distance of 0.2. The power of the point light was set to 20000. To enhance rendering precision, the entire scene was scaled by a factor of 10.For our translucent material initialization, the extinction coefficient 𝜎 𝑡 , which describes light attenuation through the medium, was initialized to [1.5, 1.5, 1.5], a value close to that commonly observed in translucent materials. Higher values of 𝜎 𝑡 indicate stronger absorption and scattering, resulting in faster decay. The scattering albedo 𝛼, which represents the probability of light scattering at a location in the medium, was initialized to [0.9, 0.9, 0.9]. All these parameters were presented in the form of textures with a resolution of [512, 512] pixels.A total of 500 iterations were conducted during the optimization process, which took approximately 18 minutes on two NVIDIA 3090 GPUs. For the optimization process, the same loss function as InverseTranslucent [5] was used, along with an AdamW optimizer [28] . After performing 50 optimization iterations, an editing process was conducted. For the editing process, the style image embedding was initially trained for 500 iterations. During the inference stage, a cross-attention ratio of 0.2 and a self-attention ratio of 0.9 were used.Comparison with image editing models. Figure 7 shows the result of a comparison between our methods and the baseline image editing methods. As for baselines, we selected some state-of-theart methods, including InstructPix2Pix, 𝑆𝑡𝑦𝑇 𝑅 2 , and Artflow. The results indicate that our approach can edit images while preserving the translucent information of the material, as well as the geometric structure of the model and lighting features. Specifically, in the results from 𝑆𝑡𝑦𝑇 𝑅 2 , the image editing outcome exhibits colors similar to the style image but loses a significant amount of texture features and translucent information. The results in Artflow retain more texture features from the style image compared to 𝑆𝑡𝑦𝑇 𝑅 2 but still fall short of the desired effect. In the results of InstructPix2Pix with Multi-concept inversion, the method effectively preserves the texture details from the style image by employing multi-concept inversion to generate concept embeddings representing complex visual concepts. Moreover, the results demonstrate the ability to retain translucent information and lighting features. However, regrettably, this method loses the geometric structure of the model in the image. Comparison with 3D editing models. We compare our method with other 3D model editing approaches. As for baselines, we selected prominent methods in 3D scene editing, including Instruct-NeRF2NeRF [15] . Figure 8 shows the results of our method alongside those of the baseline methods on 3D models. NeRF2NeRF performs editing on NeRF scenes guided by text instructions, but its output exhibits limitations in expressing surfaces with complex materials. In an effort to improve the outcomes, we substituted the image editing method in Instruct-Pix2Pix [2] from NeRF2NeRF with other image editing techniques. However, since NeRF2NeRF is primarily trained on 3D NeRF scenes, it is evident that the baseline methods based on NeRF2NeRF and its variants still encounter difficulties in editing the translucent materials. Nevertheless, our method successfully transfer the style image to the 3D model's translucent material with consistence. Comparison with variants of various 3D scene editing methods. The bunny and chair models are edited using images with jade and apophyllite materials. Our results effectively preserve the texture details and translucent information of the style images.In order to verify the necessity of each component in our method, we conducted ablation studies. The qualitative differences are shown in Figure 9 : Dense view update. In this baseline, we do not employ our efficient viewpoint selection method to select high-quality viewpoints. Instead, we obtain dense and uniform viewpoints around the model. Images from all viewpoints are edited only once, and all participate in iterative training. From the results, dense viewpoints lead to the averaging of texture features in the final iteratively edited outcome, causing a loss of material features. One time update. The next method adopts a strategy of updating the dataset only once. In this baseline, we first use the efficient viewpoint selection method to choose good viewpoints, but during iterative dataset updates, we edit the rendering images of each viewpoint only once. Results indicate that a single edit is not sufficient to extract the features from the style image. Iterative update. Building upon the One-time update approach, we adopt the strategy from Instruct-NeRF2NeRF [15] to iteratively edit the translucent dataset. However, during the editing process, we refrain from using the initial image. Although this method can produce decent editing results, the outcome lacks a significant amount of translucent information in the absence of the initial image. This experiment highlights the importance of the structure preserving branch of our structure-preserving translucent editor.  
conclusion: 
 This paper introduces an innovative method for estimating parameters of heterogeneous translucent materials from a single image, enabling direct parameter estimation under natural lighting conditions and material transfer onto a 3D model. Our contributions encompass a novel single-image-based transfer method for heterogeneous translucent materials, a translucent material transfer editor preserving lighting and structure, and an iterative editing approach ensuring consistency across multiple viewpoints. Despite the practical and accessible nature of our approach, overcoming limitations of existing methods, there are some constraints. Recovering the geometry and appearance of translucent objects from sparse views under strong illumination remains a challenge, requiring further improvement in our method's learning capability for materials occluded by highlights. Additionally, our algorithm currently lacks support for scene-level 3D editing under ambient lighting. Despite these limitations, we believe our method marks a significant step in 3D object material editing, offering new possibilities for 3D modelers and artists to efficiently transfer translucent materials onto 3D models.  
 
 Appendix:  


Decision: This paper received four reviews. Three reviewers are positive about this paper. They consider that the workflow is well designed and the results are strong. The negative reviewer is concerned with the lack of quantitative results and the experiments are not sufficient. The rebuttal provides some quantitative results. Given the overall positive feedback received by this paper, we are happy to recommend the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a new approach to estimate the parameters of heterogeneous translucent materials from a single 2D image and transfer them to a 3D model.
strength_weakness: strength_weakness: The paper is well structured and the experimental results are superior to the existing SOTA.
limitations: limitations: 1) In the paper, the method is described as minimizing redundancy, but isn't some degree of redundancy required in 3D modeling processing? Is it used to validate performance by cross checking redundant areas or not?

2) Is it unavoidable that the result of StyTR-2, Artflow in Figure 7 has a different color background? It looks like it has a worse effect due to the background. Compared to the other two, StyTR-2 seems to be targeting relatively similar translucent materials. The results in the StyTR-2 paper look relatively good compared to the results shown in Figure 7. Is there a difference between the experimental environment in StyTR-2 and this paper? Is it a difference in samples? Is it possible to do a comparison with the samples used in the StyTR-2 paper?

3) Modeling performance for translucent materials is not only about the results from a single viewpoint, but also about the naturalness of the change as the viewpoint is shifted, what are the results? I know it's hard to show this in the paper, but maybe you could show it in a video, or maybe a metric to compare the degree of naturalness.

4) I understand the difficulty, but is there any way to show an objective comparison? I think it is ambiguous to judge what is better just by looking at the images in Figure 9 and the results in the supplementary document. 

5) One of the main contributions of the paper is efficient viewpoint selection, and it's a shame that the results are missing. There is a lack of experiments that show the results of efficient viewpoint selection. For example, experiments on what performance reduction is caused by excessive viewpoint selection, etc.

6) Other minor comments include the following.
- The text in Figure 2 is too small and hard to read.
- Page 4, right column, lines 438, 439, font style error.
- Page 4, right column, line 440, typo “the the”.
paper_summary: paper_summary: To automatically estimate the parameters for heterogeneous translucent materials, this paper proposes a single-image based method that includes viewpoint selection, translucent material initialization, and an iterative editing-optimization strategy.
strength_weakness: strength_weakness: - This paper proposes a workflow where each stage is carefully designed. The effectiveness is proved by a lot of solid visual results presented in the paper.
- The structure-preserving translucent editor integrates components from the latest progress in image generation while faithfully preserving the structure and lighting, overcoming the drawbacks of the traditional style-transfer methods.
limitations: limitations: - The organization of some sections is a little messy. E.g., Sec 3.3 and the PTI part of Sec 3.4 mainly explain the prior methods, so they may be moved to the preliminaries. And Fig 6 should be moved to the ablation study.
- It’s hard to figure out process E and process O mentioned in Sec 3.5. It would be better to visualize them for better understanding.
- The conclusions would be more convincing if more quantitative results are shown in the paper: e.g., some metrics to compare the texture similarity between the results and the reference image; and the accuracy of the estimated material parameters.
paper_summary: paper_summary: This paper proposes a method for transferring heterogeneous translucent materials from a single image. Their method enables the direct estimation of relevant parameters under natural lighting conditions and seamlessly transfers the material onto a 3D model. To achieve this objective, they introduce an innovative iterative editing-optimization strategy, which iteratively edits and performs inverse rendering. To achieve this objective, they introduce an innovative iterative editing optimization strategy, which iteratively edits and performs inverse rendering. To ensure consistency and coherence in our multi-view material editing results, they update the edited translucent results onto the 3D model.
strength_weakness: strength_weakness: 1. The description of the paper is clear and easy to understand.
2. This paper proposes a novel method to direct the estimation of relevant parameters under natural lighting conditions and seamlessly transfer the material onto a 3D model.
3. This paper proposes an iterative editing-optimization strategy to ensure consistency in material editing for heterogeneous translucent materials across multiple viewpoints.
4. The Efficient Viewpoints Selection (EVS) is interesting for the viewpoint chosen.
limitations: limitations: 1. The paper has few experiments; it is recommended to include more comparative experiments to verify the validity of the method.
2. The paper does not conduct a quantitative analysis of the experiment。

Further comments:
1. What results will it produce for shapes with more complex topology? For example, shapes containing more genus.
2. Can multi-modal information be added to enhance effect and controllability?
paper_summary: paper_summary: This paper is about rendering translucent material to 3D models. They present a new approach for estimating and transferring the parameters of heterogeneous translucent materials from a single 2D image to 3D models. The contribution of the methods contains a method for transferring heterogeneous translucent materials based on a single image. A translucent material transfer editor preserving
 both lighting and structure. An iterative editing-optimization strategy. The experiment results shows it has good performance.
strength_weakness: strength_weakness: 1. Compared to the previous method, the rendering performance of the proposed method is absolutely better and the results are convincing. 
2. For the experiment results, the model proposed in this paper outperforms the baseline model and the paper analyzes enough numbers of evaluation metrics.
3.  It is a well-formatted thesis. It has proper headings, figures, tables, and illustrations, making it easy to read and follow.
4. The contribution of the methods is solid.
limitations: limitations: 1. The experiment results are not enough. There are only a few number of rendering results. It makes readers worried about the generalization performance.
2. There are no quantitative results. Maybe the author can seek some quality assessment methods to evaluate the quality of the generated results and compare them to other methods.