SyncTweedies: A General Generative Framework Based on Synchronized Diffusions

Authors: jaihoon  kim
juil  koo
kyeongmin  yeo
minhyuk  sung
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 15:11:11

Abstract: We introduce a general diffusion synchronization framework for generating diverse visual content, including ambiguous images, panorama images, 3D mesh textures, and 3D Gaussian splats textures, using a pretrained image diffusion model. We first present an analysis of various scenarios for synchronizing multiple diffusion processes through a canonical space. Based on the analysis, we introduce a synchronized diffusion method, SyncTweedies, which averages the outputs of Tweedie’s formula while conducting denoising in multiple instance spaces. Compared to previous work that achieves synchronization through finetuning, SyncTweedies is a zero-shot method that does not require any finetuning, preserving the rich prior of diffusion models trained on Internet-scale image datasets without overfitting to specific domains. We verify that SyncTweedies offers the broadest applicability to diverse applications and superior performance compared to the previous state-of-the-art for each application. Our project page is at https://synctweedies.github.io.

Main paper: 
 introduction: 
 Image diffusion models [47, 38] have shown unprecedented ability to generate plausible images that are indistinguishable from real ones. The generative power of these models stems not only from their capacity to learn from a vast diversity of potential data but also from being trained on Internet-scale image datasets [49, 51] .Our goal is to expand the capabilities of pretrained image diffusion models to produce a wide range of 2D and 3D visual content, including panoramic images and textures for 3D objects, as shown in Figure 1 , without the need to train diffusion models for each specific visual content. Despite the existence of general image datasets on the scale of billions [49] , collecting other forms of visual data at this scale is not feasible. Nonetheless, most visual content can be converted into a regular image of a specific size through certain mappings, such as projecting for panoramic images and rendering for textures of 3D objects. Thus, we employ such a bridging function between each type of visual content and images, along with pretrained image diffusion models [47, 38] .We introduce a general generative framework that generates data points in the desired visual content space-referred to as canonical space-by combining the denoising process of diffusion models in the conventional image space-referred to as instance spaces. Given the bridging functions connecting the canonical space and instance spaces, we first explore performing individual denoising processes in each instance space while synchronizing them in the canonical space via the mapping. Another approach is to denoise directly in the canonical space, although it is not immediately feasible due to the absence of diffusion models trained on the canonical space. We investigate redirecting the noise prediction to the instance spaces but aggregating the outputs later in the canonical space.Depending on the timing of aggregating the outputs of computation in the instance spaces, we identify five main possible options for the diffusion synchronization processes. Previous works [5, 18, 35] have investigated each of the possible cases only for specific applications, and none of them have analyzed and compared them across a range of applications. For the first time, we present a general framework for diffusion synchronization processes, within which the previous works [5, 18, 35] are contextualized as specific cases. We then present extensive analyses of different choices of diffusion synchronization processes. Based on the analyses, we demonstrate that the approach, which conducts denoising processes in instance spaces (not the canonical space) and synchronizes the outputs of Tweedie's formula [46] in the canonical space, provides the broadest applicability across a range of applications and the best performance. We name this approach SyncTweedies and showcase its superior performance in multiple visual content creation tasks compared with previous state-of-the-art methods.Previous works [56, 34, 52, 63] finetune pretrained diffusion models to generate new types of outputs such as 360 • panorama images and 3D mesh texture images. However, this approach requires a large quantity of target content for high-quality outputs which is prohibitively expensive to acquire. When it comes to generating visual content that can be parameterized into an image, a notable zero-shot approach not utilizing diffusion synchronization is Score Distillation Sampling (SDS) [41] , which has shown particular effectiveness in 3D generation and texturing [31, 60, 62, 37] . However, this alternative application of diffusion models has been observed to produce suboptimal results and also requires a high CFG [22] weight for convergence, leading to over-saturation. For 3D mesh texture generation, specifically, an approach that iteratively updates each view image has also been explored in multiple previous works [10, 44, 8, 23, 17] . However, the accumulation of errors over iterations has been identified as a challenge. We demonstrate that our diffusion-synchronization-based approach outperforms these methods in terms of generation quality across various applications.Overall, our contributions can be summarized as follows:• We propose, for the first time, a general generative framework for diffusion synchronization processes. • Through extensive analyses of various options for diffusion synchronization processes, including previous works [35, 18, 5, 33] , we identify that the approach which synchronizes the outputs of Tweedie's formula and performs denoising in the instance space, SyncTweedies, offers the broadest applicability and superior performance. • In our experiments, we verify the superior performance and versatility of SyncTweedies across diverse applications, including texturing on 3D meshes and Gaussian Splats [26] , and depth-to-360-panorama generation. Compared to the previous state-of-the-art methods based on finetuning, optimization, and iterative updates, SyncTweedies demonstrates significantly better results. 
problem definition: 
 We consider a generative process that samples data within a space we term the canonical space Z, where a pretrained diffusion model is not provided. Instead, we leverage diffusion models trained in other spaces called the instance spaces {W i } i=1:N , where a subset of the canonical space can be instantiated into each of them via a mapping: f i : Z → W i ; we refer to this mapping as the projection. Let g i denote the unprojection, which is the inverse of f i , mapping the instance space to a subset of the canonical space. We assume that the entire canonical space Z can be expressed as a composition of multiple instance spaces W i , meaning that for any data point z ∈ Z, there existEQUATIONwhere A is an aggregation function that averages the data points from the multiple instance spaces in the canonical space. Our objective is to introduce a general framework for the generative process in the canonical space by integrating multiple denoising processes from different instance spaces through synchronization. 
diffusion synchronization: 
 We first outline the denoising procedure of DDIM [53] and then present possible options for diffusion synchronization processes based on it.Song et al. [53] have proposed DDIM, a generalized denoising process that controls the level of randomness during denoising. In DDIM [53] , the posterior of the forward process is represented as follows:q σt x (t-1) |x (t) , x (0) = N ψ (t) σt (x (t) , x (0) ),EQUATIONwhere ψ(t) σt (x (t) , x (0) ) = √ α t-1 x (0) + 1-αt-1-σ 2 t 1-αt • (x (t) - √ α t x (0)) and σ t is a hyperparameter determining the level of randomness. In this paper, we consider a deterministic process where σ t = 0 for all t, thus ψ (t) σt=0 will be denoted as ψ (t) for simplicity. During denoising process, to sample x (t-1) from its unknown original clean data point x (0) , we estimate x (0) using Tweedie's formula [46] :EQUATIONwhere ϵ θ is a noise prediction network, and for simplicity, the time input and condition term in ϵ θ are dropped. In short, each deterministic denoising step of DDIM [53] is expressed as follows:x (t-1) = ψ (t) (x (t) , ϕ (t) (x (t) , ϵ θ (x (t) ))).(4)We now explore various scenarios of sampling z ∈ Z by leveraging the composition of multiple denoising processes in the instance spaces {W i } i=1:N . Consider the denoising step of the diffusion model at each time step t in each instance space W i :EQUATIONi , ϵ θ (w(t) i ))).(5) A naïve approach to generating data in the canonical space through the denoising processes in instance spaces would be to perform the processes independently in each instance space and then aggregate the final denoised outputs in the canonical space at the end using the averaging function A. However, this approach results in poor outcomes that lack consistency across outputs in different instance spaces. Hence, we propose to synchronize the denoising processes at each time step t through the unprojection operation g i from each instance space to the canonical space and the aggregation operation A, after which the results will be back-projected via the projection operation f i to each instance space again. Note that, as described in Equation 4, the estimated mean of the posterior distribution ψ (t) (•, •) involves multiple layers of computations: noise prediction ϵ θ (•), Tweedie's formula [46] ϕ (t) (•, •) approximating the final output x (0) each time step, and the final linear combination ψ (t) (•, •). Synchronization through the sequence of unprojection g i , aggregation in the canonical space A, and projection f i can thus be performed after each layer of these computations, resulting in the following three cases: Case 1 :w (t-1) i = ψ (t) (w (t) i , ϕ (t) (w (t) i , f i (A({g j (ϵ θ (w (t) j ))} N j=1 ))))Case 2 :w (t-1) i = ψ (t) (w (t) i , f i (A({g j (ϕ (t) (w (t) j , ϵ θ (w (t) j )))} N j=1 ))Case 3 :w (t-1) i = f i (A({g j (ψ (t) (w (t) j , ϕ (t) (w (t) j , ϵ θ (w (t) j ))))} N j=1 )).In each case, we highlight the computation layer to be synchronized in red.Another notable approach is to conduct the denoising process directly on the canonical space:EQUATIONalthough it is not directly feasible because the noise prediction network in the canonical space ϵ θ (z (t) ) is not available. Nevertheless, it can be achieved by redirecting the noise prediction to the instance spaces as follows:(a) project the intermediate noisy data point z (t) from the canonical space to each instance space, resulting in f i (z (t) ), (b) apply a subsequence of the operations: ϵ θ , ϕ (t) , and ψ (t) , (c) unproject the outputs back to the canonical space via g i and then average them using the aggregation function A, and (d) perform the remaining operations in the canonical space.Such an approach of performing the denoising process in the canonical space leads to the following two additional cases depending on the subsequence of operations at step (b):Case 4 : z (t-1) = ψ (t) (z (t) , ϕ (t) (z (t) , A({g i (ϵ θ (f i (z (t) )))} N i=1 )))) Case 5 : z (t-1) = ψ (t) (z (t) , A({g i (ϕ (t) (f i (z (t) ), ϵ θ (f i (z (t) ))))} N i=1 )).Illustration of the aforementioned diffusion synchronization processes are shown in Figure 2 . Note the analogy between Cases 1 and 4, and Cases 2 and 5 in terms of the variable averaged in the canonical space with the aggregation operator A: either the outputs ofϵ θ (•) or ϕ (t) (•, •).While it is also feasible to conduct the aggregation A multiple times with the output of different layers within a single denoising step, and to denoise data both in instance spaces and the canonical space, we empirically find that such more convoluted cases perform worse. In Appendix H, we detail our exploration of all possible cases and present experimental analyses.Below, we first review previous works each corresponding to a specific case of the aforementioned possible diffusion synchronization processes while focusing on a specific application. Then, we discuss finetuning-based approaches and their limitations. In Section 4, we also review literature targeting the same applications but without diffusion synchronization.Ambiguous Image Generation. Ambiguous images are images that exhibit different appearances under certain transformations, such as a 90 • rotation or flipping. They can be generated through a diffusion synchronization process, considering both the canonical space Z and instance spaces {W i } i=1:N as the same space of the image, with the projection operation f i representing the transformation producing each appearance. Visual Anagrams [18] uses Case 4 which aggregates the noise predictions ϵ θ (•) to generate ambiguous images.Arbitrary-Sized Image Generation. In arbitrary-sized image generation, the canonical space Z is the space of the arbitrary-sized image, while the instance spaces {W i } i=1:N are overlapping patches across the arbitrary-sized image, matching the resolution of the images that the pretrained image diffusion model can generate. The projection operation f i corresponds to the cropping operation applied to each patch. MultiDiffusion [5] and SyncDiffusion [29] introduce arbitrary-sized image generation methods using Case 3, averaging the mean of the posterior distribution ψ (t) (•, •).Mesh Texturing. In 3D mesh texturing, the texture image space serves as the canonical space Z, and the rendered images from each view serve as the instance spaces {W i } i=1:N . The projection operation f i corresponds to rendering 3D textured meshes into 2D images. SyncMVD [35] proposes a 3D mesh texturing method by leveraging Case 5, which performs denoising in the canonical space and unprojects the outputs of Tweedie's formula [46] ϕ (t) (•, •).In addition to the aforementioned works, there have been attempts to achieve synchronization through finetuning. In multi-view image generation, SyncDreamer [34] and MVDream [52] finetune pretrained image diffusion models to achieve consistency across different views. MVDiffusion [56] and DiffCollage [65] generate 360 • panorama images through finetuning. Additionally, Paint3D [63] trains an encoder to directly generate 3D mesh texture images in the UV space. However, these finetuning-based methods use target sample datasets [16, 9, 15, 13] that are smaller by orders of magnitude compared to Internet-scale image datasets [49] , e.g., 10K panorama images [9] vs. 5B images [49] . As a result, they are prone to ovefitting and losing the rich prior and generalizability of pretrained image diffusion models [47, 48] . Additionally, the poor quality of textures in most 3D model datasets results in unsatisfactory texturing outcomes, even when using relatively large-scale datasets [16, 15] . In our experiments, we demonstrate that our zero-shot synchronization method, fully leveraging the pretrained model without bias toward a specific dataset, provides the best realism and widest diversity, assessed by FID and KID, compared to the finetuning-based methods. Here, we compare the five cases of diffusion synchronization processes in Section 3.2 and analyze their characteristics through various toy experiments.For the toy experiment setup, we employ the task of generating ambiguous images introduced by Geng et al. [18] (see Section 3.3.1 for descriptions of ambiguous images). In this setup, we consider two-view ambiguous image generation, where two different transformations are applied, each producing a distinct appearance. Note that one of the transformations is an identity transformation, while the other is chosen to simulate different scenarios of mapping pixels from the canonical space to the instance space: 1-to-1, 1-to-n, and n-to-1 projection. In 1-to-1 and n-to-1 projections, we use the 10 transformations from Visual Anagrams [18] , while for the 1-to-n projection, we apply rotation transformations with randomly sampled angles. For all projection cases, we use the 95 prompts from [18] . For more details on the experiment setups, refer to Appendix B.1.In 1-to-1 projection case, the five cases of diffusion synchronization processes become identical, as shown in Appendix D. The quantitative and qualitative results of diffusion synchronization processes are presented in Table 1 and the first row of Figure 3 , respectively, where the fully denoised instance variables, w1 and w (0)2 , are displayed side by side. The results confirm that all diffusion synchronization processes produce the same outputs.We further investigate the five cases of diffusion synchronization processes with different transformations for ambiguous images. It is important to note that all the transformations previously mentioned are perfectly invertible, meaning: f i (g i (w i )) = w i . However, in certain applications, the projection f i is often not a function but an 1-to-n mapping, thus not allowing its inverse. For example, consider generating a texture image of a 3D object while treating the texture image space as the canonical space and the rendered image spaces as instance spaces. When mapping each pixel of a specific view image to a pixel in the texture image in the rendering process-with nearest neighbor sampling, one pixel in the texture space can be projected to multiple pixels. Hence, the unprojection g i cannot be a perfect inverse of the projection f i but can only be an approximation, making the reprojection error ∥w i -f i (g i (w i ))∥ small. This violates the initial conditions required for the proof in Appendix D that states Cases 1-5 become identical, and we observe that such a case of having 1-to-n projection f i can significantly impact the diffusion synchronization process.As a toy experiment setup illustrating such a case with ambiguous image generation, we replace the 1-to-1 transformations used in Section 3.4.2 to rotation transformations with nearest-neighbor sampling. We randomly select an angle and rotate an inner circle of the image while leaving the rest of the region unchanged. Due to discretization, rotating an image followed by an inverse rotation may not perfectly restore the original image.The second row of Table 1 and Figure 3 present the quantitative and qualitative results of 1-ton projection experiment. Note that the performance of Case 1 and Visual Anagrams [18] (Case 4), which aggregate the predicted noises ϵ θ (•) from either instance variables w (t) i or a projected canonical variable f i (z (t) ) respectively, significantly declines. Also, the performance of Case 3, which aggregates the posterior means ψ (t) (•, •), shows a minor decline. The quality of Cases 2 and 5, however, remain almost unchanged. This highlights that the denoising process is highly sensitive to the predicted noise and to the intermediate noisy data points, while it is much more robust to the outputs of Tweedie's formula [46] ϕ (t) (•, •), the prediction of the final clean data point at an intermediate stage.Then, do the results above conclude that both Cases 2 and 5 are suitable for all applications? Lastly, we consider the case when the projection f i also involves an n-to-1 mapping. Such a scenario can arise when coloring not a solid mesh but a neural 3D representation rendered with the volume rendering equation [25, 26, 39] . Due to the nature of volume rendering, which involves sampling multiple points along a ray and taking a weighted sum of their information, the projection operation f i includes an n-to-1 mapping. Note that this case also violates the initial conditions of the proof in Appendix D, which states that the diffusion synchronization cases become identical under specific initial conditions. Additionally, Case 5 results in poor outcomes due to a variance decrease issue. Let {x i } i=1:N be random variables, each sampled from x i ∼ N (µ i , σ 2 t I), and x = N i=1 w i x i be the weighted sum, where 0 ≤ w i ≤ 1 andN i=1 w i = 1. Then, x also follows the Gaussian distribution x ∼ N N i=1 w i µ i , N i=1 w 2 i σ 2 t I .From the triangle inequality [40] , the sum of squares is always less than or equal to the square of the sum:N i=1 w 2 i ≤ ( N i=1 w i ) 2 = 1,implying that the variance of x is mostly less than the variance of x i .Consequently, when f i includes an n-to-1 mapping, the variance of w (t) i , computed as a weighted sum over multiple points in the canonical space, is mostly less than the variance of z (t) . Thus, the final output of Case 5 becomes blurry and coarse since each intermediate noisy latent in instance spaces w (t) i experiences a decrease in variance compared to that of z (t) . We validate our analysis with another toy experiment, where we use the same set of transformations used by Geng et al. [18] but with a multiplane image (MPI) [57] as the canonical space. The image of each instance space is rendered by first averaging colors in the multiplane of the canonical space and then applying the transformation. Ten planes are used for the multiplane image representation in our experiments. The results are presented in the third row of Table 1 and Figure 3 . Notably, Case 5 fails to produce plausible images like the other cases, whereas Case 2 still generates realistic images.Table 2 below summarizes suitable cases for each projection type. Note that Case 2 is the only case that is applicable to any type of projection function. Since Case 2 involves averaging the outputs of Tweedie's formula in the instance spaces, we name this case SyncTweedies. Experimental results with additional applications are demonstrated in Section 5, and analysis of all possible cases is presented in Appendix H. ✔ ✔ ✔ ✔ ✔ 1-to-n 360 • panoramas, 3D mesh texturing ✗ ✔ ✗ ✗ ✔ n-to-1 3D Gaussian Splats [26] texturing ✗ ✔ ✗ ✗ ✗ Previous Work - - MultiDiffusion [5]Visual Anagrams [18] SyncMVD [35] 4 Related WorkIn addition to Section 3.3.1 introducing previous works on diffusion synchronization, in this section, we review other previous works that utilize pretrained diffusion models in different ways to generate or edit visual content.Optimization-Based Methods. Poole et al. [41] first introduced Score Distillation Sampling (SDS), which facilitates data sampling in a canonical space by leveraging the loss function of the diffusion model training and performing gradient descent. This idea, originally introduced for 3D generation [58, 31, 55] , has been widely applied to various applications, including vector image generation [24] , ambiguous image generation [7] , mesh texturing [37, 11, 62] , mesh deformation [61] , and 4D generation [32, 3] . Subsequent works [20, 27, 28] also proposed modified loss functions not to generate data but to edit existing data while preserving their identities. This approach, exploiting diffusion models not for denoising but for gradient-descent-based updating, generally produces less realistic outcomes and is more time-consuming compared to denoising-based generation.Iterative View Updating Methods. Particularly for 3D object/scene texturing and editing, there are approaches to iteratively update each view image and subsequently refine the 3D object/scene. TEXTure [44] , Text2Tex [10] , and TexFusion [8] are previous works that sequentially update a partial texture image from each view and unproject it onto the 3D object mesh. For texturing 3D scene meshes, Text2Room [23] and SceneScape [17] take a similar approach and update scene textures sequentially. Instruct-NeRF2NeRF [19] proposed to edit a 3D scene by iteratively replacing each view image used in the reconstruction process. However, sequentially updating the canonical sample leads to error accumulations, resulting in blurriness or inconsistency across different views.Utilization of One-Step Predictions. Previous works have utilized the outputs of Tweedie's formula to restore images [12, 66] and to guide the generation process [4, 29] . However, the one-step predicted samples are used to compute the gradient from a predefined loss function to guide the sampling process, rather than for synchronization, which differentiates from our approach.Concurrent works [50, 59] also average the outputs of Tweedie's formula, similar to our approach, but they focus only on specific applications. For the first time, we present a general framework for diffusion synchronization and provide a comprehensive analysis of different synchronization methods across various applications. Case 2Case 3 Case 4Sync-MVD [35] Case 5Paint3D [63] Paint-it [62] TEXTure [44] Text2Tex [10] "Minivan" "Baseball glove" "Light bulb"Figure 4 : Qualitative results of 3D mesh texturing. SyncTweedies and SyncMVD [35] generate realistic texture images, achieving better results than other baselines including finetuning-based method. Other diffusion synchronization cases fail to produce plausible textures.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
applications: 
 We quantitatively and qualitatively compare SyncTweedies with the other diffusion synchronization processes, as well as the state-of-the-art methods of each application: 3D mesh texturing (Section 5.1), depth-to-360-panorama generation (Section 5.2) ,and 3D Gaussian splats [26] texturing (Section 5.3). Additional experiments and detailed setups are provided in Appendix, including (1) additional qualitative results, (2) implementation details of each application, (3) arbitrary-sized image generation, (4) 3D mesh texture editing and diversity comparison, (6) runtime and VRAM usage comparisons, and ( 7) user preference evaluations.Experiment Setup. In the case of instance variable denoising processes introduced in Section 3.2 (Cases 1-3), we initialize instance variables by projecting an initial canonical latent z (T ) sampled from a standard Gaussian distribution N (0, I): w(T ) i ← f i (z (T )). For n-to-1 projection cases (e.g.,3D Gaussian splats texturing), the instance variables are directly initialized from a standard Gaussian distribution which can avoid the variance decrease issue discussed in Section 3.4.4.For instance space denoising processes, the final canonical variables are obtained by synchronizing the fully denoised instance variables at the end of the diffusion synchronization processes. Refer to Section 3.3.1 for the detailed definition of the canonical space Z, the instance spaces {W i } i=1:N , the projection operation f i , and the unprojection operation g i in each application.Evaluation Setup. Across all applications, we compute FID [21] and KID [6] to assess the fidelity of the generated images and CLIP similarity [42] (CLIP-S) to evaluate text alignment. We use a depth-conditioned ControlNet [64] as the pretrained image diffusion model.In 3D mesh texturing, projection operation f i is a rendering function which outputs perspective view images from a 3D mesh with a texture image. This operation represents a 1-to-n projection due to discretization. We evaluate five diffusion synchronization cases along with Paint3D [63] , a finetuning-based method, Paint-it [62] , an optimization-based method, and TEXTure [44] and Text2Tex [10] , which are iterative-view-updating-based methods. We use 429 pairs of meshes and prompts used in TEXTure [44] and Text2Tex [10] .Results. We present quantitative and qualitative results in Table 3 and Figure 4 , respectively. The results in Table 3 align with the observations shown in the 1-to-n projection case discussed in Section 3.4.3. SyncTweedies and SyncMVD [35] outperform other baselines across all metrics, but ours demonstrates superior performance compared to SyncMVD.Notably, SyncTweedies outperforms Paint3D [63] , a finetuning-based method, indicating that finetuning with a relatively small set of synthetic 3D objects [16] is not sufficient for realistic texture generation. This is further evidenced by the cartoonish texture of the car in row 1 of Figure 4 . Optimization-based and iterative-view-updating-based methods produce unrealistic texture images, often exhibiting high saturation and visible seams, as seen in the baseball glove and light bulb in rows 2 and 3 of Figure 4 . These issues are also reflected in the relatively high FID and KID scores in Table 3 . See Appendix A for additional qualitative results.We generate 360 • panorama images from 360 • depth maps obtained from the 360MonoDepth [43] dataset. Here, f i projects a 360 • panorama to a perspective view image, which is an 1-to-n projection due to discretization. We compare SyncTweedies with previous diffusion-synchronization-based methods [5, 18, 35] and MVDiffusion [56] , which is finetuned using 3D scenes in the ScanNet [13] dataset. We generate a total of 500 360 • panorama images at 0 • elevation, with a field of view of 72 • .Results. We report quantitative results of the five diffusion synchronization processes discussed in Section 3.2 in Table 4 . Table 4 demonstrates a trend consistent with the 1-to-n projection toy experiment results shown in Section 3.4.3. Specifically, SyncTweedies and Case 5, which synchronize the outputs of Tweedie's formula ϕ (t) (•, •), exhibit the best performance. Notably, SyncTweedies demonstrates slightly superior performance across all metrics. On the other hand, MVDiffusion [56] , which is finetuned using indoor scenes, fails to adapt to new, unseen domains and shows inferior results. The qualitative results are presented in Appendix A due to page limit. Figure 5 : Qualitative results of 3D Gaussian splats [26] texturing. [S * ] is a prefix prompt. We use "Make it to" for IN2N [19] and "A photo of" for the others. SyncTweedies generates high-fidelity textures, while Case 5 lacks fine details due to the variance reduction issue.Lastly, to verify the difference between SyncTweedies and Case 5 both of which demonstrate applicability up to 1-to-n projections as outlined in Section 3.4.3, we explore texturing 3D Gaussian Splats [26] , exemplifying an n-to-1 projection case. In 3D Gaussian splats texturing, the projection operation f i is an n-to-1 case, characterized by a volumetric rendering function [25] . This function computes a weighted sum of n 3D Gaussian splats in the canonical space to render a pixel in the instance space. Note that in 3D Gaussian splats texturing, the unprojection g i and the aggregation A operation are performed using optimization.While recent 3D generative models [55, 54] generate plausible 3D objects represented as 3D Gaussian splats, they often lack fine details in the appearance. We validate the effectiveness of SyncTweedies on pretrained 3D Gaussian splats [26] from the Synthetic NeRF dataset [39] . We use 50 views for texture generation and evaluate the results from 150 unseen views. For baselines, we evaluate diffusion-synchronization-based methods, the optimization-based methods, SDS [41] , MVDream-SDS [52] , and the iterative-view-updating-based method, Instruct-NeRF2NeRF (IN2N) [19] .Results. Table 5 and Figure 5 present quantitative and qualitative comparisons of 3D Gaussian splats [26] texturing. SyncTweedies, unaffected by the variance decrease issue, outperforms Case 5 both quantitatively and qualitatively, which is consistent with the observations from the toy experiments in Section 3.4.4. When compared to other baselines based on optimization (SDS [41] and MVDream-SDS [52] ) and iterative view updating (IN2N [19] ), ours outperforms across all metrics, especially by a large margin in FID [21] . As shown in Figure 5 , optimization-based methods tend to generate textures with high saturation, while the iterative-view-updating-based method produces textures lacking fine details. Additional qualitative results are shown in Appendix A. 
conclusion: 
 We have explored various scenarios of diffusion synchronization and evaluated their performance across a range of applications, including ambiguous image generation, panorama generation, and texturing on 3D mesh and 3D Gaussian splats. Our analysis shows that SyncTweedies, which averages the outputs of Tweedie's formula while conducting denoising in multiple instance spaces, offers the best performance and the widest applicability.Limitations and Societal Impacts. Despite the superior performance of SyncTweedies across diverse applications, updating both the geometry and appearance of 3D objects remains an open problem. Also, since the pretrained image diffusion model may have been trained with uncurated images, SyncTweedies might inadvertently produce harmful content.A Qualitative Results 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide a runtime comparison of ours and other baselines in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss societal impacts in the last section of the main paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: NA.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have properly cited all sources.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: As shown in Figure 6 , SyncTweedies and SyncMVD [35] generate the most realistic output images, aligning with the results of n-to-1 projection scenarios discussed in Section 3.4.3. Notably, Paint3D [63] , a finetuing-based method, produces inferior textures, losing fine-details, as seen in the appearance of the clock in row 3 and the patterns of the ladybug in row 5. This demonstrates the challenge of acquiring a sufficient amount of high-quality texture images for satisfactory results.The optimization-based method [62] tends to produce images with high-contrast, unnatural colors, as evidenced in rows 4 and 6. Lastly, the iterative-view-updating-based methods [44, 10] show inconsistencies across views noticeable in the dumpster in row 1 and the television in row 9. [35] exhibit comparable results, outperforming other baselines. Finetuning-based method [63] produces images without fine details as it was trained on a dataset with coarse texture images. The optimization-based method [62] tends to produce unrealistic and high saturation textures, while iterative-view-updatingbased methods [10, 44] show view inconsistencies.As shown in Figure 7 , SyncTweedies and Case 5 demonstrate the best results, aligning well with the input depth maps, with SyncTweedies showing a slightly better alignment as indicated by the red arrow in Figure 7 . On the other hand, MVDiffusion [56] , which is finetuned with the depth maps of indoor 3D scenes from the ScanNet [13] dataset, produces suboptimal results and fails to generate realistic 360 • panoramas for out-of-domain scenes. This demonstrates that MVDiffusion [56] is overfitting to the scenes encountered during finetuning, resulting in a loss of generalizability. Cases 1 and 4, which aggregate the predicted noise ϵ θ (•), produce noisy outputs. Case 3 yields suboptimal panoramas, characterized by monochromatic appearances and a lack of detail.Outdoor Scenes "a house at night" "a church in a large building" InputFigure 8 shows that SyncTweedies generates high-fidelity results with intricate details, such as the carvings of an excavator in row 2, while Case 5 lacks fine details. Optimization-based methods, SDS [41] and MVDream-SDS [52] , produce artifacts characterized by high saturation, such as the corns in row 1 and the carrots in row 4. Notably, a finetuning-based method, MVDream-SDS [52] , shows inferior quality to SDS. As discussed in Section 3.3.2, the poor quality of textures in the finetuning dataset [16] results in quality degradation. Iterative-view-updating-based method, IN2N [19] , fails to preserve fine details, such as the head of the microphone in row 7. ] is a prefix prompt. We use "Make it to" for IN2N [19] and "A photo of" for the other methods. Case 5 tends to lose details due to the variance decrease issue, whereas SyncTweedies generates realistic images by avoiding this issue. The optimization-based methods [41, 52] produce high contrast, unnatural colors, and the iterative view updating method [19] yields suboptimal outputs due to error accumulation.In this section, we provide details on the experiments discussed in Section 5 of the main paper. For all diffusion synchronization processes, we use a fully deterministic DDIM [53] sampling with 30 steps, unless specified otherwise.We use DeepFloyd [14] as the pretrained diffusion model for the ambiguous image generation which denoises images in the pixel space. For the depth-to-360-panorama generation, 3D mesh texturing, and 3D Gaussian splats texturing, we employ a pretrained depth-conditioned ControlNet [64] which is based on a latent diffusion model, specifically Stable Diffusion [47] . For applications utilizing ControlNet, synchronization during the intermediate steps of diffusion synchronization processes occurs within the same latent space, except for 3D Gaussian splats texturing. In the case of 3D Gaussian splats texturing, synchronization takes place in the RGB space, and detailed explanations are provided in Section B.4.In the 1-to-n projection cases, each instance space sample is unprojected into the canonical space, resulting in N unprojected samples, {g i (w(t) i )} N i=1, where N is the number of views. The canonical space sample z (t) is then obtained by averaging these unprojected samples. The averaging can be weighted based on the visibility from each view. An illustration of the process is shown in Figure 9 . Evaluation Metrics. For all applications, we evaluate diversity and fidelity of the generated images using FID [21] and KID [6] . These metrics compute scores based on the distance between the distribution of the generated image set and that of the reference image set, with the reference set forming the target distribution. Refer to each application section for detailed description of constructing the generated image set and the reference image set.To evaluate the text alignment of the generated images, we report CLIP similarity score [42] (CLIP-S) which measures the similarity between the generated images w (0) i and their corresponding text prompts p i in CLIP [42] embedding space. Additionally, in the ambiguous image generation, we report CLIP alignment score (CLIP-A) and CLIP concealment score (CLIP-C) following previous work, Visual Anagrams [18] . To compute the metrics, we begin by calculating a CLIP similarity matrix S ∈ R N ×N from N pairs of transformations and text prompts:EQUATIONwhere E img (•) and E text (•) are the image encoder and the text encoder of the pretrained CLIP model [42] , respectively. CLIP-A quantifies the worst alignment among the corresponding image-text pairs, specifically computed as min diag(S). However, this metric does not account for misalignment failure cases, where p i is visualized in w (0) j for i ̸ = j. CLIP-C considers alignment of an (a) image (prompt) to all prompts (images) by normalizing the similarity matrix S using softmax:EQUATIONwhere tr(•) denotes the trace of a matrix, and τ is the temperature parameter of CLIP [42] . We set τ to 0.07.B.1 Details on Ambiguous Image Generation -Section 3.4.1We present the details of the ambiguous image generation experiments in Section 3.4.1. Quantitative and qualitative results are presented in Table 1 and Figure 3 .Evaluation Setup. To evaluate the fidelity of the generated images using FID [21] and KID [6] , we create a reference set consisting of 5,000 generated images from Stable Diffusion 1.5 [47] with the same text prompts used in the generation of ambiguous images.Implementation Details. We use DeepFloyd [14] which is a two-stage cascaded pixel-space diffusion model. In the first stage, we generate 64 × 64 images that are upscaled to 256 × 256 images in the subsequent stage.Definition of Operations. In the context of ambiguous image generation, both the instance variables {w i } i=1:N and canonical variables z share the same image space. However, instance variables exhibit different appearances from the canonical variable upon applying certain transformations.In the 1-to-1 projection case, we use the 10 transformations used in Visual Anagrams [18] , all of which are 1-to-1 mappings. The projection operation f i is defined as the transformation itself, and the unprojection operation g i is defined as the inverse of the transformation matrix.In the scenario of 1-to-n projection, we employ inner circle rotation as the projection operation f i . This involves rotating the pixels within an inner circle of an image while keeping the outer pixels unchanged. The unprojection operation g i is the inverse of f i . We use 14 inner circle rotation transformations, with rotation angles evenly spaced in the range [45• , 175 • ].For evaluation, we utilize the same 95 prompts as in the 1-to-1 case for each transformation, generating 14 × 95 = 1, 350 ambiguous images. After applying a rotation transformation, the grid of the rotated image does not align with the original image grid. Thus, we use the nearest-neighbor sampling to retrieve pixel colors from the original image to the rotated image. This sampling process leads to a scenario where a single pixel in the original image z can be mapped to multiple pixels in the rotated image w i , which is an 1-to-n mapping.For n-to-1 projection, we use the same transformations and text prompts as in the 1-to-1 projection experiment, thus resulting in a total of 10 × 95 = 950 ambiguous images. The only difference from the 1-to-1 projection experiment is that the canonical space variable z is now represented as multiplane images (MPI) [57] , where a collection of planes {p j } j=1:M represents a single canonical variable. Specifically, we compute z by averaging the multiplane images: z = 1 M M j=1 p j . In the context of n-to-1 projection, we substitute the sequence of the unprojection g i and the aggregation A operation with an optimization process. The multiplane images {p j } are optimized using the following objective function:EQUATIONwhere wet set the number of planes M = 10.We provide details of the 3D mesh texturing experiments presented in Section 5.1. Quantitative and qualitative results are shown in Table 3 and Figure 6 .Evaluation Setup. We use 429 mesh and prompt pairs collected from previous works, TEX-Ture [44] and Text2Tex [10] . For texture generation, we use eight views sampled around the object with 45 • intervals at 0 • elevation. Two additional views are sampled at 0 • and 180 • azimuths with 30 • elevation. For evaluation, we render each 3D mesh to ten perspective views with randomly sampled azimuths at 0 • elevation, resulting 10 × 429 = 4,290 images. Following SyncMVD [35] , the reference set images are generated by ControlNet [64] using the same depth maps and text prompts used in the texture generation.Implementation Details. The resolution of the latent texture image is 1, 536 × 1, 536, and that of the latent perspective view images is 96 × 96. In the RGB space, the resolution of the texture image is 1, 024 × 1, 024 and that of the perspective view images is 768 × 768.We adopt two approaches introduced in SyncMVD [35] : Voronoi-diagram-based filling [2] and modified self-attention layers. First, the high resolution of the latent texture image results in a texture image with sparse pixel distribution. To address this issue, we propagate the unprojected pixels to the visible regions of the texture image using the Voronoi-diagram-based filling. Second, spatially distant views tend to generate inconsistent outputs. Therefore, we adopt the modified self-attention mechanism that attends to other views when computing the attention output.Definition of Operations. In the 3D mesh texturing, the canonical variable z is the texture image of a 3D mesh, and the instance variables {w i } i=1:N are rendered images from the 3D mesh. The projection operation f i is a rendering function where nearest-neighbor sampling is utilized to retrieve the color from the texture image to perspective view images.The unprojection operation g i is performed using optimization where the texture image z is updated to minimize the rendering loss with the multi-view images {w i } i=1:N . The projection operation of 3D mesh texturing may involve mapping one pixel in the texture image z to multiple pixels in a rendered image w i . Hence, this application corresponds to the 1-to-n projection case as in Section 3.4.3.We provide details of the depth-to-360-panorama generation experiments presented in Section 5.2. Refer to Table 4 and Figure 7 for quantitative and qualitative results.Evaluation Setup. We evaluate SyncTweedies and the baselines on 500 pairs of 360 • panorama images and depth maps randomly sampled from the 360MonoDepth [43] dataset. For each 360 • panorama image, we generate a text prompt using the output of BLIP [30] by providing a perspective view image of the panorama as input.In the 360 • panorama generation, we use eight perspective views by evenly sampling azimuths with 45 • intervals at 0 • elevation. Each perspective view has a field of view of 72 • for diffusionsynchronization-based methods and 90 • for MVDiffusion [56] . For evaluation, we project the generated 360 • panorama image to ten perspective views with randomly sampled azimuths at 0 • elevation and a field of view of 60 • . Similarly, the reference set images are obtained by projecting each ground truth 360 • panorama image into ten perspective views with azimuths randomly sampled and at 0 • elevation. In total, we use 500 × 10 = 5, 000 perspective view images for evaluation.Implementation Details. We set the resolution of a latent panorama image to 2,048 × 4,096 and that of the latent perspective view images to 64 × 64. In the RGB space, a panorama image has a resolution of 1,024 × 2,048, and perspective view images have a resolution of 512 × 512. As done in the 3D mesh texturing, we apply the Voronoi-diagram-based filling [2] after each unprojection operation and employ the modified self-attention mechanism.Definition of Operations. In the 360 • panorama generation, the canonical variable z represents a 360 • panorama image, while the instance variables {w i } i=1:N correspond to perspective views of the panorama. The mappings between the panorama image and the perspective views are computed as follows: First, we unproject the pixels of the perspective view image to the 3D space. Then, we apply two rotation matrices based on the azimuth and elevation angles. The pixels are then reprojected onto the surface of a unit sphere, represented as longitudes and latitudes. These spherical coordinates are finally converted to 2D coordinates on the panorama image.Given the mappings, the projection operation f i samples colors from the panorama image using the nearest-neighbor method. Since a single pixel of a panorama image z can be mapped to multiple pixels of a perspective view image w i , the 360 • panorama generation is a 1-to-n projection case, as discussed in Section 3.4.3.We provide details of the 3D Gaussian splats texturing experiment presented in Section 5.3. Quantitative and qualitative results are provided in Table 5 and Figure 8 .Evaluation Setup. For evaluation, we use 3D Gaussian splats trained with multi-view images from the Synthetic NeRF dataset [39] , consisting of 8 objects. We generate 40 textured 3D Gaussian splats by utilizing five different prompts per scene. We use 50 views for texture generation and 150 unseen views for evaluation.Implementation Details. As described in Section B, we employ ControlNet [64] which denoises latent images. To render the latent images, we replace the spherical harmonics coefficients of a 3D Gaussian splats to a 4-channel latent vector. For the optimization, we run 2,000 iterations with a learning rate of 0.025. When applicable, we perform the optimization in RGB space by decoding the latent variables for diffusion-synchronization-based methods.Definition of Operations. The canonical variables {z j } j=1:M are 3D Gaussian splats and the instance space variables {w i } i=1:N are the rendered images from the 3D Gaussian splats. The projection operation f i is a volume rendering function [25, 26] where the colors (latent vectors) of multiple 3D Gaussian splats are composited to render a pixel. This corresponds to the n-to-1 projection as discussed in Section 3.4.4. In 3D Gaussian splats texturing, only the colors of 3D Gaussian splats z = {s j } j=1:M are optimized from multi-view images {w i } i=1:N , while keeping other parameters, such as positions, fixed, as done in the n-to-1 experiment in Section 3.4.4. In addition to the 1-to-1 projection case presented in Section 3.4.2, we present arbitrary-sized image generation. In contrast to depth-to-360-panorama generation, which corresponds to the 1-to-n projection case, arbitrary-sized image generation is a 1-to-1 projection case.Evaluation Setup. We follow the evaluation setup used in SyncDiffusion [29] . Using Stable Diffusion 2.0 [47] as the pretrained diffusion model, we generate 500 arbitrary-sized images of 512 × 3, 072 resolution per prompt. With six text prompts from SyncDiffusion [29] , we generate a total of 500 × 6 = 3, 000 arbitrary-sized images. For quantitative evaluation, we report FID [21] , KID [6] , and CLIP-S [42] . Each generated arbitrary-sized image is randomly cropped to partial view images with 512 × 512 resolution. For the reference set, we generate 3, 000 images with a resolution of 512 × 512 from the pretrained diffusion model using the same text prompts.Implementation Details. The resolution of latent arbitrary-sized image is 64 × 384, and the resolution of an instance space sample is 64 × 64. We use deterministic DDIM [53] sampling with 50 steps.Definition of Operations. The projection operation f i corresponds to cropping a partial view of the arbitrary-sized image, which is a 1-to-1 projection. The unprojection operation g i is the inverse of the f i which pastes the partial view image onto the canvas of the arbitrary-sized image.Results. We report quantitative results in Table 6 and qualitative results in Figure 10 . As mathematically proven in Section D, the quantitative results show that all diffusion synchronization cases exhibit comparable performances, which aligns with the observations from the 1-to-1 experiment in Section 3.4.2. This is further supported by the qualitative results in Figure 10 , where all cases produce identical arbitrary-sized images, indicating that any option can be used when the projection is 1-to-1.Results using Gaudi Intel-v2. Additionally, we present qualitative results of arbitrary-sized image generation using Intel Gaudi-v2 in Figure 11 , along with a comparison of computation times between Intel Gaudi-v2 and NVIDIA A6000 in Figure 12 . We observe that Intel Gaudi-v2 achieves 1.8 to 1.9 times faster runtimes compared to the NVIDIA A6000. It is mathematically guaranteed that Cases 1-5 become identical when the mappings are 1-to-1 and noises are initialized by projecting from the canonical space w(T ) i = f i (z (T )), where z (T ) ∼ N (0, I). Note that ϕ (t) (•, •) and ψ (t) (•, •) are linear operations and commutative with other linear operation such as f i , A, and g i . Assume the following conditions hold:EQUATIONEQUATIONBased on induction, we have:EQUATIONwhere the last equality holds the induction hypothesis. This proves that Cases 4-5 are identical.For instance variable denoising cases we have:EQUATION= f i (A({g j (ψ (t) (wEQUATION= f i (A({g j (wEQUATIONwhere the last equality holds the induction hypothesis. This proves that Cases 1-3 are identical. Lastly, based on the definition of the projection operation, we have:EQUATION)This proves that canonical variable denoising cases (Cases 4-5) are equivalent to Case 3.We validate the proof both qualitatively and quantitatively in applications with 1-to-1 projection: ambiguous image generation and arbitrary-sized image generation in Section 3.4.2 and Section C, respectively, where all cases generate identical results.Input 3D mesh [1] Edited 3D mesh (SyncTweedies)"A nascar" "A car with graffiti" "A lantern" "A Chinese style lantern" "A turtle" "A golden statue of a turtle" In this section, we extend the 3D mesh texture generation from Section 5.1 and present a texture editing application, along with a diversity comparison of SyncTweedies to the optimization-based method Paint-it [62] .Despite the recent successes of 3D generation models [1, 34] , the textures of the generated 3D meshes often lack fine details. We utilize SyncTweedies to edit the textures of the generated 3D meshes, and enhance the texture quality. Specifically, we use the 3D meshes generated from a text-to-3D model, Genie [1] .We follow SDEdit [36] to edit the textures of 3D meshes. We begin by adding noise at an intermediate time t ′ to the texture image of the 3D mesh and then perform a reverse process starting from t ′ .Implementation Details. We set the CFG weight [22] to 30 and t ′ to 0.8. For other settings, we follow the 3D mesh texture generation experiment presented in Section 5.1.Results. We present qualitative results of 3D mesh texture editing in Figure 13 . The 3D meshes edited with SyncTweedies exhibit fine details, including graffiti on the car in row 1, paintings on the lantern in row 2, and the intricate shells of the turtle in row 3.In Figure 14 , we present qualitative results of 3D mesh texturing using the optimization-based method (Paint-it [62] ) and SyncTweedies with different random seeds. SyncTweedies generates more diverse texture images compared to Paint-it. As discussed in Section 4, one of the advantages of diffusion synchronization processes is the fast computational speed. We compare the runtime performance of SyncTweedies with optimizationbased and iterative-view-updating-based methods in the 3D mesh texturing and the 3D Gaussian splats texturing. The quantitative results are presented in Table 7 .In the 3D mesh texturing application, SyncTweedies shows faster running time than other baselines except TEXTure [44] which shows comparable running time. However, TEXTure [44] generates suboptimal texture outputs as observed in Table 3 and Figure 6 . The finetuning-based method Paint3D [63] shows a comparable running time to SyncTweedies, but it shows inferior quality, as seen in Table 3 and Figure 6 . Another iterative-view-updating-based method, Text2Tex [10] , improves quality of texture image by integrating a refinement module, but this comes at the cost of additional overhead in terms of running time. In contrast, SyncTweedies achieves running times that are 7 times faster than Text2Tex and even outperforms across all metrics as shown in Table 3 . Lastly, SyncTweedies shows 11 times faster running time when compared to Paint-it [62] , an optimization-based method.In the 3D Gaussian splats texturing, SyncTweedies achieves the fastest running time.SyncTweedies is 3 times faster than the iterative-view-updating-based method IN2N [19] , and 8 times faster than the optimization-based method, SDS [41] . This shows that SyncTweedies not only generates high-fidelity textures, but also excels other baselines in computational speed. We use the NVIDIA RTX A6000 for the runtime comparisons.Additionally, in Table 8 , we present results comparing the VRAM usage of SyncTweedies and the baselines, where our SyncTweedies requires around 6-9 GiB of memory, making it suitable for most GPUs.We conduct user studies to evaluate the textures of the generated 3D Gaussian splats [26] through Amazon's Mechanical Turk. Following the methodology of Ritchie [45] , participants were presented with input text prompts and randomly sampled output images generated by our method and the baseline methods. Participants are asked to choose the most plausible image that aligns with the given text prompt. In Table 9 , our results are the most preferred in the human evaluations compared to the other baselines.Details on User Study. We conduct separate user studies comparing our method to diffusionsynchronization-based methods, optimization-based methods (SDS [41] , MVDream-SDS [52] ), and iterative-view-updating-based method (IN2N [19] ). For each user study, we use 20 images in a shuffled order including five vigilance tasks. We collect survey responses only from participants who pass the vigilance tasks. Specifically, 94 out of 100 participants passed in the test with Case 5, 90 out of 100 passed with SDS [41] , 95 out of 100 passed with MVDream-SDS [52] , and 92 out of 100 passed with IN2N [19] . Screenshots of the user study, including an example of vigilance tasks, are shown in Figure 15 . As outlined in Section 3.4.4, we present a comprehensive analysis of all possible diffusion synchronization processes, including the representative five diffusion synchronization processes introduced in Section 3.2. Following the main paper, we categorize diffusion synchronization processes into two types: the instance variable denoising process, where instance variables {wi } are denoised, and the canonical variable denoising process, which denoises a canonical variable z (t) directly. Unlike the representative cases, other all feasible cases either take inconsistent inputs when computing ϵ θ (•), ϕ (t) (•, •) and ψ (t) (•, •) or conduct the aggregation A multiple times. Additionally, for a more exhaustive analysis, we introduce another type of diffusion synchronization processes, named the combined variable denoising process, which denoises {w (t) i } and z (t) together. We present a total of 46 feasible cases for the instance variable denoising process, 8 for the canonical variable denoising process, and an additional 6 representative cases for the combined variable denoising process. We provide instance variable denoising cases in Section H.2, and canonical variable denoising cases in Section H.3. Additionally, the six representative cases for the combined variable denoising process are detailed in Section H.4. We conduct a quantitative comparison of all listed cases following the experiment setup outlined in Section 3.4.1, and the results are presented in Section H.5.We provide the representative trajectories in Figure 16, where (a)-(b), (c)-(d), (e)-(f), and (g)-(h ) follow the same trajectory but differ in the denoising variable, either instance or canonical, respectively. In each denoising case, there are 2 2 = 4 possible trajectories determined by whether ϕ (t) (•, •) and ψ (t) (•, •) are computed in the canonical space or instance space. This is because among the three computation layers-ϵ θ (•), ϕ (t) (•, •) and ψ (t) (•, •)-only the last two operations can be computed in both the canonical space and the instance space unlike noise prediction which is only available in the instance space. Table 10 summarizes the computation spaces of ϕ (t) (•, •) and ψ (t) (•, •), along with their corresponding trajectories.Table 10 : Computation space of each denoising trajectory. ϕ (t) (•, •) and ψ (t) (•, •) can be computed in both instance space W i or canonical space Z, whereas noise prediction ϵ θ (•) can only be computed in the instance space.Trajectoryϕ (t) (•, •) Computation space ψ (t) (•, •) Computation space Trajectory 1 Wi Wi Trajectory 2 Z Wi Trajectory 3 Z Z Trajectory 4Wi ZNext, we introduce an additional operator F i that synchronizes instance variables. This operator unprojects a set of instance variables and averages them in the canonical space. Subsequently, the aggregated variables are reprojected to the instance space:EQUATIONThe red arrows in the diagrams of Figure 16 indicate the potential incorporation of F i . Thus, a total of 2 N different cases can be derived from a trajectory marked by N red arrows, depending on whether F i is applied to each variable or not.Lastly, we review the five representative diffusion synchronization processes discussed in Section 3.2, along with two additional denoising processes: an instance variable denoising process that proceeds without synchronization (No Synchronization) and a canonical variable denoising process that averages the outputs of ψ (t) (•, •) (Case 6):No Synchronization :w (t-1) i = ψ (t) (w (t) i , ϕ (t) (w (t) i , ϵ θ (w (t) i ))) Case 1 : w (t-1) i = ψ (t) (w (t) i , ϕ (t) (w (t) i , F i (ϵ θ (w (t) i )))) Case 2 : w (t-1) i = ψ (t) (w (t) i , F i (ϕ (t) (w (t) i , ϵ θ (w (t) i )))) Case 3 : w (t-1) i = F i (ψ (t) (w (t) i , ϕ (t) (w (t) i , ϵ θ (w (t) i )))) Case 4 : z (t-1) = ψ (t) (z (t) , ϕ (t) (z (t) , A({g i (ϵ θ (f i (z (t) )))}))) Case 5 : z (t-1) = ψ (t) (z (t) , A({g i (ϕ (t) (f i (z (t) ), ϵ θ (f i (z (t) ))))})) Case 6 : z (t-1) = A({g i (ψ (t) (f i (z (t) ), ϕ (t) (f i (z (t) ), ϵ θ (f i (z (t) )))))}).Note that Case 3 and 6 are identical except for the initialization, which can be either {w (T ) i } or z (T ) . For the independent instance variable denoising process (No Synchronization), synchronization is only applied at the end of the denoising process.Here, we explore all possible instance variable denoising processes. Here, the canonical space Z is employed to synchronize the outputs of ϵ θ (•), ϕ (t) (•, •) and ψ (t) (•, •) in the instance spaces.Following the trajectory 1 shown in part (a) of Figure 16 , marked by five red arrows, there are a total of 2 5 = 32 possible denoising processes. This includes the independent instance variable denoising process (No Synchronization), where F i is not applied at any red arrow. Additionally, the three representative instance variable denoising processes, Cases 1-3, are also included, along with Cases 7-34 which are presented below: Case 7 :w (t-1) i = ψ (t) (w (t) i , ϕ (t) (w (t) i , ϵ θ (F i (w (t) i )))) Case 8 : w (t-1) i = ψ (t) (w (t) i , ϕ (t) (w (t) i , F i (ϵ θ (F i (w (t) i ))))) Case 9 : w (t-1) i = ψ (t) (w (t) i , ϕ (t) (F i (w (t) i ), ϵ θ (w (t) i ))) Case 10 : w (t-1) i = ψ (t) (w (t) i , ϕ (t) (F i (w (t) i ), ϵ θ (F i (w (t) i )))) Case 11 : w (t-1) i = ψ (t) (w (t) i , ϕ (t) (F i (w (t) i ), F i (ϵ θ (w (t) i )))) Case 12 : w (t-1) i = ψ (t) (w (t) i , ϕ (t) (F i (w (t) i ), F i (ϵ θ (F i (w (t) i ))))) Case 13 : w (t-1) i = ψ (t) (w (t) i , F i (ϕ (t) (w (t) i , ϵ θ (F i (w (t) i ))))) Case 14 : w (t-1) i = ψ (t) (w (t) i , F i (ϕ (t) (w (t) i , F i (ϵ θ (w (t) i ))))) Case 15 : w (t-1) i = ψ (t) (w (t) i , F i (ϕ (t) (w (t) i , F i (ϵ θ (F i (w (t) i )))))) Case 16 : w (t-1) i = ψ (t) (w (t) i , F i (ϕ (t) (F i (w (t) i ), ϵ θ (w (t) i )))) Case 17 : w (t-1) i = ψ (t) (w (t) i , F i (ϕ (t) (F i (w (t) i ), ϵ θ (F i (w (t) i ))))) Case 18 : w (t-1) i = ψ (t) (w (t) i , F i (ϕ (t) (F i (w (t) i ), F i (ϵ θ (w (t) i ))))) Case 19 : w (t-1) i = ψ (t) (w (t) i , F i (ϕ (t) (F i (w (t) i ), F i (ϵ θ (F i (w (t) i )))))) Case 20 : w (t-1) i = ψ (t) (F i (w (t) i ), ϕ (t) (w (t) i , ϵ θ (w (t) i ))) Case 21 : w (t-1) i = ψ (t) (F i (w (t) i ), ϕ (t) (w (t) i , ϵ θ (F i (w (t) i )))) Case 22 : w (t-1) i = ψ (t) (F i (w (t) i ), ϕ (t) (w (t) i , F i (ϵ θ (w (t) i )))) Case 23 : w (t-1) i = ψ (t) (F i (w (t) i ), ϕ (t) (w (t) i , F i (ϵ θ (F i (w (t) i ))))) Case 24 : w (t-1) i = ψ (t) (F i (w (t) i ), ϕ (t) (F i (w (t) i ), ϵ θ (w (t) i ))) Case 25 : w (t-1) i = ψ (t) (F i (w (t) i ), ϕ (t) (F i (w (t) i ), F i (ϵ θ (w (t) i )))) Case 26 : w (t-1) i = F i (ψ (t) (w (t) i , ϕ (t) (w (t) i , F i (ϵ θ (w (t) i ))))) Case 27 : w (t-1) i = ψ (t) (F i (w (t) i ), F i (ϕ (t) (w (t) i , ϵ θ (w (t) i )))) Case 28 : w (t-1) i = ψ (t) (F i (w (t) i ), F i (ϕ (t) (w (t) i , ϵ θ (F i (w (t) i ))))) Case 29 : w (t-1) i = ψ (t) (F i (w (t) i ), F i (ϕ (t) (w (t) i , F i (ϵ θ (w (t) i ))))) Case 30 : w (t-1) i = ψ (t) (F i (w (t) i ), F i (ϕ (t) (w (t) i , F i (ϵ θ (F i (w (t) i )))))) Case 31 : w (t-1) i = ψ (t) (F i (w (t) i ), F i (ϕ (t) (F i (w (t) i ), ϵ θ (w (t) i )))) Case 32 : w (t-1) i = F i (ψ (t) (w (t) i , F i (ϕ (t) (w (t) i , ϵ θ (w (t) i ))))) Case 33 : w (t-1) i = ψ (t) (F i (w (t) i ), F i (ϕ (t) (F i (w (t) i ), F i (ϵ θ (w (t) i ))))) Case 34 : w (t-1) i = F i (ψ (t) (w (t) i , F i (ϕ (t) (w (t) i , F i (ϵ θ (w (t) i )))))).Similarly, four cases are derived from the trajectory 2 shown in part (c) of Figure 16 . These correspond to Cases 35-38 below: Case 35 :w (t-1) i = ψ (t) (w (t) i , f i (ϕ (t) (A({g j (w (t) j )}), A({g j (ϵ θ (w (t) j ))}))))Case 36 :w (t-1) i = ψ (t) (w (t) i , f i (ϕ (t) (A({g j (w (t) j )}), A({g j (ϵ θ (F i (w (t) j ))})))))Case 37 :w (t-1) i = ψ (t) (F i (w (t) i ), f i (ϕ (t) (A({g j (w (t) j )}), A({g j (ϵ θ (w (t) j ))}))))Case 38 :w (t-1) i = ψ (t) (F i (w (t) i ), f i (ϕ (t) (A({g j (w (t) j )}), A({g j (ϵ θ (F i (w (t) j ))}))))).The trajectory 3 shown in part (e) of Figure 16 accounts for two cases, corresponding to Cases 39-40 below:Case 39 :w (t-1) i = f i (ψ (t) (A({g j (w (t) j )}), ϕ (t) (A({g j (w (t) j )}), A({g j (ϵ θ (w (t) j ))}))))Case 40 :w (t-1) i = f i (ψ (t) (A({g j (w (t) j )}), ϕ (t) (A({g j (w (t) j )}), A({g j (ϵ θ (F i (w (t) j )))})))).Lastly, the trajectory 4 shown in part (g) of Figure 16 includes Cases 41-48 below: Case 41 :w (t-1) i = f i (ψ (t) (A({g j (w (t) j )}), A({g j (ϕ (t) (w (t) j , ϵ θ (w (t) j )))})))Case 42 :w (t-1) i = f i (ψ (t) (A({g j (w (t) j )}), A({g j (ϕ (t) (w (t) j , ϵ θ (F i (w (t) j ))))})))Case 43 :w (t-1) i = f i (ψ (t) (A({g j (w (t) j )}), A({g j (ϕ (t) (w (t) j , F i (ϵ θ (w (t) j ))))})))Case 44 :w (t-1) i = f i (ψ (t) (A({g j (w (t) j )}), A({g j (ϕ (t) (w (t) j , F i (ϵ θ (F i (w (t) j )))))})))Case 45 :w (t-1) i = f i (ψ (t) (A({g j (w (t) j )}), A({g j (ϕ (t) (F i (w (t) j ), ϵ θ (w (t) j )))}))) Case 46 : w (t-1) i = f i (ψ (t) (A({g j (w (t) j )}), A({g j (ϕ (t) (F i (w (t) j ), ϵ θ (F i (w (t) j ))))})))Case 47 :w (t-1) i = f i (ψ (t) (A({g j (w (t) j )}), A({g j (ϕ (t) (F i (w (t) j ), F i (ϵ θ (w (t) j ))))}))) Case 48 : w (t-1) i = f i (ψ (t) (A({g j (w (t) j )}), A({g j (ϕ (t) (F i (w (t) j ), F i (ϵ θ (F i (w (t) j )))))}))).Here, we present all possible canonical variable denoising processes. Due to the absence of noise prediction in the canonical space, a process first redirects canonical variable z (t) to the instance spaces where a subsequence of operations ϵ θ (•), ϕ (t) (•, •) and ψ (t) (•, •) are computed.We exclude the application of F i to w (t) i ← f i (z (t) ), as the variable remains unchanged after the operation. Therefore, applying F i to w (t) i ← f i (z (t) ) for the inputs of ϵ θ (•), ϕ (t) (•, •) and ψ (t) (•, •) is not considered.(t-1) = ψ (t) (z (t) , A({g i (ϕ (t) (f i (z (t) ), F i (ϵ θ (f i (z (t) ))))})))In the trajectory 1, 2 2 = 4 cases are possible, as shown in part (b) of Figure 16 . This includes Case 6 along with Cases 50-52 below: performances. Except for Cases 55-56, the combined variable denoising processes (Cases 54-59) show suboptimal performances with FID [21] scores over 100. This indicates that denoising either instance variables or a canonical variable is sufficient to produce satisfactory and consistent results.When it comes to the 1-to-n projection scenario, Cases 2 and 5 outperform the others, with some exceptions such as Cases 11 and 35. This trend is also consistent with the results in Section 3.4.3, highlighting the effectiveness of synchronizing the outputs of Tweedie's formula [46] ϕ (t) (•, •) even when compared to more complex diffusion synchronization processes.Lastly, in the n-to-1 projection scenario, Case 2 (SyncTweedies) is the only one that outperforms the others across all metrics.In conclusion, as shown in Table 11 , Case 2 (SyncTweedies) distinctly exhibits superior performance across various projection scenarios, outperforming even more convoluted cases. Justification: In the last section of the main paper, we address the limitation of our work. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: All reviewers recognized the quality of the work : 
- "The identification of three different tasks [...] enriches the analysis and allows for a clear identification of the strengths and weaknesses of each aggregation strategy."
- "the framework's versatility is impressive"
- "This methodological innovation could inspire new directions in the field of generative models."
- "extensive experiments"
- "The paper is enjoyable to read"
- "The toy experiment is well-designed."
- "The experiments cover a range of tasks..."

Several concerns have been raised by the reviewers but the discussion period has resolved those and all reviewers have positively judged the authors response.

Reviews: review: paper_summary: paper_summary: This paper investigates content generation within a target space using pretrained diffusion models operating in projected subspaces. The authors analyze five variants of the DDIM procedure, performed separately in each subspace and aggregated in the target space using known projection and unprojection operators. They explore different sequences of projection and aggregation, concluding that the optimal approach is to aggregate each subspace's "estimated x0 from xt" (the output of Tweedie's formula).

The method is evaluated through texturing tasks involving depth, mesh, and 3DGS, each representing distinct projection and unprojection scenarios. The ablation study and comparisons against baselines demonstrate the robustness of the proposed aggregation stage.
strength_weakness: strength_weakness: - Missing Explanations: The figures are difficult to understand due to missing information. For example, Figure 3 does not explicitly state the experimental setup, including the definitions of the canonical space and the subspaces.
- The formulation of cases 1-3 that operate in the subspace does not explain how the recovered signal in z is finally returned. This is related to my previous comment, where it seems Figure 3 shows individual subspace generations rather than the end goal/task.
- The initialization process is not discussed in the main paper. It is unclear whether the subspaces share the same initial noise. If they do, how is noise being generated in z? This is particularly challenging when z is 3DGS.
- In the appendix, it states that the 3DGS are optimized towards the predicted clean images. Is this optimization considered the unprojection operation "g"? If so, this should be explicitly stated and moved to the main paper. More details on this optimization process are needed, including whether it is performed until convergence at each denoising time during generation.
- In lines 129-130, phi and psi are used in the target domain. Please explain how this is done properly, given that noise is not injected in that space. Why would the coefficients in eq (3) and (4) hold true?

- The caption of Figure 1, “Diverse visual content generated by SyncTweedies,” and the abstract phrase "generating diverse visual content" are misleading. They suggest the generation of general 3D content, while the method, as mentioned in the limitations, relies on known mappings between the subspaces and the target space, limiting it to texturing tasks. The authors should tone down these statements.
- If the mapping from the canonical space to the instance spaces is simple (say, linear or even the Idnetity mapping), it seems all "cases" perform similarly both quantitatively and qualitatively. Do these methods mathematically converge to the same method, or are there still differences? In the 1-1 projection example in Figure 3, it appears all methods produce the same result. Is this correct? Table 10A shows almost identical results for different methods, and in Table 1, the scores for the 1-1 cases are very similar.
- The claim of being the first to propose this method and that it was previously "overlooked" is too bold. Guidance literature shows that guidance in "\hat{x_0}" offers more stable optimization and results. While guidance differs from multi-diffusion, it is similar enough to make the analogy. The authors should discuss similarities and differences, referencing papers like "Universal Guidance for Diffusion Models" and motion diffusion papers such as MDM and Trace&Pace. Further, if in degenerate cases of f and g maps (e.g. if they are the identity function) some "cases" are indistinguishable, be mindful when attributing aggregation choices to these methods (e.g., MultiDiffusion). Methods requiring tuning, like DiffCollage, should not be dismissed as baselines. The main contribution of this paper is identifying the best "place" to aggregate, and tuning is somewhat orthogonal. If "Case 2" improves DiffCollage, this should be demonstrated.
- The term "1:1 case" is confusing and suggests disjoint individual subspaces.
- The integration process of 1:n should also be better explained. It may also be useful to add illustration figures. 
- Improve Figure 2 for clarity with a more detailed caption. Connecting the diagram to the notations introduced in the paper would also help readability.
questions: questions: please see weaknesses.
limitations: limitations: See weaknesses -- the work in it sucrrent form is limited to known geometry
paper_summary: paper_summary: This paper introduces SyncTweedies, a novel framework to generate diverse visual content such as ambiguous images, panoramas, mesh textures, and Gaussian splat textures. The method uses a synchronization process that averages outputs of Tweedie's formula across multiple instance spaces, eliminating the need for fine-tuning. The authors claim their method preserves the rich priors from large-scale datasets, enhancing generalizability. Experimental results show that SyncTweedies outperforms existing methods in various applications.
strength_weakness: strength_weakness: (1) The method assumes that the projection and unprojection functions are accurate, but in practice, these operations can introduce errors, especially in complex transformations such as those required for 3D mesh texturing. The author does not address how SyncTweedies handles these projection errors, which can accumulate and degrade the quality of the generated content.   

(2) The theoretical basis of SyncTweedies seems to implicitly assume that the data distributions in the instance and canonical spaces are well-aligned. However, this assumption may not hold in practice, especially when dealing with diverse and complex datasets. The paper does not explore the theoretical implications of misaligned data distributions and how they might affect the performance and stability of the synchronization process.
questions: questions: (1) How does SyncTweedies handle scenarios where projection and unprojection functions between instance spaces and the canonical space are not perfectly invertible? What impact does this have on the quality of generated content?  

(2) I am also wondering if the synchronization process can be optimized further to reduce computational overhead without compromising the quality of the generated content.
limitations: limitations: 1. The computational complexity and scalability of SyncTweedies are not thoroughly discussed (Lines 271-290). The synchronization process involves multiple denoising steps and averaging operations, which can be computationally intensive. There is no detailed analysis of the time and space complexity of the method, making it challenging to evaluate its feasibility for large-scale or real-time applications.    

2. The paper assumes a level of consistency and stability across instance spaces, which may not always be realistic (Lines 202-214). For example, in 3D mesh texturing, different views can have significantly different appearances due to factors like lighting, occlusion, and perspective distortion. SyncTweedies does not provide a thorough analysis of how it handles such variability. Robustness under varying conditions is crucial for practical applications
paper_summary: paper_summary: SyncTweedie attempts to elucidate the design space for synchronized diffusion. Synchronized diffusion means to optimize some representation (e.g., 3D mesh) by jointly diffusing its lower-dimensional projections (e.g., 2D image). The paper analyzes existing synchronized diffusion approaches on different tasks under the same umbrella on a diagnostic benchmark, and provides an empirical insight that one underexplored design is the best choice. The paper then verifies the insight in a range of tasks, including panorama image generation and 3D texturing.
strength_weakness: strength_weakness: 1. The depth-to-360$^\circ$ panorama result is blurry. In the meantime, the generated arbitrary-sized image seems to be fine. Is there any reason for that?
2. It would be better if generation diversity could be discussed, which I think would be an advantage over SDS-based works. How would different noise initialization affect the generated results and how robust is the approach given different random seeds?
3. Some implementation details are missing. 
     - The paper seems to miss the implementation details of the aggregation function for different tasks. For example, how is the view aggregation for 3DGS performed? Is it a gradient-based optimization or an analytical weighted sum as in MultiDiffusion[4]? This will also help to understand the runtime of the method.
     - What are the number of projected views (i.e., $N$) for each task? How are the views sampled?

Minor:
- L95: tdeterministic -> deterministic
- L594 refers to something in L628.
questions: questions: - The paper mentions that the framework cannot jointly optimize geometry and texture for 3D generation, could the authors elaborate more on why there is such a limitation? Does it suggest that the aggregation function $A$ cannot be too under-constrained?
limitations: limitations: Yes.

Comments: comment: We sincerely thank the reviewers for taking the time and effort to review our paper. We appreciate thoughtful and constructive comments, and they are indeed invaluable in improving our work. We will address additional suggestions in our final revision.
I appreciate the authors detailed response. My concerns have been addressed and I recommend acceptance.
Thanks for submitting the rebuttal which addresses my concerns. I think this is a good paper and deserves publication. As such, I will increase my rating to weakly accept.
Thanks for the rebuttal. It addresses most of my concerns. The clarification on blurry output, implementation details, and diversity looks good to me. One minor point about the joint texture and geometry optimization: "When the geometry is not given, the mappings cannot be defined", I think the mapping could be the 3D-to-2D rendering. I will keep my score.