Title: SyncTweedies: A General Generative Framework Based on Synchronized Diffusions
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion Models

Synchronization

Texturing

3D Gaussian Splatting

Mesh

Panorama
Abstract: We introduce a general diffusion synchronization framework for generating diverse visual content, including ambiguous images, panorama images, 3D mesh textures, and 3D Gaussian splats textures, using a pretrained image diffusion model. We first present an analysis of various scenarios for synchronizing multiple diffusion processes through a canonical space. Based on the analysis, we introduce a synchronized diffusion method, SyncTweedies, which averages the outputs of Tweedie’s formula while conducting denoising in multiple instance spaces. Compared to previous work that achieves synchronization through finetuning, SyncTweedies is a zero-shot method that does not require any finetuning, preserving the rich prior of diffusion models trained on Internet-scale image datasets without overfitting to specific domains. We verify that SyncTweedies offers the broadest applicability to diverse applications and superior performance compared to the previous state-of-the-art for each application. Our project page is at https://synctweedies.github.io.

SyncTweedies: A General Generative Framework Based on Synchronized Diffusions

Abstract

We introduce a general diffusion synchronization framework for generating diverse visual content, including ambiguous images, panorama images, 3D mesh textures, and 3D Gaussian splats textures, using a pretrained image diffusion model. We first present an analysis of various scenarios for synchronizing multiple diffusion processes through a canonical space. Based on the analysis, we introduce a synchronized diffusion method, SyncTweedies , which averages the outputs of Tweedie's formula while conducting denoising in multiple instance spaces. Compared to previous work that achieves synchronization through finetuning, SyncTweedies is a zero-shot method that does not require any finetuning, preserving the rich prior of diffusion models trained on Internet-scale image datasets without overfitting to specific domains. We verify that SyncTweedies offers the broadest applicability to diverse applications and superior performance compared to the previous state-of-the-art for each application. Our project page is at https://synctweedies.github.io .

1 Introduction

Image diffusion models [47, 38] have shown unprecedented ability to generate plausible images that are indistinguishable from real ones. The generative power of these models stems not only from their capacity to learn from a vast diversity of potential data but also from being trained on Internet-scale image datasets [49, 51].

Our goal is to expand the capabilities of pretrained image diffusion models to produce a wide range of 2D and 3D visual content, including panoramic images and textures for 3D objects, as shown in Figure 1, without the need to train diffusion models for each specific visual content. Despite the existence of general image datasets on the scale of billions [49], collecting other forms of visual data at this scale is not feasible. Nonetheless, most visual content can be converted into a regular image of a specific size through certain mappings, such as projecting for panoramic images and rendering for textures of 3D objects. Thus, we employ such a bridging function between each type of visual content and images, along with pretrained image diffusion models [47, 38].

We introduce a general generative framework that generates data points in the desired visual content space-referred to as canonical space-by combining the denoising process of diffusion models in the conventional image space-referred to as instance spaces. Given the bridging functions connecting the canonical space and instance spaces, we first explore performing individual denoising processes in each instance space while synchronizing them in the canonical space via the mapping. Another approach is to denoise directly in the canonical space, although it is not immediately feasible due to the absence of diffusion models trained on the canonical space. We investigate redirecting the noise prediction to the instance spaces but aggregating the outputs later in the canonical space.

Depending on the timing of aggregating the outputs of computation in the instance spaces, we identify five main possible options for the diffusion synchronization processes. Previous works [5, 18, 35] have investigated each of the possible cases only for specific applications, and none of them have analyzed and compared them across a range of applications. For the first time, we present a general framework for diffusion synchronization processes, within which the previous works [5, 18, 35] are contextualized as specific cases. We then present extensive analyses of different choices of diffusion synchronization processes. Based on the analyses, we demonstrate that the approach, which conducts denoising processes in instance spaces (not the canonical space) and synchronizes the outputs of Tweedie's formula [46] in the canonical space, provides the broadest applicability across a range of applications and the best performance. We name this approach SyncTweedies and showcase its superior performance in multiple visual content creation tasks compared with previous state-of-the-art methods.

Previous works [56, 34, 52, 63] finetune pretrained diffusion models to generate new types of outputs such as 360 ◦ panorama images and 3D mesh texture images. However, this approach requires a large quantity of target content for high-quality outputs which is prohibitively expensive to acquire. When it comes to generating visual content that can be parameterized into an image, a notable zero-shot approach not utilizing diffusion synchronization is Score Distillation Sampling (SDS) [41], which has shown particular effectiveness in 3D generation and texturing [31, 60, 62, 37]. However, this alternative application of diffusion models has been observed to produce suboptimal results and also requires a high CFG [22] weight for convergence, leading to over-saturation. For 3D mesh texture generation, specifically, an approach that iteratively updates each view image has also been explored in multiple previous works [10, 44, 8, 23, 17]. However, the accumulation of errors over iterations has been identified as a challenge. We demonstrate that our diffusion-synchronization-based approach outperforms these methods in terms of generation quality across various applications.

Overall, our contributions can be summarized as follows:

We propose, for the first time, a general generative framework for diffusion synchronization processes.

Through extensive analyses of various options for diffusion synchronization processes, including previous works [35, 18, 5, 33], we identify that the approach which synchronizes the outputs of Tweedie's formula and performs denoising in the instance space, SyncTweedies , offers the broadest applicability and superior performance.

In our experiments, we verify the superior performance and versatility of SyncTweedies across diverse applications, including texturing on 3D meshes and Gaussian Splats [26], and depth-to360-panorama generation. Compared to the previous state-of-the-art methods based on finetuning, optimization, and iterative updates, SyncTweedies demonstrates significantly better results.

2 Problem Definition

We consider a generative process that samples data within a space we term the canonical space Z , where a pretrained diffusion model is not provided. Instead, we leverage diffusion models trained in other spaces called the instance spaces {W i } i =1: N , where a subset of the canonical space can be instantiated into each of them via a mapping: f i : Z → W i ; we refer to this mapping as the projection . Let g i denote the unprojection , which is the inverse of f i , mapping the instance space to a subset of the canonical space. We assume that the entire canonical space Z can be expressed as a composition of multiple instance spaces W i , meaning that for any data point z ∈ Z , there exist { w i | w i ∈ W i } i =1: N such that

where A is an aggregation function that averages the data points from the multiple instance spaces in the canonical space. Our objective is to introduce a general framework for the generative process in the canonical space by integrating multiple denoising processes from different instance spaces through synchronization.

3 Diffusion Synchronization

We first outline the denoising procedure of DDIM [53] and then present possible options for diffusion synchronization processes based on it.

3.1 Denoising Process of DDIM [53]

Song et al. [53] have proposed DDIM, a generalized denoising process that controls the level of randomness during denoising. In DDIM [53], the posterior of the forward process is represented as follows:

where ψ ( t ) σ t ( x ( t ) , x (0) ) = √ α t -1 x (0) + √ 1 -α t -1 -σ 2 t 1 -α t · ( x ( t ) - √ α t x (0) ) and σ t is a hyperparameter determining the level of randomness. In this paper, we consider a deterministic process where σ t = 0 for all t , thus ψ ( t ) σ t =0 will be denoted as ψ ( t ) for simplicity. During denoising process, to sample x ( t -1) from its unknown original clean data point x (0) , we estimate x (0) using Tweedie's formula [46]:

where ϵ θ is a noise prediction network, and for simplicity, the time input and condition term in ϵ θ are dropped. In short, each deterministic denoising step of DDIM [53] is expressed as follows:

3.2 Diffusion Synchronization Processes

We now explore various scenarios of sampling z ∈ Z by leveraging the composition of multiple denoising processes in the instance spaces {W i } i =1: N . Consider the denoising step of the diffusion model at each time step t in each instance space W i :

A naïve approach to generating data in the canonical space through the denoising processes in instance spaces would be to perform the processes independently in each instance space and then aggregate the final denoised outputs in the canonical space at the end using the averaging function A . However, this approach results in poor outcomes that lack consistency across outputs in different instance spaces. Hence, we propose to synchronize the denoising processes at each time step t through the unprojection operation g i from each instance space to the canonical space and the aggregation operation A , after which the results will be back-projected via the projection operation f i to each instance space again. Note that, as described in Equation 4, the estimated mean of the posterior distribution ψ ( t ) ( · , · ) involves multiple layers of computations: noise prediction ϵ θ ( · ) , Tweedie's formula [46] ϕ ( t ) ( · , · ) approximating the final output x (0) each time step, and the final linear combination ψ ( t ) ( · , · ) . Synchronization through the sequence of unprojection g i , aggregation in the canonical space A , and projection f i can thus be performed after each layer of these computations, resulting in the following three cases:

(a) Instance variable denoising process

(b) Canonical variable denoising process

Case 1

Case 2

Case 3

Case 4

Case 5

In each case, we highlight the computation layer to be synchronized in red.

Another notable approach is to conduct the denoising process directly on the canonical space:

although it is not directly feasible because the noise prediction network in the canonical space ϵ θ ( z ( t ) ) is not available. Nevertheless, it can be achieved by redirecting the noise prediction to the instance spaces as follows:

(a) project the intermediate noisy data point z ( t ) from the canonical space to each instance space, resulting in f i ( z ( t ) ) ,

(b) apply a subsequence of the operations: ϵ θ , ϕ ( t ) , and ψ ( t ) ,

(c) unproject the outputs back to the canonical space via g i and then average them using the aggregation function A , and

(d) perform the remaining operations in the canonical space.

Such an approach of performing the denoising process in the canonical space leads to the following two additional cases depending on the subsequence of operations at step (b):

Illustration of the aforementioned diffusion synchronization processes are shown in Figure 2. Note the analogy between Cases 1 and 4, and Cases 2 and 5 in terms of the variable averaged in the canonical space with the aggregation operator A : either the outputs of ϵ θ ( · ) or ϕ ( t ) ( · , · ) .

While it is also feasible to conduct the aggregation A multiple times with the output of different layers within a single denoising step, and to denoise data both in instance spaces and the canonical space, we empirically find that such more convoluted cases perform worse. In Appendix H, we detail our exploration of all possible cases and present experimental analyses.

3.3 Connection to Previous Diffusion Synchronization Methods

Below, we first review previous works each corresponding to a specific case of the aforementioned possible diffusion synchronization processes while focusing on a specific application. Then, we discuss finetuning-based approaches and their limitations. In Section 4, we also review literature targeting the same applications but without diffusion synchronization.

3.3.1 Zero-Shot-Based Methods

Ambiguous Image Generation. Ambiguous images are images that exhibit different appearances under certain transformations, such as a 90 ◦ rotation or flipping. They can be generated through

a diffusion synchronization process, considering both the canonical space Z and instance spaces {W i } i =1: N as the same space of the image, with the projection operation f i representing the transformation producing each appearance. Visual Anagrams [18] uses Case 4 which aggregates the noise predictions ϵ θ ( · ) to generate ambiguous images.

Arbitrary-Sized Image Generation. In arbitrary-sized image generation, the canonical space Z is the space of the arbitrary-sized image, while the instance spaces {W i } i =1: N are overlapping patches across the arbitrary-sized image, matching the resolution of the images that the pretrained image diffusion model can generate. The projection operation f i corresponds to the cropping operation applied to each patch. MultiDiffusion [5] and SyncDiffusion [29] introduce arbitrary-sized image generation methods using Case 3, averaging the mean of the posterior distribution ψ ( t ) ( · , · ) .

Mesh Texturing. In 3D mesh texturing, the texture image space serves as the canonical space Z , and the rendered images from each view serve as the instance spaces {W i } i =1: N . The projection operation f i corresponds to rendering 3D textured meshes into 2D images. SyncMVD [35] proposes a 3D mesh texturing method by leveraging Case 5, which performs denoising in the canonical space and unprojects the outputs of Tweedie's formula [46] ϕ ( t ) ( · , · ) .

3.3.2 Finetuning-Based Methods

In addition to the aforementioned works, there have been attempts to achieve synchronization through finetuning. In multi-view image generation, SyncDreamer [34] and MVDream [52] finetune pretrained image diffusion models to achieve consistency across different views. MVDiffusion [56] and DiffCollage [65] generate 360 ◦ panorama images through finetuning. Additionally, Paint3D [63] trains an encoder to directly generate 3D mesh texture images in the UV space. However, these finetuning-based methods use target sample datasets [16, 9, 15, 13] that are smaller by orders of magnitude compared to Internet-scale image datasets [49], e.g., 10K panorama images [9] vs. 5B images [49]. As a result, they are prone to ovefitting and losing the rich prior and generalizability of pretrained image diffusion models [47, 48]. Additionally, the poor quality of textures in most 3D model datasets results in unsatisfactory texturing outcomes, even when using relatively large-scale datasets [16, 15]. In our experiments, we demonstrate that our zero-shot synchronization method, fully leveraging the pretrained model without bias toward a specific dataset, provides the best realism and widest diversity, assessed by FID and KID, compared to the finetuning-based methods.

3.4 Comparison Across the Diffusion Synchronization Processes

Here, we compare the five cases of diffusion synchronization processes in Section 3.2 and analyze their characteristics through various toy experiments.

3.4.1 Toy Experiment Setup: Ambiguous Image Generation

For the toy experiment setup, we employ the task of generating ambiguous images introduced by Geng et al. [18] (see Section 3.3.1 for descriptions of ambiguous images). In this setup, we consider two-view ambiguous image generation, where two different transformations are applied, each producing a distinct appearance. Note that one of the transformations is an identity transformation, while the other is chosen to simulate different scenarios of mapping pixels from the canonical space

Case 1

SyncTweedies

Case 2

Case 3

Visual Anagrams [18]

Case 4

Case 5

w

(0)

1

w

(0)

2

w

(0)

1

w

(0)

2

w

(0)

1

w

(0)

2

w

(0)

1

w

(0)

2

w

(0)

1

w

(0)

2

Figure 3: Qualitative results of ambiguous image generation. While all diffusion synchronization processes show identical results with 1 -to1 projections, Case 1, Case 3 and Visual Anagrams [18] (Case 4) exhibit degraded performance when the projections are 1 -ton . Notably, SyncTweedies can be applied to the widest range of projections, including n -to1 projections, where Case 5 fails to generate plausible outputs.

to the instance space: 1 -to1 , 1 -ton , and n -to1 projection. In 1 -to1 and n -to1 projections, we use the 10 transformations from Visual Anagrams [18], while for the 1 -ton projection, we apply rotation transformations with randomly sampled angles. For all projection cases, we use the 95 prompts from [18]. For more details on the experiment setups, refer to Appendix B.1.

3.4.2 1 -to1 Projection

In 1 -to1 projection case, the five cases of diffusion synchronization processes become identical, as shown in Appendix D. The quantitative and qualitative results of diffusion synchronization processes are presented in Table 1 and the first row of Figure 3, respectively, where the fully denoised instance variables, w (0) 1 and w (0) 2 , are displayed side by side. The results confirm that all diffusion synchronization processes produce the same outputs.

3.4.3 1 -ton Projection

We further investigate the five cases of diffusion synchronization processes with different transformations for ambiguous images. It is important to note that all the transformations previously mentioned are perfectly invertible, meaning: f i ( g i ( w i )) = w i . However, in certain applications, the projection f i is often not a function but an 1 -ton mapping, thus not allowing its inverse. For example, consider generating a texture image of a 3D object while treating the texture image space as the canonical space and the rendered image spaces as instance spaces. When mapping each pixel of a specific view image to a pixel in the texture image in the rendering process-with nearest neighbor sampling, one pixel in the texture space can be projected to multiple pixels. Hence, the unprojection g i cannot be a perfect inverse of the projection f i but can only be an approximation, making the reprojection error ∥ w i -f i ( g i ( w i )) ∥ small. This violates the initial conditions required for the proof in Appendix D that states Cases 1-5 become identical, and we observe that such a case of having 1 -ton projection f i can significantly impact the diffusion synchronization process.

As a toy experiment setup illustrating such a case with ambiguous image generation, we replace the 1 -to1 transformations used in Section 3.4.2 to rotation transformations with nearest-neighbor sampling. We randomly select an angle and rotate an inner circle of the image while leaving the rest of the region unchanged. Due to discretization, rotating an image followed by an inverse rotation may not perfectly restore the original image.

The second row of Table 1 and Figure 3 present the quantitative and qualitative results of 1 -ton projection experiment. Note that the performance of Case 1 and Visual Anagrams [18] (Case

4), which aggregate the predicted noises ϵ θ ( · ) from either instance variables w ( t ) i or a projected canonical variable f i ( z ( t ) ) respectively, significantly declines. Also, the performance of Case 3, which aggregates the posterior means ψ ( t ) ( · , · ) , shows a minor decline. The quality of Cases 2 and 5, however, remain almost unchanged. This highlights that the denoising process is highly sensitive to the predicted noise and to the intermediate noisy data points, while it is much more robust to the outputs of Tweedie's formula [46] ϕ ( t ) ( · , · ) , the prediction of the final clean data point at an intermediate stage.

3.4.4 n -to1 Projection

Then, do the results above conclude that both Cases 2 and 5 are suitable for all applications? Lastly, we consider the case when the projection f i also involves an n -to1 mapping. Such a scenario can arise when coloring not a solid mesh but a neural 3D representation rendered with the volume rendering equation [25, 26, 39]. Due to the nature of volume rendering, which involves sampling multiple points along a ray and taking a weighted sum of their information, the projection operation f i includes an n -to1 mapping. Note that this case also violates the initial conditions of the proof in Appendix D, which states that the diffusion synchronization cases become identical under specific initial conditions. Additionally, Case 5 results in poor outcomes due to a variance decrease issue. Let { x i } i =1: N be random variables, each sampled from x i ∼ N ( µ i , σ 2 t I ) , and x = ∑ N i =1 w i x i be the weighted sum, where 0 ≤ w i ≤ 1 and ∑ N i =1 w i = 1 . Then, x also follows the Gaussian distribution x ∼ N ( ∑ N i =1 w i µ i , ∑ N i =1 w 2 i σ 2 t I ) . From the triangle inequality [40], the sum of squares is always less than or equal to the square of the sum: ∑ N i =1 w 2 i ≤ ( ∑ N i =1 w i ) 2 = 1 , implying that the variance of x is mostly less than the variance of x i .

Consequently, when f i includes an n -to1 mapping, the variance of w ( t ) i , computed as a weighted sum over multiple points in the canonical space, is mostly less than the variance of z ( t ) . Thus, the final output of Case 5 becomes blurry and coarse since each intermediate noisy latent in instance spaces w ( t ) i experiences a decrease in variance compared to that of z ( t ) .

We validate our analysis with another toy experiment, where we use the same set of transformations used by Geng et al. [18] but with a multiplane image (MPI) [57] as the canonical space. The image of each instance space is rendered by first averaging colors in the multiplane of the canonical space and then applying the transformation. Ten planes are used for the multiplane image representation in our experiments. The results are presented in the third row of Table 1 and Figure 3. Notably, Case 5 fails to produce plausible images like the other cases, whereas Case 2 still generates realistic images.

Table 2 below summarizes suitable cases for each projection type. Note that Case 2 is the only case that is applicable to any type of projection function. Since Case 2 involves averaging the outputs of Tweedie's formula in the instance spaces, we name this case SyncTweedies . Experimental results with additional applications are demonstrated in Section 5, and analysis of all possible cases is presented in Appendix H.

4 Related Work

In addition to Section 3.3.1 introducing previous works on diffusion synchronization, in this section, we review other previous works that utilize pretrained diffusion models in different ways to generate or edit visual content.

Optimization-Based Methods. Poole et al. [41] first introduced Score Distillation Sampling (SDS), which facilitates data sampling in a canonical space by leveraging the loss function of the diffusion model training and performing gradient descent. This idea, originally introduced for 3D generation [58, 31, 55], has been widely applied to various applications, including vector image generation [24], ambiguous image generation [7], mesh texturing [37, 11, 62], mesh deformation [61], and 4D generation [32, 3]. Subsequent works [20, 27, 28] also proposed modified loss functions not to generate data but to edit existing data while preserving their identities. This approach, exploiting diffusion models not for denoising but for gradient-descent-based updating, generally produces less realistic outcomes and is more time-consuming compared to denoising-based generation.

Iterative View Updating Methods. Particularly for 3D object/scene texturing and editing, there are approaches to iteratively update each view image and subsequently refine the 3D object/scene. TEXTure [44], Text2Tex [10], and TexFusion [8] are previous works that sequentially update a partial texture image from each view and unproject it onto the 3D object mesh. For texturing 3D scene meshes, Text2Room [23] and SceneScape [17] take a similar approach and update scene textures sequentially. Instruct-NeRF2NeRF [19] proposed to edit a 3D scene by iteratively replacing each view image used in the reconstruction process. However, sequentially updating the canonical sample leads to error accumulations, resulting in blurriness or inconsistency across different views.

Utilization of One-Step Predictions. Previous works have utilized the outputs of Tweedie's formula to restore images [12, 66] and to guide the generation process [4, 29]. However, the one-step predicted samples are used to compute the gradient from a predefined loss function to guide the sampling process, rather than for synchronization, which differentiates from our approach.

Concurrent works [50, 59] also average the outputs of Tweedie's formula, similar to our approach, but they focus only on specific applications. For the first time, we present a general framework for diffusion synchronization and provide a comprehensive analysis of different synchronization methods across various applications.

Prompt

Diffusion Synchronization

Finetuning

-Based

-Based

Iter. View

Updating

Case 1

Sync-

Tweedies

Case 2

Case 3

Case 4

Sync-

MVD[35]

Case 5

Paint3D

[63]

Paint-it

[62]

TEXTure

[44]

Text2Tex

[10]

'Baseball

glove'

'Light

bulb'

5 Applications

We quantitatively and qualitatively compare SyncTweedies with the other diffusion synchronization processes, as well as the state-of-the-art methods of each application: 3D mesh texturing (Section 5.1), depth-to-360-panorama generation (Section 5.2) ,and 3D Gaussian splats [26] texturing (Section 5.3). Additional experiments and detailed setups are provided in Appendix , including (1) additional

qualitative results, (2) implementation details of each application, (3) arbitrary-sized image generation, (4) 3D mesh texture editing and diversity comparison, (6) runtime and VRAM usage comparisons, and (7) user preference evaluations.

Experiment Setup. In the case of instance variable denoising processes introduced in Section 3.2 (Cases 1-3), we initialize instance variables by projecting an initial canonical latent z ( T ) sampled from a standard Gaussian distribution N ( 0 , I ) : w ( T ) i ← f i ( z ( T ) ) . For n -to1 projection cases (e.g.,3D Gaussian splats texturing), the instance variables are directly initialized from a standard Gaussian distribution which can avoid the variance decrease issue discussed in Section 3.4.4.

For instance space denoising processes, the final canonical variables are obtained by synchronizing the fully denoised instance variables at the end of the diffusion synchronization processes. Refer to Section 3.3.1 for the detailed definition of the canonical space Z , the instance spaces {W i } i =1: N , the projection operation f i , and the unprojection operation g i in each application.

Evaluation Setup. Across all applications, we compute FID [21] and KID [6] to assess the fidelity of the generated images and CLIP similarity [42] (CLIP-S) to evaluate text alignment. We use a depth-conditioned ControlNet [64] as the pretrained image diffusion model.

5.1 3D Mesh Texturing

In 3D mesh texturing, projection operation f i is a rendering function which outputs perspective view images from a 3D mesh with a texture image. This operation represents a 1 -ton projection due to discretization. We evaluate five diffusion synchronization cases along with Paint3D [63], a finetuning-based method, Paint-it [62], an optimization-based method, and TEXTure [44] and Text2Tex [10], which are iterative-view-updating-based methods. We use 429 pairs of meshes and prompts used in TEXTure [44] and Text2Tex [10].

Results. We present quantitative and qualitative results in Table 3 and Figure 4, respectively. The results in Table 3 align with the observations shown in the 1 -ton projection case discussed in Section 3.4.3. SyncTweedies and SyncMVD [35] outperform other baselines across all metrics, but ours demonstrates superior performance compared to SyncMVD.

Notably, SyncTweedies outperforms Paint3D [63], a finetuning-based method, indicating that finetuning with a relatively small set of synthetic 3D objects [16] is not sufficient for realistic texture generation. This is further evidenced by the cartoonish texture of the car in row 1 of Figure 4. Optimization-based and iterative-view-updating-based methods produce unrealistic texture images, often exhibiting high saturation and visible seams, as seen in the baseball glove and light bulb in rows 2 and 3 of Figure 4. These issues are also reflected in the relatively high FID and KID scores in Table 3. See Appendix A for additional qualitative results.

5.2 Depth-to-360-Panorama

We generate 360 ◦ panorama images from 360 ◦ depth maps obtained from the 360MonoDepth [43] dataset. Here, f i projects a 360 ◦ panorama to a perspective view image, which is an 1 -ton projection due to discretization. We compare SyncTweedies with previous diffusion-synchronization-based methods [5, 18, 35] and MVDiffusion [56], which is finetuned using 3D scenes in the ScanNet [13] dataset. We generate a total of 500 360 ◦ panorama images at 0 ◦ elevation, with a field of view of 72 ◦ .

Results. We report quantitative results of the five diffusion synchronization processes discussed in Section 3.2 in Table 4. Table 4 demonstrates a trend consistent with the 1 -ton projection toy experiment results shown in Section 3.4.3. Specifically, SyncTweedies and Case 5, which synchronize the outputs of Tweedie's formula ϕ ( t ) ( · , · ) , exhibit the best performance. Notably, SyncTweedies demonstrates slightly superior performance across all metrics. On the other hand, MVDiffusion [56], which is finetuned using indoor scenes, fails to adapt to new, unseen domains and shows inferior results. The qualitative results are presented in Appendix A due to page limit.

Metric

Diffusion Synchronization

Optim.-

Based

Iter. View

Updating

Case 1

Sync-

Tweedies

Case 2

Case 3

Case 4

Case 5

SDS [41]

MVDream-

SDS [52]

IN2N [19]

FID [21]

↓

211.65

106.47

120.52

114.53

116.73

110.29

141.77

109.65

KID [6]

↓

85.11

14.62

19.15

17.11

18.35

19.71

38.69

15.73

CLIP-S [42]

↑

24.69

29.55

29.53

29.30

29.12

29.33

28.69

29.25

Prompt

Input

3DGS [26]

Diffusion Synchronization

Optim.-

Based

Iter. View

Updating

Case 1

Sync-

Tweedies

Case 2

Case 3

Case 4

Case 5

SDS [41]

MVDream-

SDS [52]

IN2N [19]

'[S

∗

] an

intricate wooden

carving of

a ship'

'[S

∗

] purple

microphone'

5.3 3D Gaussian Splats Texturing

Lastly, to verify the difference between SyncTweedies and Case 5 both of which demonstrate applicability up to 1 -ton projections as outlined in Section 3.4.3, we explore texturing 3D Gaussian Splats [26], exemplifying an n -to1 projection case. In 3D Gaussian splats texturing, the projection operation f i is an n -to1 case, characterized by a volumetric rendering function [25]. This function computes a weighted sum of n 3D Gaussian splats in the canonical space to render a pixel in the instance space. Note that in 3D Gaussian splats texturing, the unprojection g i and the aggregation A operation are performed using optimization.

While recent 3D generative models [55, 54] generate plausible 3D objects represented as 3D Gaussian splats, they often lack fine details in the appearance. We validate the effectiveness of SyncTweedies on pretrained 3D Gaussian splats [26] from the Synthetic NeRF dataset [39]. We use 50 views for texture generation and evaluate the results from 150 unseen views. For baselines, we evaluate diffusion-synchronization-based methods, the optimization-based methods, SDS [41], MVDreamSDS [52], and the iterative-view-updating-based method, Instruct-NeRF2NeRF (IN2N) [19].

Results. Table 5 and Figure 5 present quantitative and qualitative comparisons of 3D Gaussian splats [26] texturing. SyncTweedies , unaffected by the variance decrease issue, outperforms Case 5 both quantitatively and qualitatively, which is consistent with the observations from the toy experiments in Section 3.4.4. When compared to other baselines based on optimization (SDS [41] and MVDream-SDS [52]) and iterative view updating (IN2N [19]), ours outperforms across all metrics, especially by a large margin in FID [21]. As shown in Figure 5, optimization-based methods tend to generate textures with high saturation, while the iterative-view-updating-based method produces textures lacking fine details. Additional qualitative results are shown in Appendix A.

6 Conclusion

We have explored various scenarios of diffusion synchronization and evaluated their performance across a range of applications, including ambiguous image generation, panorama generation, and texturing on 3D mesh and 3D Gaussian splats. Our analysis shows that SyncTweedies , which averages the outputs of Tweedie's formula while conducting denoising in multiple instance spaces, offers the best performance and the widest applicability.

Limitations and Societal Impacts. Despite the superior performance of SyncTweedies across diverse applications, updating both the geometry and appearance of 3D objects remains an open problem. Also, since the pretrained image diffusion model may have been trained with uncurated images, SyncTweedies might inadvertently produce harmful content.

Acknowledgments

Thank you to Phillip Y. Lee for valuable discussions on diffusion synchronization, and to Jisung Hwang for providing the 3D mesh renderer. This work was supported by the NRF grant (RS-202300209723), IITP grants (RS-2022-II220594, RS-2023-00227592, RS-2024-00399817), and KEIT grant (RS-2024-00423625), all funded by the Korean government (MSIT and MOTIE), as well as grants from the DRB-KAIST SketchTheFuture Research Center, NAVER-Intel Co-Lab, Hyundai NGV, KT, and Samsung Electronics.

References

Luma AI. Genie.

Franz Aurenhammer. Voronoi Diagrams-a survey of a fundamental geometric data structure. CSUR , 1991.

Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4D-fy: Text-to-4d generation using hybrid score distillation sampling. arXiv preprint arXiv:2311.17984 , 2023.

Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In CVPR , 2023.

Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. In ICML , 2023.

Mikołaj Bi´ nkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In ICLR , 2018.

Ryan Burgert, Xiang Li, Abe Leite, Kanchana Ranasinghe, and Michael S Ryoo. Diffusion illusions: Hiding images in plain sight. arXiv preprint arXiv:2312.03817 , 2023.

Tianshi Cao, Karsten Kreis, Sanja Fidler, Nicholas Sharp, and Kangxue Yin. Texfusion: Synthesizing 3d textures with text-guided image diffusion models. In CVPR , 2023.

Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In International Conference on 3D Vision (3DV) , 2017.

Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. In ICCV , 2023.

Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3D: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22246-22256, 2023.

Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In ICLR , 2023.

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5828-5839, 2017.

DeepFloyd. Deepfloyd if. https://www.deepfloyd.ai/deepfloyd-if/ .

Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. In NeurIPS , 2024.

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR , 2023.

Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent scene generation. In NeurIPS , 2024.

Daniel Geng, Inbum Park, and Andrew Owens. Visual anagrams: Generating multi-view optical illusions with diffusion models. arXiv preprint arXiv:2311.17919 , 2023.

Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instructnerf2nerf: Editing 3d scenes with instructions. In ICCV , 2023.

Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. In ICCV , 2023.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS , 2018.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS , 2021.

Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In ICCV , 2023.

Ajay Jain, Amber Xie, and Pieter Abbeel. Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models. In CVPR , 2023.

James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. ACM TOG , 1984.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG , 2023.

Subin Kim, Kyungmin Lee, June Suk Choi, Jongheon Jeong, Kihyuk Sohn, and Jinwoo Shin. Collaborative score distillation for consistent visual editing. In NeurIPS , 2024.

Juil Koo, Chanho Park, and Minhyuk Sung. Posterior distillation sampling. In CVPR , 2024.

Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. Syncdiffusion: Coherent montage via synchronized joint diffusions. In NeurIPS , 2023.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML , 2022.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3D: High-resolution text-to-3d content creation. In CVPR , 2023.

Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. arXiv preprint arXiv:2312.13763 , 2023.

Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In ECCV , 2022.

Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. SyncDreamer: Generating multiview-consistent images from a single-view image. In ICLR , 2023.

Yuxin Liu, Minshan Xie, Hanyuan Liu, and Tien-Tsin Wong. Text-guided texturing by synchronized multi-view diffusion. arXiv preprint arXiv:2311.12891 , 2023.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR , 2021.

Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-NeRF for shapeguided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12663-12673, 2023.

Midjourney. Midjourney. https://www.midjourney.com/ .

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2021.

Dragoslav S Mitrinovic, Josip Pecaric, and Arlington M Fink. Classical and new inequalities in analysis . Springer Science & Business Media, 2013.

Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3D using 2D diffusion. In ICLR , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Manuel Rey-Area, Mingze Yuan, and Christian Richardt. 360MonoDepth: High-resolution 360deg monocular depth estimation. In CVPR , 2022.

Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided texturing of 3d shapes. ACM TOG , 2023.

Daniel Ritchie. Rudimentary framework for running two-alternative forced choice (2afc) perceptual studies on mechanical turk.

Herbert E Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory . Springer, 1956.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. In NeurIPS , 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An open large-scale dataset for training next generation image-text models. In NeurIPS , 2022.

Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as a generative prior. 2024.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL , 2018.

Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3d generation. In ICLR , 2024.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2021.

Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. LGM: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054 , 2024.

Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR , 2023.

Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion. In NeurIPS , 2023.

Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In CVPR , 2020.

Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2D diffusion models for 3D generation. In CVPR , 2023.

Xiaojuan Wang, Janne Kontkanen, Brian Curless, Steven M Seitz, Ira Kemelmacher-Shlizerman, Ben Mildenhall, Pratul Srinivasan, Dor Verbin, and Aleksander Holynski. Generative powers of ten. In CVPR , 2024.

Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurIPS , 2024.

Seungwoo Yoo, Kunho Kim, Vladimir G Kim, and Minhyuk Sung. As-Plausible-As-Possible: PlausibilityAware Mesh Deformation Using 2D Diffusion Priors. In CVPR , 2024.

Kim Youwang, Tae-Hyun Oh, and Gerard Pons-Moll. Paint-it: Text-to-texture synthesis via deep convolutional texture map optimization and physically-based rendering. arXiv preprint arXiv:2312.11360 , 2023.

Xianfang Zeng, Xin Chen, Zhongqi Qi, Wen Liu, Zibo Zhao, Zhibin Wang, BIN FU, Yong Liu, and Gang Yu. Paint3d: Paint anything 3d with lighting-less texture diffusion models. In CVPR , 2024.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In CVPR , 2023.

Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, and Ming yu Liu. Diffcollage: Parallel generation of large content with diffusion models. In CVPR , 2023.

Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-play image restoration. In CVPR , 2023.

Appendix

Table of Contents

A Qualitative Results

A.1 3D Mesh Texturing

As shown in Figure 6, SyncTweedies and SyncMVD [35] generate the most realistic output images, aligning with the results of n -to1 projection scenarios discussed in Section 3.4.3. Notably, Paint3D [63], a finetuing-based method, produces inferior textures, losing fine-details, as seen in the appearance of the clock in row 3 and the patterns of the ladybug in row 5. This demonstrates the challenge of acquiring a sufficient amount of high-quality texture images for satisfactory results. The optimization-based method [62] tends to produce images with high-contrast, unnatural colors, as evidenced in rows 4 and 6. Lastly, the iterative-view-updating-based methods [44, 10] show inconsistencies across views noticeable in the dumpster in row 1 and the television in row 9.

Prompt

Diffusion Synchronization

Finetuning

-Based

-Based

Iter. View

Updating

Case 1

Sync-

Tweedies

Case 2

Case 3

Case 4

Sync-

MVD[35]

Case 5

Paint3D

[63]

Paint-it

[62]

TEXTure

[44]

Text2Tex

[10]

'Dumpster'

'Toilet'

'Clock'

'Jeep'

'ladybug'

'iPod"

'Excavator"

'Orangutan'

'Tele-

vision set'

truck'

'UGG

boot'

'latern'

A.2 Depth-to-360-Panorama Generation

As shown in Figure 7, SyncTweedies and Case 5 demonstrate the best results, aligning well with the input depth maps, with SyncTweedies showing a slightly better alignment as indicated by the red arrow in Figure 7. On the other hand, MVDiffusion [56], which is finetuned with the depth maps of indoor 3D scenes from the ScanNet [13] dataset, produces suboptimal results and fails to generate realistic 360 ◦ panoramas for out-of-domain scenes. This demonstrates that MVDiffusion [56] is overfitting to the scenes encountered during finetuning, resulting in a loss of generalizability. Cases 1 and 4, which aggregate the predicted noise ϵ θ ( · ) , produce noisy outputs. Case 3 yields suboptimal panoramas, characterized by monochromatic appearances and a lack of detail.

Outdoor Scenes

'a house at

night'

'a church in

a large building'

Input

Depth

Case 1

SyncTweedies

Case 2

Case 3

Case 4

Case 5

MVDiffusion [56]

Indoor Scenes

'a room with

some empty walls'

'a living room inside a palace'

Input

Depth

Case 1

SyncTweedies

Case 2

Case 3

Case 4

Case 5

MVDiffusion [56]

A.3 3D Gaussian Splats Texturing

Figure 8 shows that SyncTweedies generates high-fidelity results with intricate details, such as the carvings of an excavator in row 2, while Case 5 lacks fine details. Optimization-based methods, SDS [41] and MVDream-SDS [52], produce artifacts characterized by high saturation, such as the corns in row 1 and the carrots in row 4. Notably, a finetuning-based method, MVDreamSDS [52], shows inferior quality to SDS. As discussed in Section 3.3.2, the poor quality of textures in the finetuning dataset [16] results in quality degradation. Iterative-view-updating-based method, IN2N [19], fails to preserve fine details, such as the head of the microphone in row 7.

Prompt

Input

3DGS [26]

Diffusion Synchronization

Optim.-

Based

Iter. View

Updating

Case 1

Sync-

Tweedies

Case 2

Case 3

Case 4

Case 5

SDS [41]

MVDream-

SDS [52]

IN2N [19]

'[S

∗

] corn'

'[S

∗

] a wooden

carving of a

excavator'

'[S

∗

] a drum

kit made

of ruby'

'[S

∗

] carrots'

'[S

∗

] a

military

ship at sea'

'[S

∗

] a

leather

chair'

'[S

∗

] a wooden

carving of a

microphone'

十：

B Details on Experiments

In this section, we provide details on the experiments discussed in Section 5 of the main paper. For all diffusion synchronization processes, we use a fully deterministic DDIM [53] sampling with 30 steps, unless specified otherwise.

We use DeepFloyd [14] as the pretrained diffusion model for the ambiguous image generation which denoises images in the pixel space. For the depth-to-360-panorama generation, 3D mesh texturing, and 3D Gaussian splats texturing, we employ a pretrained depth-conditioned ControlNet [64] which is based on a latent diffusion model, specifically Stable Diffusion [47]. For applications utilizing ControlNet, synchronization during the intermediate steps of diffusion synchronization processes occurs within the same latent space, except for 3D Gaussian splats texturing. In the case of 3D Gaussian splats texturing, synchronization takes place in the RGB space, and detailed explanations are provided in Section B.4.

In the 1 -ton projection cases, each instance space sample is unprojected into the canonical space, resulting in N unprojected samples, { g i ( w ( t ) i ) } N i =1 , where N is the number of views. The canonical space sample z ( t ) is then obtained by averaging these unprojected samples. The averaging can be weighted based on the visibility from each view. An illustration of the process is shown in Figure 9.

𝑔𝑖

𝑔𝑖

𝑔𝑖

𝑔𝑖

𝒵

𝒜

𝒲

𝐳 (𝑡)

𝑔𝑖 (𝐰

𝑖

𝑡

)

𝑖=1

𝑁

(a) Unprojection operation

(b) Aggregation operation

Evaluation Metrics. For all applications, we evaluate diversity and fidelity of the generated images using FID [21] and KID [6]. These metrics compute scores based on the distance between the distribution of the generated image set and that of the reference image set, with the reference set forming the target distribution. Refer to each application section for detailed description of constructing the generated image set and the reference image set.

To evaluate the text alignment of the generated images, we report CLIP similarity score [42] (CLIP-S) which measures the similarity between the generated images w (0) i and their corresponding text prompts p i in CLIP [42] embedding space. Additionally, in the ambiguous image generation, we report CLIP alignment score (CLIP-A) and CLIP concealment score (CLIP-C) following previous work, Visual Anagrams [18]. To compute the metrics, we begin by calculating a CLIP similarity matrix S ∈ R N × N from N pairs of transformations and text prompts:

̸

where E img ( · ) and E text ( · ) are the image encoder and the text encoder of the pretrained CLIP model [42], respectively. CLIP-A quantifies the worst alignment among the corresponding image-text pairs, specifically computed as min diag ( S ) . However, this metric does not account for misalignment failure cases, where p i is visualized in w (0) j for i = j . CLIP-C considers alignment of an (a) image (prompt) to all prompts (images) by normalizing the similarity matrix S using softmax:

where tr ( · ) denotes the trace of a matrix, and τ is the temperature parameter of CLIP [42]. We set τ to 0 . 07 .

B.1 Details on Ambiguous Image Generation - Section 3.4.1

We present the details of the ambiguous image generation experiments in Section 3.4.1. Quantitative and qualitative results are presented in Table 1 and Figure 3.

Evaluation Setup. To evaluate the fidelity of the generated images using FID [21] and KID [6], we create a reference set consisting of 5,000 generated images from Stable Diffusion 1.5 [47] with the same text prompts used in the generation of ambiguous images.

Implementation Details. We use DeepFloyd [14] which is a two-stage cascaded pixel-space diffusion model. In the first stage, we generate 64 × 64 images that are upscaled to 256 × 256 images in the subsequent stage.

Definition of Operations. In the context of ambiguous image generation, both the instance variables { w i } i =1: N and canonical variables z share the same image space. However, instance variables exhibit different appearances from the canonical variable upon applying certain transformations.

In the 1 -to1 projection case, we use the 10 transformations used in Visual Anagrams [18], all of which are 1 -to1 mappings. The projection operation f i is defined as the transformation itself, and the unprojection operation g i is defined as the inverse of the transformation matrix.

In the scenario of 1 -ton projection, we employ inner circle rotation as the projection operation f i . This involves rotating the pixels within an inner circle of an image while keeping the outer pixels unchanged. The unprojection operation g i is the inverse of f i . We use 14 inner circle rotation transformations, with rotation angles evenly spaced in the range [45 ◦ , 175 ◦ ] . For evaluation, we utilize the same 95 prompts as in the 1 -to1 case for each transformation, generating 14 × 95 = 1 , 350 ambiguous images. After applying a rotation transformation, the grid of the rotated image does not align with the original image grid. Thus, we use the nearest-neighbor sampling to retrieve pixel colors from the original image to the rotated image. This sampling process leads to a scenario where a single pixel in the original image z can be mapped to multiple pixels in the rotated image w i , which is an 1 -ton mapping.

For n -to1 projection, we use the same transformations and text prompts as in the 1 -to1 projection experiment, thus resulting in a total of 10 × 95 = 950 ambiguous images. The only difference from the 1 -to1 projection experiment is that the canonical space variable z is now represented as multiplane images (MPI) [57], where a collection of planes { p j } j =1: M represents a single canonical variable. Specifically, we compute z by averaging the multiplane images: z = 1 M ∑ M j =1 p j . In the context of n -to1 projection, we substitute the sequence of the unprojection g i and the aggregation A operation with an optimization process. The multiplane images { p j } are optimized using the following objective function:

where wet set the number of planes M = 10 .

B.2 Details on 3D Mesh Texturing - Section 5.1

We provide details of the 3D mesh texturing experiments presented in Section 5.1. Quantitative and qualitative results are shown in Table 3 and Figure 6.

Evaluation Setup. We use 429 mesh and prompt pairs collected from previous works, TEXTure [44] and Text2Tex [10]. For texture generation, we use eight views sampled around the object with 45 ◦ intervals at 0 ◦ elevation. Two additional views are sampled at 0 ◦ and 180 ◦ azimuths with 30 ◦ elevation. For evaluation, we render each 3D mesh to ten perspective views with randomly sampled azimuths at 0 ◦ elevation, resulting 10 × 429 = 4,290 images. Following SyncMVD [35], the reference set images are generated by ControlNet [64] using the same depth maps and text prompts used in the texture generation.

Implementation Details. The resolution of the latent texture image is 1 , 536 × 1 , 536 , and that of the latent perspective view images is 96 × 96 . In the RGB space, the resolution of the texture image is 1 , 024 × 1 , 024 and that of the perspective view images is 768 × 768 .

We adopt two approaches introduced in SyncMVD [35]: Voronoi-diagram-based filling [2] and modified self-attention layers. First, the high resolution of the latent texture image results in a texture image with sparse pixel distribution. To address this issue, we propagate the unprojected pixels to the visible regions of the texture image using the Voronoi-diagram-based filling. Second, spatially distant views tend to generate inconsistent outputs. Therefore, we adopt the modified self-attention mechanism that attends to other views when computing the attention output.

Definition of Operations. In the 3D mesh texturing, the canonical variable z is the texture image of a 3D mesh, and the instance variables { w i } i =1: N are rendered images from the 3D mesh. The projection operation f i is a rendering function where nearest-neighbor sampling is utilized to retrieve the color from the texture image to perspective view images.

The unprojection operation g i is performed using optimization where the texture image z is updated to minimize the rendering loss with the multi-view images { w i } i =1: N . The projection operation of 3D mesh texturing may involve mapping one pixel in the texture image z to multiple pixels in a rendered image w i . Hence, this application corresponds to the 1 -ton projection case as in Section 3.4.3.

B.3 Details on Depth-to-360-Panorama Generation - Section 5.2

We provide details of the depth-to-360-panorama generation experiments presented in Section 5.2. Refer to Table 4 and Figure 7 for quantitative and qualitative results.

Evaluation Setup. We evaluate SyncTweedies and the baselines on 500 pairs of 360 ◦ panorama images and depth maps randomly sampled from the 360MonoDepth [43] dataset. For each 360 ◦ panorama image, we generate a text prompt using the output of BLIP [30] by providing a perspective view image of the panorama as input.

In the 360 ◦ panorama generation, we use eight perspective views by evenly sampling azimuths with 45 ◦ intervals at 0 ◦ elevation. Each perspective view has a field of view of 72 ◦ for diffusionsynchronization-based methods and 90 ◦ for MVDiffusion [56]. For evaluation, we project the generated 360 ◦ panorama image to ten perspective views with randomly sampled azimuths at 0 ◦ elevation and a field of view of 60 ◦ . Similarly, the reference set images are obtained by projecting each ground truth 360 ◦ panorama image into ten perspective views with azimuths randomly sampled and at 0 ◦ elevation. In total, we use 500 × 10 = 5 , 000 perspective view images for evaluation.

Implementation Details. We set the resolution of a latent panorama image to 2,048 × 4,096 and that of the latent perspective view images to 64 × 64. In the RGB space, a panorama image has a resolution of 1,024 × 2,048, and perspective view images have a resolution of 512 × 512. As done in the 3D mesh texturing, we apply the Voronoi-diagram-based filling [2] after each unprojection operation and employ the modified self-attention mechanism.

Definition of Operations. In the 360 ◦ panorama generation, the canonical variable z represents a 360 ◦ panorama image, while the instance variables { w i } i =1: N correspond to perspective views of the panorama. The mappings between the panorama image and the perspective views are computed as follows: First, we unproject the pixels of the perspective view image to the 3D space. Then, we apply two rotation matrices based on the azimuth and elevation angles. The pixels are then reprojected onto the surface of a unit sphere, represented as longitudes and latitudes. These spherical coordinates are finally converted to 2D coordinates on the panorama image.

Given the mappings, the projection operation f i samples colors from the panorama image using the nearest-neighbor method. Since a single pixel of a panorama image z can be mapped to multiple pixels of a perspective view image w i , the 360 ◦ panorama generation is a 1 -ton projection case, as discussed in Section 3.4.3.

B.4 Details on 3D Gaussian Splats Texturing - Section 5.3

We provide details of the 3D Gaussian splats texturing experiment presented in Section 5.3. Quantitative and qualitative results are provided in Table 5 and Figure 8.

Evaluation Setup. For evaluation, we use 3D Gaussian splats trained with multi-view images from the Synthetic NeRF dataset [39], consisting of 8 objects. We generate 40 textured 3D Gaussian splats by utilizing five different prompts per scene. We use 50 views for texture generation and 150 unseen views for evaluation.

Implementation Details. As described in Section B, we employ ControlNet [64] which denoises latent images. To render the latent images, we replace the spherical harmonics coefficients of a 3D Gaussian splats to a 4-channel latent vector. For the optimization, we run 2,000 iterations with a learning rate of 0.025. When applicable, we perform the optimization in RGB space by decoding the latent variables for diffusion-synchronization-based methods.

Definition of Operations. The canonical variables { z j } j =1: M are 3D Gaussian splats and the instance space variables { w i } i =1: N are the rendered images from the 3D Gaussian splats. The

projection operation f i is a volume rendering function [25, 26] where the colors (latent vectors) of multiple 3D Gaussian splats are composited to render a pixel. This corresponds to the n -to1 projection as discussed in Section 3.4.4. In 3D Gaussian splats texturing, only the colors of 3D Gaussian splats z = { s j } j =1: M are optimized from multi-view images { w i } i =1: N , while keeping other parameters, such as positions, fixed, as done in the n -to1 experiment in Section 3.4.4.

C Arbitray-Sized Image Generation

In addition to the 1 -to1 projection case presented in Section 3.4.2, we present arbitrary-sized image generation. In contrast to depth-to-360-panorama generation, which corresponds to the 1 -ton projection case, arbitrary-sized image generation is a 1 -to1 projection case.

Evaluation Setup. We follow the evaluation setup used in SyncDiffusion [29]. Using Stable Diffusion 2.0 [47] as the pretrained diffusion model, we generate 500 arbitrary-sized images of 512 × 3 , 072 resolution per prompt. With six text prompts from SyncDiffusion [29], we generate a total of 500 × 6 = 3 , 000 arbitrary-sized images. For quantitative evaluation, we report FID [21], KID [6], and CLIP-S [42]. Each generated arbitrary-sized image is randomly cropped to partial view images with 512 × 512 resolution. For the reference set, we generate 3 , 000 images with a resolution of 512 × 512 from the pretrained diffusion model using the same text prompts.

Implementation Details. The resolution of latent arbitrary-sized image is 64 × 384 , and the resolution of an instance space sample is 64 × 64 . We use deterministic DDIM [53] sampling with 50 steps.

Definition of Operations. The projection operation f i corresponds to cropping a partial view of the arbitrary-sized image, which is a 1 -to1 projection. The unprojection operation g i is the inverse of the f i which pastes the partial view image onto the canvas of the arbitrary-sized image.

Results. We report quantitative results in Table 6 and qualitative results in Figure 10. As mathematically proven in Section D, the quantitative results show that all diffusion synchronization cases exhibit comparable performances, which aligns with the observations from the 1 -to1 experiment in Section 3.4.2. This is further supported by the qualitative results in Figure 10, where all cases produce identical arbitrary-sized images, indicating that any option can be used when the projection is 1-to-1.

Results using Gaudi Intel-v2. Additionally, we present qualitative results of arbitrary-sized image generation using Intel Gaudi-v2 in Figure 11, along with a comparison of computation times between Intel Gaudi-v2 and NVIDIA A6000 in Figure 12. We observe that Intel Gaudi-v2 achieves 1.8 to 1.9 times faster runtimes compared to the NVIDIA A6000.

'A photo of

a city skyline at night'

Case 1

SyncTweedies

Case 2

MultiDiffusion [5]

Case 3

Case 4

Case 5

'A photo of

a forest with a misty

fog'

Case 1

SyncTweedies

Case 2

MultiDiffusion [5]

Case 3

Case 4

Case 5

'Desert landscape with vast stretches of sand dunes'

'Vast mountain range with snow'

D 1 -to1 Projection

It is mathematically guaranteed that Cases 1-5 become identical when the mappings are 1 -to1 and noises are initialized by projecting from the canonical space w ( T ) i = f i ( z ( T ) ) , where z ( T ) ∼ N ( 0 , I ) . Note that ϕ ( t ) ( · , · ) and ψ ( t ) ( · , · ) are linear operations and commutative with other linear operation such as f i , A , and g i . Assume the following conditions hold:

Based on induction, we have:

where the last equality holds the induction hypothesis. This proves that Cases 4-5 are identical. For instance variable denoising cases we have:

NVIDIARTXA6000

100

Intel Gaudi-v2

80

Seconds (s)

60

40

20

0

512

1024

2048

3072

Resolution (Width; Height is fixed to 512)

where the last equality holds the induction hypothesis. This proves that Cases 1-3 are identical. Lastly, based on the definition of the projection operation, we have:

This proves that canonical variable denoising cases (Cases 4-5) are equivalent to Case 3.

We validate the proof both qualitatively and quantitatively in applications with 1 -to1 projection: ambiguous image generation and arbitrary-sized image generation in Section 3.4.2 and Section C, respectively, where all cases generate identical results.

Edited 3D mesh (

SyncTweedies)

'A nascar'

'A car with

'A lantern'

'A Chinese style lantern'

'A turtle'

'A golden statue of a turtle'

鑫

(a) Paint-it

(b) SyncTweedies

E 3D Mesh Texture Editing and Diversity

In this section, we extend the 3D mesh texture generation from Section 5.1 and present a texture editing application, along with a diversity comparison of SyncTweedies to the optimization-based method Paint-it [62].

E.1 3D Mesh Texture Editing

Despite the recent successes of 3D generation models [1, 34], the textures of the generated 3D meshes often lack fine details. We utilize SyncTweedies to edit the textures of the generated 3D meshes, and enhance the texture quality. Specifically, we use the 3D meshes generated from a text-to-3D model, Genie [1].

We follow SDEdit [36] to edit the textures of 3D meshes. We begin by adding noise at an intermediate time t ′ to the texture image of the 3D mesh and then perform a reverse process starting from t ′ .

Implementation Details. We set the CFG weight [22] to 30 and t ′ to 0 . 8 . For other settings, we follow the 3D mesh texture generation experiment presented in Section 5.1.

Results. We present qualitative results of 3D mesh texture editing in Figure 13. The 3D meshes edited with SyncTweedies exhibit fine details, including graffiti on the car in row 1, paintings on the lantern in row 2, and the intricate shells of the turtle in row 3.

E.2 Diversity of SyncTweedies

In Figure 14, we present qualitative results of 3D mesh texturing using the optimization-based method (Paint-it [62]) and SyncTweedies with different random seeds. SyncTweedies generates more diverse texture images compared to Paint-it.

F Runtime and VRAM Usage Comparison

As discussed in Section 4, one of the advantages of diffusion synchronization processes is the fast computational speed. We compare the runtime performance of SyncTweedies with optimizationbased and iterative-view-updating-based methods in the 3D mesh texturing and the 3D Gaussian splats texturing. The quantitative results are presented in Table 7.

In the 3D mesh texturing application, SyncTweedies shows faster running time than other baselines except TEXTure [44] which shows comparable running time. However, TEXTure [44] generates suboptimal texture outputs as observed in Table 3 and Figure 6. The finetuning-based method Paint3D [63] shows a comparable running time to SyncTweedies , but it shows inferior quality, as seen in Table 3 and Figure 6. Another iterative-view-updating-based method, Text2Tex [10], improves quality of texture image by integrating a refinement module, but this comes at the cost of additional overhead in terms of running time. In contrast, SyncTweedies achieves running times that are 7 times faster than Text2Tex and even outperforms across all metrics as shown in Table 3. Lastly, SyncTweedies shows 11 times faster running time when compared to Paint-it [62], an optimization-based method.

In the 3D Gaussian splats texturing, SyncTweedies achieves the fastest running time. SyncTweedies is 3 times faster than the iterative-view-updating-based method IN2N [19], and 8 times faster than the optimization-based method, SDS [41]. This shows that SyncTweedies not only generates high-fidelity textures, but also excels other baselines in computational speed. We use the NVIDIA RTX A6000 for the runtime comparisons.

Additionally, in Table 8, we present results comparing the VRAM usage of SyncTweedies and the baselines, where our SyncTweedies requires around 6-9 GiB of memory, making it suitable for most GPUs.

G User Study

We conduct user studies to evaluate the textures of the generated 3D Gaussian splats [26] through Amazon's Mechanical Turk. Following the methodology of Ritchie [45], participants were presented with input text prompts and randomly sampled output images generated by our method and the baseline methods. Participants are asked to choose the most plausible image that aligns with the given text prompt. In Table 9, our results are the most preferred in the human evaluations compared to the other baselines.

Details on User Study. We conduct separate user studies comparing our method to diffusionsynchronization-based methods, optimization-based methods (SDS [41], MVDream-SDS [52]), and iterative-view-updating-based method (IN2N [19]). For each user study, we use 20 images in a shuffled order including five vigilance tasks. We collect survey responses only from participants who pass the vigilance tasks. Specifically, 94 out of 100 participants passed in the test with Case 5, 90 out of 100 passed with SDS [41], 95 out of 100 passed with MVDream-SDS [52], and 92 out of 100 passed with IN2N [19]. Screenshots of the user study, including an example of vigilance tasks, are shown in Figure 15.

Left image

Right image

Trial 1 out of 25

Below are two images generated from the same text prompt. Based on the prompt

provided, which of the two images show better quality and alignment with the text

prompt?

Prompt:

'A photo of white cruise ship at sea.'

Trial 7 out of 25

Below are two images generated from the same text prompt. Based on the prompt

provided, which of the two images show better quality and alignment with the text

prompt?

Prompt:

'Please choose an image of a dog.'

Left image

Right image

(a) Main problem

(b) Vigilance task

(c) Instance variable denoising trajectory 2

(d) Canonical variable denoising trajectory 2

(a) Instance variable denoising trajectory 1

(b) Canonical variable denoising trajectory 1

(e) Instance variable denoising trajectory 3

(f) Canonical variable denoising trajectory 3

(g) Instance variable denoising trajectory 4

(h) Canonical variable denoising trajectory 4

>

(x)2

f

(t-1)

Z

A

gi

（t-1）

W

(2)

y(t)

Ot

b(t)

y(t)

Ot

(t)

W

E0

W:

(t-1)

(t)

W

2

E0

()

b(t)

(t)

Z

↑

A

gi

(t)

W

7

f

(t)

W!

(t-1)

Z

A

gi

（t-1）

W

A

gi

→

E0

A

gi

→

E0

f

f

y(t)

Ot

y(t)

Ot

W

(t-1)

+

b(t)

→

y(t)

t

g(t)

→

y(t)

Ot

(t)

Z

T

A

gi

(t)

W

2

z(t-1)

f

2(t)

f

→

z(t-1)

A

gi

→

E0

A

gi

→

E0

W

(t-1)

2

(t)

W

2

y(t)

七

y(t)

Ot

(t)

Z

T

A

gi

(t)

W

2

Z

（t-1)

f

Z(t)

f

→

z(t-1)

A

gi

(2

A

gi

→

(2)

E0

W

(t-1)

(t)

W

2

E0

H Analysis of Diffusion Synchronization Processes

As outlined in Section 3.4.4, we present a comprehensive analysis of all possible diffusion synchronization processes, including the representative five diffusion synchronization processes introduced in Section 3.2. Following the main paper, we categorize diffusion synchronization processes into two types: the instance variable denoising process , where instance variables { w ( t ) i } are denoised, and the canonical variable denoising process , which denoises a canonical variable z ( t ) directly. Unlike the representative cases, other all feasible cases either take inconsistent inputs when computing ϵ θ ( · ) , ϕ ( t ) ( · , · ) and ψ ( t ) ( · , · ) or conduct the aggregation A multiple times. Additionally, for a more exhaustive analysis, we introduce another type of diffusion synchronization processes, named the combined variable denoising process , which denoises { w ( t ) i } and z ( t ) together.

We present a total of 46 feasible cases for the instance variable denoising process, 8 for the canonical variable denoising process, and an additional 6 representative cases for the combined variable denoising process. We provide instance variable denoising cases in Section H.2, and canonical variable denoising cases in Section H.3. Additionally, the six representative cases for the combined variable denoising process are detailed in Section H.4. We conduct a quantitative comparison of all listed cases following the experiment setup outlined in Section 3.4.1, and the results are presented in Section H.5.

H.1 Overview

Weprovide the representative trajectories in Figure 16, where (a)-(b), (c)-(d), (e)-(f), and (g)-(h) follow the same trajectory but differ in the denoising variable, either instance or canonical, respectively. In each denoising case, there are 2 2 = 4 possible trajectories determined by whether ϕ ( t ) ( · , · ) and ψ ( t ) ( · , · ) are computed in the canonical space or instance space. This is because among the three computation layersϵ θ ( · ) , ϕ ( t ) ( · , · ) and ψ ( t ) ( · , · ) -only the last two operations can be computed in both the canonical space and the instance space unlike noise prediction which is only available in the instance space. Table 10 summarizes the computation spaces of ϕ ( t ) ( · , · ) and ψ ( t ) ( · , · ) , along with their corresponding trajectories.

Next, we introduce an additional operator F i that synchronizes instance variables. This operator unprojects a set of instance variables and averages them in the canonical space. Subsequently, the aggregated variables are reprojected to the instance space:

The red arrows in the diagrams of Figure 16 indicate the potential incorporation of F i . Thus, a total of 2 N different cases can be derived from a trajectory marked by N red arrows, depending on whether F i is applied to each variable or not.

Lastly, we review the five representative diffusion synchronization processes discussed in Section 3.2, along with two additional denoising processes: an instance variable denoising process that proceeds without synchronization (No Synchronization) and a canonical variable denoising process that averages the outputs of ψ ( t ) ( · , · ) (Case 6):

Note that Case 3 and 6 are identical except for the initialization, which can be either { w ( T ) i } or z ( T ) . For the independent instance variable denoising process (No Synchronization), synchronization is only applied at the end of the denoising process.

H.2 Instance Variable Denoising Process

Here, we explore all possible instance variable denoising processes. Here, the canonical space Z is employed to synchronize the outputs of ϵ θ ( · ) , ϕ ( t ) ( · , · ) and ψ ( t ) ( · , · ) in the instance spaces.

Following the trajectory 1 shown in part (a) of Figure 16, marked by five red arrows, there are a total of 2 5 = 32 possible denoising processes. This includes the independent instance variable denoising process (No Synchronization), where F i is not applied at any red arrow. Additionally, the three representative instance variable denoising processes, Cases 1-3, are also included, along with Cases 7-34 which are presented below:

Similarly, four cases are derived from the trajectory 2 shown in part (c) of Figure 16. These correspond to Cases 35-38 below:

The trajectory 3 shown in part (e) of Figure 16 accounts for two cases, corresponding to Cases 39-40 below:

Lastly, the trajectory 4 shown in part (g) of Figure 16 includes Cases 41-48 below:

H.3 Canonical Variable Denoising Process

Here, we present all possible canonical variable denoising processes. Due to the absence of noise prediction in the canonical space, a process first redirects canonical variable z ( t ) to the instance spaces where a subsequence of operations ϵ θ ( · ) , ϕ ( t ) ( · , · ) and ψ ( t ) ( · , · ) are computed.

We exclude the application of F i to w ( t ) i ← f i ( z ( t ) ) , as the variable remains unchanged after the operation. Therefore, applying F i to w ( t ) i ← f i ( z ( t ) ) for the inputs of ϵ θ ( · ) , ϕ ( t ) ( · , · ) and ψ ( t ) ( · , · ) is not considered.

Case 4 which belongs to the trajectory 3, is visualized in part (f) of Figure 16. Cases 5 and 49 derive from the trajectory 4 which are shown in part (h) of Figure 16.

In the trajectory 1, 2 2 = 4 cases are possible, as shown in part (b) of Figure 16. This includes Case 6 along with Cases 50-52 below:

Lastly, trajectory 2, shown in part (d) of Figure 16, encompasses one possible case, corresponding to Case 53:

H.4 Combined Variable Denoising Process

In this section, we introduce combined variable denoising processes where both instance and canonical variables are denoised. This process synchronizes instance variables and a canonical variable by aggregating the unprojected instance variables and the canonical variable in the canonical space.

For clarity, we introduce additional operations below. δ Z ( · ) takes a variable in the canonical space z ∈ Z , projects it into the instance spaces, predicts noises in those spaces, and aggregates them back in the canonical space after the unprojection. Φ ( t ) Z ( · ) then computes Tweedie's formula [46] based on the noise term computed by δ Z ( · ) .

Similarly, given a set of variables in the instance spaces { w i } , the following operators aggregate the unprojected outputs of ψ ( t ) ( · , · ) , ϵ θ ( · ) and ϕ ( t ) ( · , · ) in the canonical space:

We present joint variable denoising cases on the representative cases discussed in Section H:

Cases 54-59 correspond to the combined variable denoising processes from Cases 1-6, respectively. In each of the above cases, we highlight the terms already present in the original representative case in orange and newly added variable to be synchronized together in purple.

H.5 Quantitative Results

In Table 11, we present the quantitative results of the 60 diffusion synchronization processes listed above. We follow the same toy experiment setup described in Section 3.4.1 and Section B.1. As outlined in Section H.1, for all instance variable denoising processes, including the independent denoising case (No Synchronization), we perform the final synchronization at the end of the denoising process. For n -to1 projection, we utilize M = 10 multiplane images as done in Section 3.4.4.

We report the quantitative results of all cases in Table 11. The results align with the observations of Table 1. In the 1 -to1 projection scenario, most diffusion synchronization processes exhibit similar

performances. Except for Cases 55-56, the combined variable denoising processes (Cases 54-59) show suboptimal performances with FID [21] scores over 100. This indicates that denoising either instance variables or a canonical variable is sufficient to produce satisfactory and consistent results.

When it comes to the 1 -ton projection scenario, Cases 2 and 5 outperform the others, with some exceptions such as Cases 11 and 35. This trend is also consistent with the results in Section 3.4.3, highlighting the effectiveness of synchronizing the outputs of Tweedie's formula [46] ϕ ( t ) ( · , · ) even when compared to more complex diffusion synchronization processes.

Lastly, in the n -to1 projection scenario, Case 2 ( SyncTweedies ) is the only one that outperforms the others across all metrics.

In conclusion, as shown in Table 11, Case 2 ( SyncTweedies ) distinctly exhibits superior performance across various projection scenarios, outperforming even more convoluted cases.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We discuss the claims made in the abstract and introduction throughout the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the last section of the main paper, we address the limitation of our work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: All the results shown in the paper are obtained through experiments.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide experiment setups and implementation details in the main paper and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is publicly released.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed experiment setups in the main paper and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Due to our limited computational resources, we were unable to report error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide a runtime comparison of ours and other baselines in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss societal impacts in the last section of the main paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: NA.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have properly cited all sources.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: NA.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: We provide detailed setups for our user study in the appendix.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: We conducted the user study with IRB approval.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSyncTweedie attempts to elucidate the design space for synchronized diffusion. Synchronized diffusion means to optimize some representation (e.g., 3D mesh) by jointly diffusing its lower-dimensional projections (e.g., 2D image). The paper analyzes existing synchronized diffusion approaches on different tasks under the same umbrella on a diagnostic benchmark, and provides an empirical insight that one underexplored design is the best choice. The paper then verifies the insight in a range of tasks, including panorama image generation and 3D texturing.\n\nSTRENGTHS:\n1. The paper is enjoyable to read. The analysis of the 5 categories of diffusion is very clear. The 3 types of tasks (1-1, 1-N, N-1) are also nicely defined. There have indeed been extensive studies on different tasks based on a similar idea, this paper has done a nice job of reviewing them and elucidating the design space.\n2. The toy experiment is well-designed. It is a simple task, and the authors slightly modify it so it comprehensively covers all challenges in all types of tasks. The toy experiment is also diagnostic enough to provide empirical insights, which makes Case 2 (SyncTweedies) stands out.\n3. The experiments cover a range of tasks including arbitrary-sized image generation, panorama image generation, mesh texturing, and 3DGS texturing. I find these tasks sufficient to support the generalizability of the proposed approach.\n\nWEAKNESSES:\n1. The depth-to-360$^\\circ$ panorama result is blurry. In the meantime, the generated arbitrary-sized image seems to be fine. Is there any reason for that?\n2. It would be better if generation diversity could be discussed, which I think would be an advantage over SDS-based works. How would different noise initialization affect the generated results and how robust is the approach given different random seeds?\n3. Some implementation details are missing. \n     - The paper seems to miss the implementation details of the aggregation function for different tasks. For example, how is the view aggregation for 3DGS performed? Is it a gradient-based optimization or an analytical weighted sum as in MultiDiffusion[4]? This will also help to understand the runtime of the method.\n     - What are the number of projected views (i.e., $N$) for each task? How are the views sampled?\n\nMinor:\n- L95: tdeterministic -> deterministic\n- L594 refers to something in L628.\n\nQUESTIONS:\n- The paper mentions that the framework cannot jointly optimize geometry and texture for 3D generation, could the authors elaborate more on why there is such a limitation? Does it suggest that the aggregation function $A$ cannot be too under-constrained?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SyncTweedies, a novel framework to generate diverse visual content such as ambiguous images, panoramas, mesh textures, and Gaussian splat textures. The method uses a synchronization process that averages outputs of Tweedie's formula across multiple instance spaces, eliminating the need for fine-tuning. The authors claim their method preserves the rich priors from large-scale datasets, enhancing generalizability. Experimental results show that SyncTweedies outperforms existing methods in various applications.\n\nSTRENGTHS:\n(1) From my point of view, the framework's versatility is impressive, as it addresses various visual content generation tasks. This general applicability is a significant advantage over more specialized methods.  The use of Tweedie's formula for synchronizing diffusion processes across different instance spaces is genuinely novel. This methodological innovation could inspire new directions in the field of generative models. \n\n(2) SyncTweedies' ability to function without fine-tuning is a strong point. This feature ensures that the model retains its generalization capabilities, making it effective on diverse and previously unseen datasets. \n\n(3) Besides, the authors have conducted extensive experiments, comparing their method against several state-of-the-art techniques. These comparisons convincingly demonstrate the superior performance of SyncTweedies across multiple tasks.\n\nWEAKNESSES:\n(1) The method assumes that the projection and unprojection functions are accurate, but in practice, these operations can introduce errors, especially in complex transformations such as those required for 3D mesh texturing. The author does not address how SyncTweedies handles these projection errors, which can accumulate and degrade the quality of the generated content.   \n\n(2) The theoretical basis of SyncTweedies seems to implicitly assume that the data distributions in the instance and canonical spaces are well-aligned. However, this assumption may not hold in practice, especially when dealing with diverse and complex datasets. The paper does not explore the theoretical implications of misaligned data distributions and how they might affect the performance and stability of the synchronization process.\n\nQUESTIONS:\n(1) How does SyncTweedies handle scenarios where projection and unprojection functions between instance spaces and the canonical space are not perfectly invertible? What impact does this have on the quality of generated content?  \n\n(2) I am also wondering if the synchronization process can be optimized further to reduce computational overhead without compromising the quality of the generated content.\n\nLIMITATIONS:\n1. The computational complexity and scalability of SyncTweedies are not thoroughly discussed (Lines 271-290). The synchronization process involves multiple denoising steps and averaging operations, which can be computationally intensive. There is no detailed analysis of the time and space complexity of the method, making it challenging to evaluate its feasibility for large-scale or real-time applications.    \n\n2. The paper assumes a level of consistency and stability across instance spaces, which may not always be realistic (Lines 202-214). For example, in 3D mesh texturing, different views can have significantly different appearances due to factors like lighting, occlusion, and perspective distortion. SyncTweedies does not provide a thorough analysis of how it handles such variability. Robustness under varying conditions is crucial for practical applications", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates content generation within a target space using pretrained diffusion models operating in projected subspaces. The authors analyze five variants of the DDIM procedure, performed separately in each subspace and aggregated in the target space using known projection and unprojection operators. They explore different sequences of projection and aggregation, concluding that the optimal approach is to aggregate each subspace's \"estimated x0 from xt\" (the output of Tweedie's formula).\n\nThe method is evaluated through texturing tasks involving depth, mesh, and 3DGS, each representing distinct projection and unprojection scenarios. The ablation study and comparisons against baselines demonstrate the robustness of the proposed aggregation stage.\n\nSTRENGTHS:\n+ The paper effectively breaks down the multi-DDIM process, clearly delineating distinct alternatives that differ in the sequencing of the projection and aggregation operations.  \n+ The identification of three different tasks, which span a range of projection and un-projection operations, enriches the analysis and allows for a clear identification of the strengths and weaknesses of each aggregation strategy. I also appreciated the toy experiments, which effectively mimic the 3D effect in a simpler 2D setup.  \n+ The final strategy advocated for in this work aligns with the standard DDIM, is straightforward to implement, requires no additional tuning, and does not incur extra computational costs.\n\nWEAKNESSES:\n- Missing Explanations: The figures are difficult to understand due to missing information. For example, Figure 3 does not explicitly state the experimental setup, including the definitions of the canonical space and the subspaces.\n- The formulation of cases 1-3 that operate in the subspace does not explain how the recovered signal in z is finally returned. This is related to my previous comment, where it seems Figure 3 shows individual subspace generations rather than the end goal/task.\n- The initialization process is not discussed in the main paper. It is unclear whether the subspaces share the same initial noise. If they do, how is noise being generated in z? This is particularly challenging when z is 3DGS.\n- In the appendix, it states that the 3DGS are optimized towards the predicted clean images. Is this optimization considered the unprojection operation \"g\"? If so, this should be explicitly stated and moved to the main paper. More details on this optimization process are needed, including whether it is performed until convergence at each denoising time during generation.\n- In lines 129-130, phi and psi are used in the target domain. Please explain how this is done properly, given that noise is not injected in that space. Why would the coefficients in eq (3) and (4) hold true?\n\n- The caption of Figure 1, “Diverse visual content generated by SyncTweedies,” and the abstract phrase \"generating diverse visual content\" are misleading. They suggest the generation of general 3D content, while the method, as mentioned in the limitations, relies on known mappings between the subspaces and the target space, limiting it to texturing tasks. The authors should tone down these statements.\n- If the mapping from the canonical space to the instance spaces is simple (say, linear or even the Idnetity mapping), it seems all \"cases\" perform similarly both quantitatively and qualitatively. Do these methods mathematically converge to the same method, or are there still differences? In the 1-1 projection example in Figure 3, it appears all methods produce the same result. Is this correct? Table 10A shows almost identical results for different methods, and in Table 1, the scores for the 1-1 cases are very similar.\n- The claim of being the first to propose this method and that it was previously \"overlooked\" is too bold. Guidance literature shows that guidance in \"\\hat{x_0}\" offers more stable optimization and results. While guidance differs from multi-diffusion, it is similar enough to make the analogy. The authors should discuss similarities and differences, referencing papers like \"Universal Guidance for Diffusion Models\" and motion diffusion papers such as MDM and Trace&Pace. Further, if in degenerate cases of f and g maps (e.g. if they are the identity function) some \"cases\" are indistinguishable, be mindful when attributing aggregation choices to these methods (e.g., MultiDiffusion). Methods requiring tuning, like DiffCollage, should not be dismissed as baselines. The main contribution of this paper is identifying the best \"place\" to aggregate, and tuning is somewhat orthogonal. If \"Case 2\" improves DiffCollage, this should be demonstrated.\n- The term \"1:1 case\" is confusing and suggests disjoint individual subspaces.\n- The integration process of 1:n should also be better explained. It may also be useful to add illustration figures. \n- Improve Figure 2 for clarity with a more detailed caption. Connecting the diagram to the notations introduced in the paper would also help readability.\n\nQUESTIONS:\nplease see weaknesses.\n\nLIMITATIONS:\nSee weaknesses -- the work in it sucrrent form is limited to known geometry", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}