SyncTweedies: A General Generative Framework Based on Synchronized Diffusions
Authors: Jaihoon Kim
Juil Koo
Kyeongmin Yeo
Minhyuk Sung
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Models
Synchronization
Texturing
3D Gaussian Splatting
Mesh
Panorama
Primary area: diffusion_based_models
TL;DR: A general framework for diffusion synchronization

Abstract: We introduce a general diffusion synchronization framework for generating diverse visual content, including ambiguous images, panorama images, 3D mesh textures, and 3D Gaussian splats textures, using a pretrained image diffusion model. We first present an analysis of various scenarios for synchronizing multiple diffusion processes through a canonical space. Based on the analysis, we introduce a synchronized diffusion method, SyncTweedies, which averages the outputs of Tweedie’s formula while conducting denoising in multiple instance spaces. Compared to previous work that achieves synchronization through finetuning, SyncTweedies is a zero-shot method that does not require any finetuning, preserving the rich prior of diffusion models trained on Internet-scale image datasets without overfitting to specific domains. We verify that SyncTweedies offers the broadest applicability to diverse applications and superior performance compared to the previous state-of-the-art for each application. Our project page is at https://synctweedies.github.io.

SyncTweedies: A General Generative Framework Based on Synchronized Diffusions
Jaihoon Kim
Juil Koo*
Kyeongmin Yeo"
Minhyuk Sung
KAIST
{jh27kim, 63days, aaaaa, masung acc.stt......................... kr
Sopth-to-360-Panorram Generation
Input Depth Map
"Án inside of church"
Generuted Panoruma
Mesh Texturing
Input Mesh
dumpster
"A clutch bug"
"A hand carved wood turtle"
3D Gaussian Splat Texturing
"A majestic red chair"
Input 3DGS
"A white cruise ship"
Input 3DGS
"A pink chair"
Input 3DGS
"A yellow excavator covered in snow"
Figure 1: Diverse visual content generated by SyncTreedies: A diffusion synchroniza- tion process applicable various downstream tasks without finetuning.
Abstract
We introduce general diffusion synchronization framework for generating diverse visual content, including ambiguous images, panorama images, 3D mesh textures, and 3D Gaussian splats textures, using a pretrained image diffusion model. We first present an analysis of various scenarios for synchronizing multiple diffu- sion processes through a canonical space. Based on the analysis, we introduce a synchronized diffusion method, SynTTweedies, which averages the outputs of Tweedie's formula while conducting denoising multiple instance spaces. Compared to previous work that achieves synchronization through finetuning, SyncTweedies a zero-shot method that does not require any finetuning, pre- serving the rich prior of diffusion models trained on Internet-scale image datasets without overfitting to specific domains. We verify that SyncTweedies offers the broadest applicability to diverse applications and superior performance com- pared to the previous state-of-the-an for each application. Our project page at https:://synctweediee github. 10
Equul contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Image diffusion models 47 38 have shown unprecedented ability to generate plausible images that are indistinguishable from real ones. The generative power of these models stems not only from their capacity to learn from vast diversity of potential data but also from being trained Intemet-scale image datasets 44 51
Our goal is to expand the capabilities of pretrained image diffusion models to produce wide range of 2D and 3D visual content, including panoramic images and textures for 3D objects, as shown in Figure 1 without the need to train diffusion models for each specific visual content. Despite the existence of general image datasets on the scale of billions [49], collecting other forms of visual data at this scale is not feasible. Nonetheless, most visual content can be converted into a regular image of a specific size through certain mappings, such as projecting for panoramic images and rendering for textures of 3D objects. Thus, we employ such bridging function between each type of visual content and images, along with pretrained image diffusion models 47 38).
We introduce a general generative framework that generates data points in the desired visual content space-referred as canonical space- by combining the denoising process of diffusion models the conventional image space- space-retees space-referred to instance spaces. Given the bridging functions connecting the canonical space and instance spaces, we first explore performing individual denoising processes in each instance space while synchronizing them in the canonical space via the mapping. Another approach is to denoise directly in the canonical space, although it is not immediately feasible due to the absence of diffusion models trained the canonical space. We investigate redirecting the noise prediction to the instance spaces but aggregating the outputs later in the canonical space.
Depending on the timing of aggregating the outputs of computation in the instance spaces, we identify five main possible options for the diffusion synchronization processes. Previous works [5] 18 35 have investigated each of the possible cases only for specific applications, and none of them have analyzed and compared them across range of applications. For the first time, we present general framework for diffusion synchronization processes, within which the previous works 55 [8 35 are contextualized as specific cases. We then present extensive analyses of different choices of diffusion synchronization processes. Based on the analyses, we demonstrate that the approach, which conducts denoising processes in instance spaces (not the canonical space) and synchronizes the outputs of Tweedie's is formula [46] in the canonical space, provides the broadest applicability across range of applications and the best performance. We name this approach SyncTweedies and showcase its superior performance in multiple visual content creation tasks compared with previous state-of-the-art methods.
52 63 finetune pretrained diffusion models to generate new types of outputs such as 360 panorama images and 3D mesh texture images. However, this approach requires a large quantity of target content for high-quality outputs which is prohibitively expensive to acquire. When it comes to generating visual content that can be parameterized into an image, a notable zero-shot approach not utilizing diffusion synchronization is Score Distillation Sampling (SDS) [11. which has shown particular effectiveness in 3D generation and texturing 31 60, 62 37). However, this alternative application of diffusion models has been observed to produce suboptimal results and also
Previous works 56
requires a high CFG 22 weight for convergence, leading to over-saturation. For 3D mesh texture generation, specifically, an approach that iteratively updates each view image has also been explored in multiple previous works (10) 44 g 23 17]. However, the accumulation of errors over iterations has been identified as challenge. We demonstrate that diffusion- diffusion-synchroone www.cssonizitiinn-aaaaaaa outperforms these methods in terms of generation quality across various applications.
approach
Overall, our contributions can be summarized follows:
We propose. for the first time, general generative framework for diffusion synchronization processes.
Through extensive analyses of various options for diffusion synchronization processes, including previous works 35 18, 5 33), we identify that the approach which synchronizes the outputs of Tweedie's 's formula and performs denoising in the instance space, SyncTweedies, offers the broadest applicability and superior performance.
In our experiments, we verify the superior performance and versatility of SyncTweedies across diverse applications, including texturing on 3D meshes and Gaussian Splats (26). and depth-to- 360-panorama generation. Compared to the previous state-of-the-art methods based on finetuning, optimization, and iterative updates, SyncTweedies demonstrates significantly better results.

---

2 Problem Definition
We consider a generative process that samples data within a space we term the canonical space Z. where a pretrained diffusion model is not provided. Instead, we leverage diffusion models trained in other spaces called the instance spaces (Wi)- where subset of the canonical space can be instantiated into each of them via a mapping: fi 2 - W;; we refer to this mapping as the projection. Let gi denote the unprojection, which the inverse fi, mapping the instance space to subset of the canonical space. We assume that the entire canonical space Z can be expressed as a composition of multiple instance spaces Wi, meaning that for any data point 2, there exist [w; w, W.) W.)= such that
z = A (fg (=):))
(I)
where A is aggregation function that averages the data points from the multiple instance spaces in the canonical space. Our objective is to introduce a general framework for the generative process in the canonical space by integrating multiple denoising processes from different instance spaces through synchronization.
3 Diffusion Synchronization
We first outline the denoising procedure of DDIM 53 and then present possible options for diffusion synchronization processes based on it.
3.1 Denoising Process of DDIM [53
Song et al. (53] have proposed DDIM, a generalized denoising process that controls the level of randomness during denoising. In DDIM (53, the posterior of the forward process is represented as follows:
Go (x 1( x( x0) = = ( (x(11) x ),,),,;)))))
(2)
where v/co (x((), x())) va-
1-a--1-0 (x(1) Va,x and or a hyperparameter
1-as
determining the level of randomness. this paper, we consider a deterministic process where a; = 0
for all 4, thus v. will be denoted as vi(t) for simplicity. During denoising process, to sample x (f-1)
from its unknown original clean data point x()), we estimate x() using Tweedie's formula [46
(1) -I- (x)
6000 (x (x(x)))
vax
where Eg is noise prediction network, and for simplicity, the time input and condition term in Eø are dropped. In short, each deterministic denoising step of DDIM (53) is expressed as follows:
x (t-1) = (0 (x (000) (x(1) co (x
3.2 Diffusion Synchronization Processes
We now explore various scenarios of sampling z E by leveraging the composition of multiple denoising processes in the instance spaces [W; Fi=1:N- Consider the denoising step of the diffusion model each time step in each instance space WA:
w (f-1) = - (6 (w!"
(00) (w." co(w, (2))).
(5)
na've approach to generating data in the canonical space through the denoising processes in instance spaces would be to perform the processes independently in each instance space and then aggregate the final denoised outputs in the canonical space at the end using the averaging function A. However, this approach results in poor outcomes that lack consistency across outputs in different instance spaces. Hence, we propose to synchronize the denoising processes at each time step t through the unprojection operation 9, from each instance space the canonical space and the aggregation operation A, after which the results will be brok-ero via the projection operation fa to each instance space again. Note that, as described in Equation 4 the estimated mean of the posterior distribution (:(1( (...) involves multiple layers of computations: noise prediction c-).), Tweedie's 's formula [46] (...) approximating the final output (") each time step, and the final linear combination (,((),,.)) Synchronization through the sequence of unprojection yi, aggregation in the canonical space A, and projection fi can thus performed after each layer of these computations,
resulting in the following three cases:
3

---

Case 3
Case
Case
20))
((-1)
Case 4
Case
E0
2
W
(C-13
w
(a) Instance variable denoising process
(b) Canonical variable denoising process
Figure 2: Diagrams of diffusion synchronization processes. The left diagram depicts denoising
instance variables {w,), while the right diagram illustrates directly denoising canonical variable z.
Case w (1("11) 1610 (00 (AAlee/Ae (w))))
Case 2 W (t-1) =6600 (w() fA(AAto) (w) (A
cw
Case 3 Wi (t-1) fi(A(Hgs (w,) (1) (0t) (w) Ea (w, (1) NN ))=
In each case, we highlight the computation layer to be synchronized in red.
Another notable approach is to conduct the denoising process directly the canonical space:
z ((-1) ()) (z! eb(t) (x, (z))))))))
(6)
although is not directly feasible because the noise prediction network in the canonical space co (z) not available. Nevertheless, it can be achieved by redirecting the noise prediction to the instance spaces as follows:
(a) project the intermediate noisy data point (1) from the canonical space to each instance space, resulting in ())
(b) apply subsequence of the operations: ep, (1)), and v1(1)
(c) unproject the outputs back to the canonical space via and then average them using the aggregation funetion A, and
(d) perform the remaining operations in the canonical space.
Such an approach performing the denoising process in the canonical space leads to the following two additional cases depending the subsequence of operations at step (b):
Case 4 (t-11) (2(0(2()) (100 A((on(es Case 5 (t-11) (2(0, (o0 (o
Illustration of the aforementioned diffusion synchronization processes shown in Figure Note the analogy between Cases and 4, and Cases 2 and 5 in terms of the variable averaged in the canonical space with the aggregation operator A: either the outputs of ca(-) or (..))
While it is also feasible to conduct the aggregation A multiple times with the output of different layers within a single denoising step, and to denoise data both in instance spaces and the canonical space, we empirically find that such more convoluted cases perform worse. In Appendix H we detail our exploration of all possible cases and present experimental analyses.
3.3 Connection Previous Diffusion Synchronization Methods
Below, we first review previous works each corresponding to specific case of the aforementioned possible diffusion synchronization processes while focusing on a specific application. Then, we discuss finetuning-basee approaches and their limitations. In Section 4 we also review literature targeting the same applications but without diffusion synchronization.
3.3.1 Zero-Shot-Basse Methods
Ambiguous Image Generation. Ambiguous images are images that exhibit different appearances under certain transformations, such 90 rotation flipping. They can be generated through

---

diffusion synchronization process, considering both the canonical space Z and instance spaces (Wii=1: as the same space the image, with the projection operation f; representing the transfor- mation producing each appearance. Visual Anagrams [8] uses Case 4 which aggregates the noise predictions Eo()) to generate ambiguous images.
Arbitrary-Sized Image Generation. arbitrary-sized image generation, the canonical space the space of the arbitrary-sized image, while the instance spaces (WiJi=l:N are overlapping patches across the arbitrary-sizes image, matching the resolution of the images that the pretrained image diffusion model generate. The projection operation fi corresponds to the cropping operation applied to each patch. MultiDiffusion [5] and SyncDiffusion [29] introduce arbitrary- bbizrary-sized image generation methods using Case averaging the mean of the posterior distribution w)t)) (t,
Mesh Texturing. In 3D mesh texturing, the texture image space serves as the canonical space 2, and the rendered images from each view serve as the instance spaces {Wi]i=1.N- The projection operation corresponds to rendering 3D textured meshes into 2D images. SyncMVD 35 proposes a 3D mesh texturing method by leveraging Case 5, which performs denoising in the canonical space and unprojects the outputs of Tweedie's s formula [46] ( (s.
3.3.2 Finetuning-Basee Based Methods
In addition to the aforementioned works, there have been attempts to achieve synchronization through finctuning. In multi-view image generation, SyncDreamer [34] and MVDream [52] finetune pretrained image diffusion models to achieve consistency across different views. MVDiffusion (56
and DiftCollage [65] generate 360 panorama images through finetuning. Additionally, Paint3D 63
trains encoder to directly generate 3D mesh texture images in the UV space. However, these finetuning- baaed methods use target sample datasets [16
9
15 13
that are smaller by orders of
magnitude compared to Internet-scale image datasets 49 e.g.. 10K panorama images [9] 5B
images [49]. As result, they are prone to ovefitting and losing the rich prior and generalizability of Additionally, the poor quality of textures in most 3D model datasets results in unsatisfactory texturing outcomes, even when using relatively large-scale datasets [16 15). In experiments, demonstrate that zero-shot synchronization method, fully leveraging the pretrained model without bias toward specific dataset, provides the best realism and widest diversity, assessed by FID and KID, compared to the finetuning-based methods.
pretrained image diffusion models 47 48
Table A quantitative comparison ambiguous image generation. KID scaled by 10
For each row, we highlight the column whose value is within 95% of the best.
SyncTweedies Case 2
Visual Anagrams
Projection
Metric
Case
Case
Case 4
Case 5
CLIP-A 18
30.35
30.4
30.32
30.35
30.34
1-to-1
CLIP 18
64.52
64.48
64.49
64.59
64.48
Projection
FID 21
85.88
86.74
85.69
86.35
86.54
KID 6
32.37
32.59
32.57
32.41
32,86
CLIP-A 18
25.97
30.16
29.94
25.64
30.23
1-to-m
CLIP-C
18
$4.77
60.86
60.64
54.15
61.01
Projection
FID 21
232.65
10..51
117.84
257.53
108.22
KID g
216.71
77.16
85.52
257.43
74.48
CLIP-A
18
t
21.28
29.56
21.58
21.33
21.09
x--o-1
CLIP-C
18
49.94
63.1
50.58
50.05
50.04
Projection
FID 21
405.82
96.3
243.23
301.2
289.82
KID g
496.98
40.91
151.11
233.11
213.45
3.4
Comparison Across the Diffusion Synchronization Processes
Here, we compare the five cases of diffusion synchronization processes in Section 3.2 and analyze their characteristics through various toy experiments.
3.4.1 Toy Experiment Setup: Ambiguous Image Generation
For the toy experiment setup, we employ the task of generating ambiguous images introduced by Geng el al. 18 (see Section 3.3. for descriptions of ambiguous images). this setup, we consider two-view ambiguous image generation, where two different transformations are applied, each producing a distinct appearance. Note that one of the transformations an identity transformation. while the other is chosen to simulate different scenarios of mapping pixels from the canonical space
5

---

SyncIveadies Case
Visual Anagrams 18
Case
Case 3
Case
Case 5
1-to-1 Projection, "a photo of ship, a dog)"
1-bon Projection, painting {a vatering can, dragon]"
n-to-1 Projection, painting of airplane)"
Figure 3: Qualitative results of ambiguous image generation. While all diffusion synchronization processes show identical results with 1-to-1 projections, Case 1, Case and Visual Anagrams [18] (Case 4) exhibit degraded performance when the projections are 1-to-n. Notably, SyncTweedies can be applied to the widest range of projections, including n-to-1 projections, where Case 5 fails to generate plausible outputs.
to the instance space: 1-to-1, 1-to-n, and n-to-1 projection. 1-to-1 and n-to-1 projections, we use the 10 transformations from Visual Anagrams [18]. while for the 1-to-n projection, we apply rotation transformations with randomly sampled angles. For all projection cases, we use the 95 prompts from [8] For more details on the experiment setups, refer Appendix B.1
3.4.2 1-to-1 Projection
In 1-to-1 projection case, the five cases of diffusion synchronization processes become identical, as shown in Appendix D The quantitative and qualitative results of diffusion synchronization processes presented in Table|[and the first row of Figure respectively, where the fully denoised
instance variables, w (9) and w, are displayed side by side. The results confirm that all diffusion
(0)
synchronization processes produce the same outputs.
3.4.3 1-to-n Projection
We further investigate the five cases of diffusion synchronization processes with different transforma- tions for ambiguous images. is important to note that all the transformations previously mentioned perfectly invertible, meaning: (g)((w)))
However, in certain applications, the projection fi is often not a function but an 1-to-n mapping, thus not allowing its inverse. For example, consider generating a texture image of 3D object while treating the texture image space the canonical space and the rendered image spaces as instance spaces. When mapping each pixel of a specific view image pixel in the texture image in the rendering process witth with nearest neighbor sampling, one cannot be
pixel in the texture space can be projected to multiple pixels. Hence, the unprojection but can only be approximation, making the reprojection [[w - (g w.)) small. This violates the initial conditions required for the proof in Appendix D that states Cases 1-5 become identical, and we observe that such a case of having 1-to-n projection J. can significantly impact the diffusson synchronization process.
perfect inverse of the projection
As toy experiment setup illustrating such a with ambiguous image generation, we replace the 1-to-1 transformations used in Section 3.4.2 to rotation transformations with neireot- sampling. We randomly select angle and rotate inner circle of the image while leaving the rest of the region unchanged. Due to discretization, rotating an image followed by an inverse rotation may not perfectly restore the original image.
The second row of Table [ and Figure 3 present the quantitative and qualitative results of 1-to- projection experiment. Note that the performance of Case I and Visual Anagrams [18] (Case

---

4), which aggregate the predicted noises Eg(-) from either instance variables w! or projected canonical variable fi((()) respectively, significantly declines. Also, the performance of Case 3, which aggregates the posterior means 26(0) (..),, shows minor decline. The quality of Cases 2 and 5, however, remain almost unchanged. This highlights that the denoising process is highly sensitive to the predicted noise and to the intermediate noisy data points, while it is much more robust to the outputs of Tweedie's 's formula [46] (it) (:,.), the prediction of the final clean data point at an intermediate stage.
3.4.4 n-to-1 Projection
Then, do the results above conclude that both Cases and are suitable for all applications? Lastly, consider the case when the projection fi also involves an n-to-1 mapping. Such scenario can arise when coloring not a solid mesh but a neural 3D representation rendered with the volume Due to the nature of volume rendering, which involves sampling multiple points along a ray and taking weighted sum of their information, the projection operation fi includes an n-to-1 mapping. Note that this case also violates the initial conditions of the proof in Appendix D which states that the diffusion synchronization cases become identical under specific initial conditions. Additionally, Case results in poor outcomes due to a variance decrease issue. Let (x:)= be random variables, each sampled from x; ~ Ni Ii)), and x = ΣN WiX; be the weighted sum, where 0 S wi 5 1 and Ei w; 1. Then, also follows the Gaussian distribution X~~ N (Ei") wifh;: Ein w From the triangle inequality [40) the sum of squares is always less than or equal to the square of the sum: E w S (Ei wx)2 = 1, implying that the variance of x is mostly less than the variance of x,.
rendering equation [25 26 39]
Consequently, when Si includes an n-to-1 mapping, the variance of w (t) computed as a weighted sum over multiple points in the canonical space, mostly less than the variance of ztt), Thus, the final output of Case 5 becomes blurry and coarse since each intermediate noisy latent in instance spaces w experiences a decrease in variance compared to that of z (0)
We validate our analysis with another toy experiment, where we use the same set of transformations used by Geng et al. [18] but with multiplane image (MPI) [57] as the canonical space. The image of each instance space is rendered by first averaging colors in the multiplane of the canonical space and then applying the transformation. Ten planes are used for the multiplane image representation in our experiments. The results are presented in the third row of Table T and Figure 3 Notably, Case fails to produce plausible images like the other cases, whereas Case 2 still generates realistic images.
Table below summarizes suitable cases for each projection type. Note that Case 2 is the only case that is applicable to any type of projection function. Since Case 2 involves averaging the outputs of Tweedie's formula in the instance spaces, we name this case SyncTweedies. Experimental results with additional applications are demonstrated in Section 5 and analysis of all possible cases is presented in Appendix H
Table 2: Analysis of diffusion synchronization processes on different projection scenarios. SyncTreedies offers the broadest range of applications.
Projection
Application
Case
SyncTveedics
Case
1-te=1
Ambiguous images, Atbitrary-sized images
1-to-n
360" panoramas, mesh texturing
x
ni-to-1
3D Gaussian
Splats
20 texturing
Previous Work
MultiDiffusion
Visual Anagrams
[B]
SyncMVD B1
4
Related Work
In addition to Section 3.3. introducing previous works on diffusion synchronization, in this section, we review other previous works that utilize pretrained diffusion models in different ways generate or edit visual content.
7

---

Optimization-Base Methods. Poole er al. 41 first introduced Score Distillation Sampling (SDS), which facilitates data sampling in a canonical space by leveraging the loss function of the diffusion model training and performing gradient descent. This idea, originally introduced for 31) generation [58
55 has been widely applied to various applications. including vector image generation 24, ambiguous image generation [7]. mesh texturing (37 62). mesh deformation [6]]
31
and 4D generation [32 3]. Subsequent works [20] 27 28 also proposed modified loss functions not to generate data but to edit existing data while preserving their identities. This approach, exploiting diffusion models not for denoising but for gradient-descent--aaee -based updating, generally produces less realistic outcomes and is more tome-consuming compared to denoising- senoising-based generation.
Iterative View Updating Methods. Particularly for 3D object/scene texturing and editing, there are approaches to iteratively update each view image and subsequently refine the 3D object/scene. TEXTure [4], Text2Tex [10], and TexFusion [8] are previous works that sequentially update a partial texture image from each view and unproject it onto the 3D object mesh. For texturing 3D scene meshes, Text2Room [23] and SceneScape 177 take a similar approach and update scene textures sequentially. Destruct-NeRF2N5 view image used in the reconstruction process. However, sequentially updating the canonical sample leads to error accumulations, resulting in blurriness or inconsistency across different views.
19 proposed to edit 3D scene by iteratively replacing each
Utilization of One-Step Predictions. Previous works have utilized the outputs of Tweedie's 's formula to restore images [2 66] and to guide the generation process 4 [9]. However, the one-step predicted samples used compute the gradient from a predefined loss function to guide the sampling process, rather than for synchronization, which differentiates from our approach.
Concurrent works 500 59 also average the outputs of Tweedie's 's formula, similar to our approach, but they focus only on specifie applications. For the first time, we present a general framework for diffusion synchronization and provide a comprehensive analysis of different synchronization methods across various applications.
Table 3: A quantitative comparison in 3D mesh texturing. KID is scaled by 10 The best in each row is highlighted by bold.
Finctuning -Based
Optim. -Based
Iter. View Updating
Diffissio Synchromization
Metric
Sync- Tweediee
Sync- MVD [35]
Paint3D 63
Paint-it 662
TEXTure 44]
Text2Txx 10
Case 1
Case 3
Case 4
Case
Case
FID 21
135.61
21.76
36.12
131.67
22.76
31.66
28.23
34.98
26.10
KID 68.63
1.46
6.60
65.70
1.74
5.69
2.30
6.83
2.51
CLIP-S
25.26
28.89
27,88
25.31
28.82
28.04
28:55
28.63
27.94
Finetuning -Based
Optin. -Rased
Iter. View Updating
Diffusion Synchronization
Prompt
Sync- Tvoadies Case 2
Sync- MVD [35
Paini3 [i]
Paint-il [0
TEXTure 44
Tex12Tex
Case
Case
Case
Case 5
"Minivan"
"Baseball
glove"
"Light
bulb"
Figure 4: Qualitative results of 3D mesh texturing. SyncTweedies and SyncMVD 35 generate realistic texture images, achieving better results than other baselines including finetuning- method. Other diffusion synchronization cases fail to produce plausible textures.
5 Applications
We quantitatively and qualitatively compare SyncTweedies with the other diffusion synchronization processes, as well as the sate-oofthe-art methods of each application: 3D mesh texturing (Section
depth-to-360-pamnone 3600-aaooamma generation (Section (.2)) ,and 3D Gaussian splats (26) texturing (Section 5.3)
Additional experiments and detailed setups provided in Appendix, including (1) additional
8

---

Table 4: quantitative comparison depth-too3360-aanorama application.. KID is scaled by 10*. The best in each row is highlighted by bold.
SyncTveodies
Metric
Case
Case 2
Case 3
Case
Case
MVDiffusion S6
FID 21
364.61
42.11
55.95
348.18
43.39
80.51
KID G
375 42
21.19
34.67
362.77
87
56.91
qualitative results, (2) implementation details of each application, (3) arbitrary-sized image generation, (4) 3D mesh texture editing and diversity comparison, (6) runtime and VRAM usage comparisons, and (7) user preference evaluations.
Experiment Setup. In the case of instance variable denoising processes introduced in Section 3.2 (Cases 1-3), we initialize instance variables by projecting an initial canonical latent 2T)) sampled from a standard Gaussian distribution (V,,,)) I): w + fz()))) For n-to-1 projection cases (e.g..3D Gaussian splats texturing), the instance variables are directly initialized from standard Gaussian distribution which can avoid the variance decrease issue discussed Section 34.4
For instance space denoising processes, the final canonical variables are obtained by synchronizing the fully denoised instance variables the end of the diffusion synchronization processes. Refer to Section 3.3.1 for the detailed definition of the canonical space 2, the instance spaces (W; hWi the projection operation fi and the unprojection operation yi each application.
Evaluation Setup. Across all applications, we compute FID 21 and KID 6 to assess the fidelity of the generated images and CLIP similarity [42] (CLIP-S) to evaluate text alignment. We use centh-conditioned ControlNet [64 as the pretrained image diffusion model.
5.1 3D Mesh Texturing
In 3D mesh texturing, projection operation f. a rendering function which outputs perspective view images from 3D mesh with texture image. This operation represents 1-to-n projection due to discretization. We evaluate five diffusion synchronization cases along with Paint3D [63], inetuning-based method, Paint-it (62), optimization-basee method, and TEXTure 44 and
Text2Tex [10), which are iterative-viens vieaiiie-vie--pdating based methods. We use 429 pairs of meshes and prompts used in TEXTure 44 and Text2Tex (10.
Results. We present quantitative and qualitative results in T and Figure 4 respectively. The results in Table 3 align with the observations shown in the 1-to-n projection case discussed in Section 3.4.3 SyncTweedier and SyneMVD (35] outperform other baselines across all metrics, but ours demonstrates superior performance compared to SyncMVD.
Notably, SyncTweedies outperforms Paint3D 663] finetuning- based method, indicating that finetuning with a relatively small set of synthetic 3D objects [16] is not sufficient for realistic texture generation. This is further evidenced by the cartoonish texture of the in 1 of Figure 4 Oprimization-based and setaaiie-vie--uudattnnn often exhibiting high saturation and visible seams, methods produce unrealistic texture images, rows 2 and 3 of Figure 4 These issues are also reflected in the relatively high FID and KID scores in Table 3 See Appendix for additional qualitative results.
in the baseball glove and light bulb in
5.2 Sopph-to-300-Panorrmm
We generate 360 panorama images from 360 depth maps obtained from the 360MonoDepth (43 dataset. Here, fi projects 360 panorama perspective view image, which is 1-to-n projection due to discretization. We compare SyncTweedies with previous catiisioossynccromizattiiooo methods (5 18 35 and MVDiffusion (56). which is finetuned using 3D scenes in the ScanNet [33
dataset. We generate a total of 500 360 panorama images at 0" elevation, with a field of view 72
Results. We report quantitative results of the five diffusion synchronization processes discussed in Section 3.2 in Table 4 Table 4 demonstrates trend consistent with the 1-to-n projection toy experiment results shown in Section 3.4.3] Specifically, SyncTweedies and Case 5, which synchronize the outputs of Tweedie's 's formula (..-), exhibit the best performance. Notably, SyncTweedies demonstrates slightly superior performance across all metrics. On the other hand, MVDiffusion (56), which is finetuned using indoor scenes, fails to adapt to new, unseen domains and shows inferior results. The qualitative results are presented in Appendix A due to page limit.

---

Table 5: A quantitative comparison in 3D Gaussian splats [26] texturing. KID is scaled by 10" The best in each row is highlighted by bold.
Optim- Based
Her View Updating
Diffusion Synchronization
Metric
Syac Tweedies Case 2
MVDream- SDS 52
Case
Case
Case
41
IN2N 19
21 FTD
211.65
106.47
120.52
114.3
116.73
110.29
141.77
109.65
KID X
85.11
14.62 29.55
19.15
17.11
18.35
19.71
38.69
15.73
CLIP-S
42
29.53
29.25
Optim.- Based
View Updating
Input 3DGS 26
Diffusion Synchronization
Prompt
Sync- Teeediea Case
MVDream- SDS
Case
Case
Case
Case
SDS +
IN2N
"[S'] an intricate wooden carving of ship"
"[s"] purple microp nicrophone'
Figure 5: Qualitative results of 3D Gaussian splats [26] texturing. [S"] is a prefix prompt. We 'Make it to" for IN2N [19] and "A photo of" for the others. SyncTweedies generates high-fidelity textures, while Case 5 lacks fine details due to the variance reduction issue.
5.3 3D Gaussian Splats Texturing
Lastly, to verify the difference between SyncTweedies and Case both of which demonstrate applicability up 1-to-n projections outlined in Section 3.4.3 we explore texturing 3D Gaussian Splats (26), exemplifying an n-to-1 projection case. In 3D Gaussian splats texturing, the projection operation Ja is n-10-1 case, characterized by volumetric rendering function (55) This function computes weighted sum of n 3D Gaussian splats in the canonical space to render a pixel in the instance space. Note that in 3D Gaussian splats texturing, the unprojection gi and the aggregation A operation are performed using optimization.
While recent 3D generative models 55 54 generate plausible 3D objects represented as 3D Gaussian splats, they often lack fine details the appearance. We validate the effectiveness of SyncTweedies on pretrained 3D Gaussian splats [26] from the Synthetic NeRF dataset (39]. We use 50 views for texture generation and evaluate the results from 150 unseen views. For baselines, we evaluate fnffussion-symccoonizzziiiooonno
methods, the optimization-based methods, SDS [41, MVDream- and the wiw.diieeiiie-uuaaiiiinnn.
SDS 52
method, Instruct-NeRR2NeRR (IN2N) [9).
Results. Table 5 and Figure 5 present quantitative and qualitative comparisons of 3D Gaussian splats [26] texturing. SyncTweedies, unaffected by the variance decrease issue, outperforms Case both quantitatively and qualitatively, which consistent with the observations from the toy experiments in Section 3.4.4 When compared to other baselines based on optimization (SDS [1] and MVDream-SDD [52) and iterative view updating (IN2N 19 ) ours outperforms across metrics, especially by large margin in FID [21]. As shown in Figure optimization based methods tend to generate textures with high saturation, while the iterative-viewa view-updating based method produces textures lacking fine details. Additional qualitative results are shown in Appendix
Conclusion
We have explored various scenarios diffusion synchronization and evaluated their performance across range of applications. including ambiguous image generation, panorama generation, and texturing on 3D mesh and 3D Gaussian splats. Our analysis shows that SyncTweedies, which averages the outputs of Tweedie's 's formula while conducting denoising in multiple instance spaces, offers the best performance and the widest applicability.
Limitations and Societal Impacts. Despite the superior performance of SyncTweedies across diverse applications, updating both the geometry and appearance of 3D objects remains open problem. Also, since the pretrained image diffusion model may have been trained with uncurated images, SyncTweedies might inadvertently produce harmful content.
10

---

Acknowledgments
Thank you to Phillip Y. Lee for valuable discussions diffusion synchronization, and to Jisung Hwang for providing the 3D mesh renderer. This work was supported by the NRF grant (RS-2023- 00209723), ITTP grants (RS-2022-1122059) grant (RS-2024-004236255) all funded by the Korean government (MSIT and MOTIE), as well as grants from the DRB- HAISKAISTT Sketch TheFutree Research Center, NAVER- Intel Co-Lab, Hyundai NGV, KT, and Samsung Electronics.
RSS2002300022599, AS-2024-0039981
and KEIT
References
Luma AL Genie.
[2] Franz Aurenhammer. Voronoi Diagrams-a survey of fundamental geometric data structure. CSUR, 1991.
[3] Sherwin Bahmani, Ivan Skorokhodov, Victor Rong. Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4D-fy: Text-to-4d generation using hybrid score distillation sampling, arXiv preprint arXiv:2311.
2023.
[4] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance diffusion models. CVPR, 2023.
Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusson paths for controlled image generation. ICML, 2023.
[6] Mikolaj Binikowsii. Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In ICLR, 2018.
[7] Ryan Burgert, Xiang Li, Abe Leite, Kanchana Ranasinghe, and Michael Ryoo. Diffusion illusions: Hiding images plain sight. arXiv preprint arXiv:2312.03877 2023.
Tianshi Cao, Karsten Kreis, Sanja Fidler. Nicholas Sharp. and Kangxue Yin. Texfusion: Synthesizing 3d textures with text-guided image diffusion models. In CVPR, 2023.
[9] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: learning from rgh-d data indoor environments. In International Conference on 3D Vision (3DV), 2017.
[10] Dave Zhenyu Chen, Yawar Siddiqui, Hsin Ying Lee, Sergey Tulyakov, and Matthias NicBner. Text2tex: Text-driven texture synthesis via diffusion models. ICCV, 2023.
[11] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3D: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22246-22256, 2023.
[12] Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare Klasky, and Jong Chul Ye, Diffusion posterior sampling for general noisy inverse problems. In ICLR, 2023.
[13] Angela Dai, Angel Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017.
[14] DeepFloyd. Deepfloyd if. sttpp::/www.deepfllog/ aidepp
[15] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe 10m+ 3d objects. In NewrlPS, 2024.
[16] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, 2023.
[17] Rafail Fridman, Amit Abecasis. Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent scene generation. In NewrIPS, 2024.
[18] Daniel Geng, Inbum Park. and Andrew Owens. Visual anagrams: Generating multi-view optical illusions with diffusion models. arXiv preprint 2rXiv:2311.1799, 2023.

---

[19] Ayaan Haque, Matthew Tancik. Alexei A Efros. Aleksander Holynski, and Angjoo Kanazawa. Instruct- nerf2nerf: Editing 3d scenes with instructions. In ICCV, 2023.
[20] Amir Hertz, Klir Aberman, and Daniel Cohen-Or. Delta denoising score. In ICCV, 2023.
[21] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local nash equilibrium. In NeurlPS, 2018.
[22] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrlPS, 2021.
[23] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias NieBner. Text2room: Extracting textured meshes from 2d text-to-image models. /CCV, 2023.
[24] Ajay Jain, Amber Xie, and Picter Abbeel. Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models. In CVPR, 2023.
[25] James Kajiya and Brian Von Herzen, Ray tracing volume densities. ACM TOG, 1984.
[26] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG, 2023.
[27] Subin Kim, Kyungmin Lee, June Suk Choi, Jongheon Jeong, Kihyuk Sohn, and Jinwoo Shin. Collaborative score distillation for consistent visual editing. In NeurlPS, 2024.
[28] Juil Koo, Chanho Park, and Minhyuk Sung. Posterior distillation sampling. In CVPR, 2024.
[29] Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. Syncdiffusion: Coherent montage via synchronized joint diffuusions. In NeurlPS, 2023.
[30] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre- pre-training for unified vision-languag understanding and generation. In JCML, 2022.
[31] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung Yi Lin. Magic3D: High-resolution text-to-3d 3d content creation. In CVPR, 2023.
[32] Huan Ling, Scung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. arXiv preprint 2317::331..,366, 2023.
[33] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenhaum. Compositional visual generation with composable diffusion models. In ECCV. 2022.
[34] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjee Liu, Taku Komura, and Wenping Wang. SyncDreamer: Generating mulsiview-coniistent images from angle-view image. In ICLR, 2023.
[35] Yuxin Liu, Minshan Xie, Hanyuan Liu, and Tien-Tsin Tsin Wong, Text-guided texturing by synchronized multi-view diffusion. arXiv preprint arXiv:2311. 2023.
[36] Chenlin Meng. Yutong He. Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2021.
[37] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-NeBR NeRF for shape- guided generation of 3d shapes and textures. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12663-12673, 2023.
[38] Midjoumey. Midjourney. httpp://vwwt www.mmddddnny...mmmm com/
[39] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2021.
[40] Dragoslav S Mitrinovic, Josip Pecaric, and Arlington M Fink. Classical and new inequalities in analysis. Springer Science & Business Media, 2013.
[41] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3D using 2D diffusion. In ICLR, 2023.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
12

---

[43] Manuel Rey-Area, Mingze Yuan, and Christian Richardt. 3epMonoDepth: High-resolution 360deg monocular depth estimation. In CVPR, 2022.
[44] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Damiel Cohen-Or. Texture: Text-guided texturing of 3d shapes. ACM TOG, 2023.
[45] Daniel Ritchic. Rudimentary framework for running two-altemative foreed choice (2afc) perceptual studies mechanical turk.
[46] Herbert E Robbins. An empirical bayes approach statistics. In Breakthroughs in Statistics: Foundations and basic theory. Springer, 1956.
[47] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[48] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton. Kamyar Ghasemipour, Raphael Gontijo Lopes. Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. NeurIPS, 2022.
[49] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An open large-scalc dataset for training next generation image-text models. In NeurlPS, 2022.
[50] Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion generative prior. 2024.
[51] Piyush Sharma, Nan Ding, Scbastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
[52] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3d generation. In ICLR, 2024.
[53] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[54] Jiaxiang Tang. Zhaoxi Chen, Xiaokang Chen, Tengfei Wang. Gang Zeng. and Ziwei Liu. LGM: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.0500 2024.
[55] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In JCLR, 2023.
[56] Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi-view image generation with correspondence-wwarr diffusion. In NeurlPS, 2023.
[57]
Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In CVPR, 2020.
[58] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2D diffiusion models for 3D generation. In CVPR, 2023.
[59] Xiaojuan Wang, Janne Kontkanen, Brian Curless, Steven M Seitz, Ira Kemelmacher-Shhizze
Ben
Mildenhall, Pratul Srinivasan, Dor Verbin, and Alcksander Holynski. Generative powers of ten. In CVPR, 2024.
[60] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun 7hu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NewrlPS, 2024.
[61] Seungwoo Yoo, Kunho Kim, Vladimir Kim, and Minhyuk Sung. As-Plausible-A AailleAAsPPsssses Pososibbee Plausibility- Aware Mesh Deformation Using 2D Diffusion Priors. In CVPR, 2024.
[62] Kim Youwang, Tae-Hyun Oh, and Gerard Pons-Moll. Paint-it: Text-to-texture synthesis via deep con- volutional texture map optimization and physically-based rendering. arXiv preprint arXiv:2312. 13360, 2023.
[63] Xianfang Zeng. Xin Chen, Zhongqi Qi. Wen Liu, Zibo Zhao, Zhibin Wang, BIN FU, Yong Liu, and Gang Yu. Paint3d: Paint anything 3d with lighting -less texture diffusion models. In CVPR, 2024.
[64] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In CVPR, 2023.
[65] Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, and Ming yu Liu. Diffcollage: Parallel generation of large content with diffission models. In CVPR, 2023.
[66] Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Lue Van Gool. Denoising diffusion models for plug-and-plas image restoration. In CVPR, 2023.
13

---

Appendix
Table of Contents
Qualitative Results
15
A.I 3D Mesh Texturing
15
A.2
Deptl-oo-360-Panorrma Generation
16
A.3 3D Gaussian Splats Texturing
17
B
Details on Experiments
17
B.1 Details on Ambiguous Image Generation
Section
18 ................................
B.2 Details on 3D Mesh Texturing - Section 5.1
19
B.3 Details on Depth-to-360- 3ottt-tto--0--PaPaamaaa Generation -Setiion 5.2
20
B.4 Details on 3D Gaussian Splats Texturing Section 5.3
20
Arbitray-Sized Image Generation
21
D 1-to-1 Projection
23
3D Mesh Texture Editing and Diversity
25
E.1 3D Mesh Texture Editing
25
Diversity of SyncTweedies
25
Runtime and VRAM Usage Comparison
25
User Study
26
Analysis of Diffusion Synchronization Processes
28
Overview
28
H.2
Instance Variable Denoising Process
29
H.3
Canonical Variable Denoising Process
30
H.4
Combined Variable Denoising Process
31
H.5
Quantitative Results
14

---

A Qualitative Results
A.1 3D Mesh Texturing
As shown in Figure (6, SyncTweedies and SyncMVD [35] generate the most realistic output im- ages, aligning with the results of n-to-1 projection scenarios discussed in Section 3.4.3 Notably, Paint3D [63]. a finetuing-bate based method, produces inferior textures, losing fine-details, as seen in the appearance the clock in row and the patterns of the ladybug row 5. This demonstrates the challenge of acquiring sufficient amount of high-quality texture images for satisfactory results. The optimization based method [62] tends to produce images with high-contrast, unnatural colors, as evidenced in rows 4 and 6. Lastly, the prrative-vie-uupdaiine
methods 44 [0] show
inconsistencies across views noticeable in the dumpster in row and the television row
Finetuning
Optim.
Itr.. View
Diffusion Syachronization
Prompt
-Based
-Based
Updating
Sync- Tesadies Case 2
Sync- MVD [31
Paint3D
Paint-it
TEXTure
Texi2Tex
Case
Case
Case 4
10
Case
"Dampster"'
"Toilet"
"Clock"
"Jeep'
"ladybug"
"iPod"
"Excavator"
'Orangutan'
"Tela-
vision
"Trailer
boot"
"latern'
Figure 6: Qualitative results mesh texturing. SyncTweedies and SyncMVD
35
exhibit
comparable results, outperforming other baselines. Finetuning- based method [63] produces images without fine details as it was trained on dataset with coarse texture images. The optimization based method (62) tends to produce unrealistic and high saturation textures, while iterative- iterative-veewuupatig based methods [10] [4] show view inconsistencies.
15

---

A.2 Depth-to-360-Panoree Generation
As shown in Figure 7 SyncTweedies and Case 5 demonstrate the best results, aligning well with the input depth maps, with SyncTweedies showing slightly better alignment as indicated by the red arrow Figure On the other hand. MVDiffusion [56). which is finetuned with the depth maps of indoor 3D scenes from the ScanNet [13] dataset, produces suboptimal results and fails to generate realistic 360 panoramas for out-of-domain scenes. This demonstrates that MVDiffusion [56] is overfitting to the scenes encountered during finetuning, resulting in a loss of generalizability. Cases and 4, which aggregate the predicted noise co(-), produce noisy outputs. Case yields suboptimal panoramas, characterized by monochromatic appearances and lack of detail.
Outdoor Scenes
night"
church
larga building"
Input Depth
Case
SyneTveedies Case
Case
Case
Case
MVDiffusion 56
Indoor Scenes
"a room with some ampty valls'
"a living room inside a palace"
Input
Depth
Case
SyncIveedies
Case
Case
Case
Case
MVDiffusion
56
Figure 7: Qualitative results of depth-to-360-paaooraa generation. SyncTweedies and Case generate consistent and high-fidelity panoramas as observed in the 1-to-n projection experiment in Section 3.4.3 MVDiffusion [56) fails to generalize out-of-domain scenes and generates suboptimal panoramas.
16

---

A.3 3D Gaussian Splats Texturing
Figure shows that SyncTweedies generates high-fidelity results with intricate details, such as the carvings of an excavator in row 2, while Case lacks fine details. Optimization-baser methods, SDS [1 and MVDream-SDD [52], produce artifacts characterized by high saturation, such as the coms in row I and the carrots in row 4. Notably, finetuning- -hased method, MVDream- SDS [52], shows inferior quality to SDS. As discussed in Section 3.3.2] the poor quality of textures seadiivv--iew-uudatnn.. baadd method, IN2N [19] fails to preserve fine details, such as the head of the microphone in row 7.
in the finetuning dataset [16] results in quality degradation.
Optim Based
Prompi
Input 3DGS
Dilfusion Synchronization
Updating
Sync- Tveediea Case 2
MVDream-
Case
Case
Casc
Case
SDS
IN2N
SDS
"[s"] corn"
"[s"] wooden
carving of
excavator"
"[S"] drun
kit made
ruby"
"[s"] carrote"
"[s"]
military
ship at ses"
"[s"] leather
chair"
"[s"] wooden carving nicrophona"
Figure 8: Qualitative results of 3D Gaussian splats 26 texturing. [s*] is a prefix prompt. We use "Make it to" for IN2N [19] and "A photo of" for the other methods. Case 5 tends to lose details due the variance decrease issue, whereas SyncTweedies generates realistic images by avoiding this issue. The optimization based methods [1 52) produce high contrast, unnatural colors. and the iterative view updating method 19 yields suboptimal outputs due to error accumulation.
B Details on Experiments
In this section, we provide details on the experiments discussed in Section of the main paper. For all diffusion synchronization processes, we use fully deterministic DDIM 53 sampling with 30 steps, unless specified otherwise.
We use DeepFloyd [14] the pretrained diffusion model for the ambiguous image generation which denoises images in the pixel space. For the ceeth-to-360-panonee generation. 3D mesh texturing, and 3D Gaussian splats texturing, employ pretrained lepth-conditioned ControlNet [64] which is based on a latent diffusion model, specifically Stable Diffusion ControlNet, synchronization during the intermediate steps of diffusion synchronization processes occurs within the same latent space, except for 3D Gaussian splats texturing. In the case of 3D Gaussian splats texturing, synchronization takes place in the RGB space, and detailed explanations are provided in Section B.4
[7]. For applications utilizing
In the 1-to-n projection cases, each instance space sample is unprojected into the canonical space, resulting in N unprojected samples, {g. (w. (() (")) where N the number of views. The canonical space sample (4)) then obtained by averaging these unprojected samples. The averaging can be
weighted based on the visibility from each view, An illustration of the process is shown in Figure 9
17

---

z
A
(a.cw, (0) }mm
z(t)
W
(a) Unpxojection operation
(b) Aggregation operation
Figure 9: Illustration of unprojection and aggregation operation. The figure shows the synchro- nization process using the 3D mesh texturing application as an example. The left figure depicts the unprojection operation, where the instance space variables unprojected into the canonical space. The right figure illustrates the aggregation operation, where the unprojected samples are averaged in the canonical space.
Evaluation Metrics. For all applications, we evaluate diversity and fidelity of the generated images using FID (1 and KID [6]. These metrics compute scores based on the distance between the distribution of the generated image set and that of the reference image set, with the reference set forming the target distribution. Refer to each application section for detailed description of constructing the generated image set and the reference image set.
To evaluate the text aligmment of the generated images, we report CLIP similarity score 42 (CLIP-S)
(0) and their corresponding text prompts p; in CLIP [42) embedding space. Additionally, the ambiguous image generation, we report CLIP alignment score (CLIP-A) and CLIP concealment score (CLIP-C) following previous work, Visual Anagrams [18]. To compute the metrics, we begin by calculating a CLIP similarity matrix S R N from N pairs of transformations and text prompts:
which measures the similarity between the generated images w;
S.y = Eung (z)(z (
where Eing() and Erex(-) are the image encoder and the text encoder of the pretrained CLIP model [42), respectively. CLIP-A quantifies the worst alignment among the corresponding image-text pairs, specifically computed as min dSa.(S). However, this metric does not account for misalignment
(0) for * j. CLIP-C considers alignment of an (a) image (prompt) to prompts (images) by normalizing the similarity matrix S using softmax:
failure cases, where p; is visualized w/)
(r(sofimax(S/r))
(8)
where tr(-) denotes the trace of a matrix, and is the temperature parameter of CL.IP 0.07.
42 We set
B.1 Details on Ambiguous Image Generation Section 3.4.1
We present the details the ambiguous image generation experiments in Section 3.4.1 and qualitative results are presented in Table D and Figure 3
Quantitative
Evaluation Setup. To evaluate the fidelity of the generated images using FID 21 and KID 6,, we create a reference set consisting of 5,000 generated images from Stable Diffusion 1.5 47 with the same text prompts used in the generation of ambiguous images.
which is two-stage cascaded pixel-space diffusion model. the first stage, we generate 64 x 64 images that are upscaled 256 x 256 images in the subsequent stage.
Implementation Details. We use DeepFloyd 14
18

---

Definition of Operations. In the context of ambiguous image generation, both the instance variables (wi and canonical variables z share the same image space, However, instance variables exhibit different appearances from the canonical variable upon applying certain transformations.
In the 1-to-1 projection case, which are 1-to-1 mappings. The projection operation fi is defined as the transformation itself, and the unprojection operation gi is defined as the inverse of the transformation matrix.
the 10 transformations used in Visual Anagrams [8] all of
In the scenario of 1-to-n projection, we employ inner circle rotation as the projection operation fi- This involves rotating the pixels within an inner circle of an image while keeping the outer pixels unchanged. The unprojection operation is the inverse of ft. We use 14 inner circle rotation transformations, with rotation angles evenly spaced in the range [45", 175%]. For evaluation, utilize the same 95 prompts as in the 1-to-1 case for each transformation, generating 14 x 95 = 1, 1,300 ambiguous images. After applying a rotation transformation, the grid of the rotated image does not align with the original image grid. Thus, we use the nearest-neighbor sampling to retrieve pixel colors from the original image to the rotated image. This sampling process leads to a scenario where a single pixel in the original image z can be mapped to multiple pixels in the rotated image w,, which an 1-to-n mapping.
For n-to-1 projection, we use the same transformations and text prompts as in the 1-to-1 projection experiment, thus resulting in total of 10 95 950 ambiguous images. The only difference from the 1-to-1 projection experiment is that the canonical space variable is now represented as multiplane images (MPI) [57] where collection of planes {Pj}j=ı:as represents single canonical variable. Specifically, we compute z by averaging the multiplane images: z = 3 E Pj. In the context of n-to-1 projection, we substitute the sequence of the unprojection 9i and the aggregation A operation with an optimization process. The multiplane images {Pj} are optimized using the following objective function:
min (P,)
ft
Pj
W;
where wet set the number of planes M = 10.
B.2
Details on 3D Mesh Texturing - Section 5.1
We provide details of the 3D mesh texturing experiments presented in Section 5.1 Quantitative and qualitative results are shown in Table and Figure 6
Evaluation Setup. We use 429 mesh and prompt pairs collected from previous works, TEX- Ture 44 and Text2Tex [0]. For texture generation, we use eight views sampled around the object with 45 intervals at 09 elevation. Two additional views are sampled at 0' and 180 azimuths with 30% elevation. For evaluation, we render each 3D mesh to ten perspective views with randomly sampled azimuths at 0 clevation, resulting 10 x 429 = 4,290 images. Following SyncMVD [35], the
reference set images are generated by ControlNet 64 using the same depth maps and text prompts used in the texture generation.
Implementation Details. The resolution of the latent texture image 1, 5,6 536, and that of the latent perspective view images is 96 x 96. In the RGB space, the resolution of the texture image is 1, 024 024 and that of the perspective view images is 768 x 768.
We adopt two approaches introduced in SyncMVD [35] doromoi-diagram-bune based filling and modified self-attention layers. First, the high resolution of the latent texture image results in texture image with sparse pixel distribution. To address this issue, we propagate the unprojected pixels to the visible regions of the texture image using the Voronoi-diagram-basee filling. Second, spatially distant views tend to generate inconsistent outputs. Therefore, we adopt the modified self-attention mechanism that attends to other views when computing the attention output.
Definition of Operations. In the 3D mesh texturing, the canonical variable is the texture image of a 3D mesh, and the instance variables (w:)i=i: are rendered images from the 3D mesh. The projection operation f; is a rendering function where nearest-neighhor sampling utilized to retrieve the color from the texture image to perspective view images.
19

---

The unprojection operation is performed using optimization where the texture image z is updated to minimize the rendering loss with the multi-view images {w;}i-1:N: The projection operation of 3D mesh texturing may involve mapping one pixel in the texture image to multiple pixels in a rendered image W;. Hence, this application corresponds to the 1-to-n projection case as Section 3.4.3
B.3 Details
Septh-to-360-Pamorre Generation- - Section 5.2
We provide details of the copth-to-360-1e 360-paaoaa generation experiments presented in Section 5.2 Refer to Table 4 and Figure 7 for quantitative and qualitative results.
Evaluation Setup. We evaluate SyncTweedies and the baselines on 500 pairs of 360 panorama images and depth maps randomly sampled from the Dephh 43 dataset. For each 360 panorama image, we generate a text prompt using the output of BLIP 30 by providing a perspective view image of the panorama as input.
In the 360 panorama generation, we use eight perspective views by evenly sampling azimuths with 45 intervals at 0* elevation. Each perspective view has field of view of 72 for diffusion- synchronization- based methods and 90º for MVDiffusion [56]. For evaluation, we project the generated 360 panorama image to ten perspective views with randomly sampled azimuths at 0 elevation and field of view of 60%. Similarly, the reference set images are obtained by projecting each ground truth 360 panorama image into ten perspective views with azimuths randomly sampled and at elevation. In total, we use 500 10 = 5, 000 perspective view images for evaluation.
Implementation Details. We set the resolution of a latent panorama image to 2,048 x 4,096 and that of the latent perspective view images to 64 64. In the RGB space, panorama image has resolution of 1,024 x 2,048, and perspective view images have resolution of 512 x 512. As done in the 3D mesh texturing, we apply the prromoiddigrram-- basse filling 2 after each unprojection operation and employ the modified self-attention mechanism.
Definition of Operations. In the 360 panorama generation, the canonical variable represents 360 panorama image, while the instance variables [wihimiay correspond to perspective views the panorama. The mappings between the panorama image and the perspective views are computed as follows: First, we unproject the pixels of the perspective view image to the 3D space. Then, we apply two rotation matrices based the azimuth and elevation angles. The pixels are then reprojected onto the surface of unit sphere, represented as longitudes and latitudes. These spherical coordinates are finally converted to 2D coordinates on the panorama image.
Given the mappings, the projection operation fi samples colors from the panorama image using the nearest-neighbor method. Since single pixel of panorama image be mapped multiple pixels of a perspective view image w,, the 360 panorama generation a 1-to-n projection case, as discussed in Section 3443
B.4 Details on 3D Gaussian Splats Texturing - Section 5.3
We provide details of the 3D Gaussian splats texturing experiment presented in Section tive and qualitative results are provided in Table Figure
S.3
Quantita-
Evaluation Setup. For evaluation, we use 3D Gaussian splats trained with multi-view images from the Synthetic NeRF dataset [39], consisting of 8 objects. We generate 40 textured 3D Gaussian splats by utilizing five different prompts per scene. We use 50 views for texture generation and 150 unseen views for evaluation.
Implementation Details. As described in Section B we employ ControlNet [64 which denoises latent images. To render the latent images, we replace the spherical harmonics coefficients of a 3D Gaussian splats to a 4-channel latent vector. For the optimization, learning rate of 0.025. When applicable, we perform the optimization in RGB space by decoding the latent variables for frffussionsyncchronizatiin baaerrooniiziioobaaadddd methods.
2,000 iterations with
Definition of Operations. The canonical variables (zy)=IeM are 3D Gaussian splats and the instance space variables {ws JeI are the rendered images from the 3D Gaussian splats. The
20

---

projection operation f: is a volume rendering function [25 26 where the colors (latent vectors) of multiple 3D Gaussian splats are composited to render pixel. This corresponds to the n-to-1 projection as discussed in Section 3.4.4. In 3D) Gaussian splats texturing, only the colors of 3D Gaussian splats {sy] are optimized from multi-view images [wi]= while keeping other parameters, such as positions, fixed, as done in the n-to-1 experiment in Section 3.4.4
Table A quantitative comparison in arbitrary-sized image generation. KID scaled by 10 For each row, we highlight the column whose value is within 95% of the best.
Metric
Case
SyncTveedies Case 2
MuhiDiffusion Case
Case 4
Case
FID 21 KID x
32.83
32.82
32.83
32.82
32.83
7.79
7.79
7.79
7.79
7.80
CLIP-S 5
31.69
31.69
31.69
31.69
31.69
C Arbitray-Sized Image Generation
In addition to the 1-to-1 projection case presented in Section 3.4.2 we present arbitrary-sized image generation. contrast to depth-to-360-ppnorree generation, which corresponds to the 1-to-n projection case, arbitrary-sized image generation is 1-to-1 projection case.
Evaluation Setup. We follow the evaluation setup used in SyncDiffusion 22]. Using Stable
Diffusion 2.0 [47 as the pretrained diffusion model, we generate 500 arbitrary-sized images of 512 x 3, 072 resolution per prompt. With six text prompts from SyncDiffusion [29], we generate
total of 500 6 = 3, 000 arbitrary-sized images. For quantitative evaluation, we report FID (21],
KID [6, and CLIP-S 42. Each generated arbitrary-sized image is randomly cropped to partial view images with 512 512 resolution. For the reference set, generate 3, 000 images with resolution of 512 x 512 from the pretrained diffusion model using the same text prompts.
Implementation Details. The resolution of latent arbitrary-sized image is 64 x 384, and the resolution of an instance space sample is 64 x 64. We use deterministic DDIM 53 sampling with 50 steps.
Definition Operations. The projection operation f. corresponds to cropping a partial view of the arbitrary-sized image, which is 1-to-1 projection. The unprojection operation gi is the inverse of the fa which pastes the partial view image onto the canvas of the arbitrary-sized image.
Results. We report quantitative results in Table 6 and qualitative results in Figure I0 As mathe- matically proven in Section D the quantitative results show that all diffusion synchronization cases exhibit comparable performances, which aligns with the observations from the 1-to-1 experiment in Section 33..2] This is further supported by the qualitative results Figure TO where all cases produce identical arbitrary-sized images, indicating that any option can be used when the projection 1-to-1.
Results using Gaudi Intel-v2. Additionally, we present qualitative results of arbitrary-sized image generation using Intel Gaudi-v2 in Figure TT along with a comparison of computation times between Intel Gaudi-v2 and NVIDIA A6000 in Figure T2 We observe that Intel Gaudi-v2 achieves 1.8 to 1.9 times faster runtimes compared to the NVIDIA A6000.
21

---

"A photo city skyline at night"
Case
SyncTusadias Case
MultiDifussonn S Case
Case
photo of forest with a nisty fog"
Case
SyncTveedIes Case
Multiiiiuusinn S Case
Case
Case
Figure 10: Qualitative results of arbitrary-sized image generation. All diffusion synchronization processes generate identical results in the 1-to-1 projection.
22

---

"Deeert landecape with vaet stretchee of sand dunes"
nountain range snog"
"Cityecape
netropolie night"
Desduter (ard2t foad
Figure 12: Runtime comparison of NVIDIA RTX A6000 and Intel Gaudi- v2. We use four different width sizes for the arbitrary-sized images: (512, 1024, 2048, 3072].
Figure 11: Qualitative results of arbitrary-sized image generation using Intel Gaudi-v2. SyncTweedies (Case 2) is used for all text prompts.
D 1-to-1 Projection
is matherattcally guaranteed that Cases 1-5 become identical when the mappings are 1-to-1 and
noises are initialized by projecting from the canonical space w; (T) f((z(T)), where 2 NN I).
Note that (((0) and v.(0(,, are linear operations and commutative with other linear operation such as Ai and gs. Assume the following conditions hold:
x A((a((A(((
(10)
A(fg:(w:))) AcforUi(AA()s v(w)
(11)
Based induction, have:
(t-1)) = 20: (2 (
(Case 4)
(12)
(2 (UAA"//AAAAAAA)""))):ateeeea1110
(13)
(Case (14)
(15)
(4"" (e (f) (f)(z Eols,
(16)
(17)
= (((
(18)
where the last equality holds the induction hypothesis. This proves that Cases 4-5 are identical.
For instance variable denoising cases we have:
w; (1-1)
(w.) ( (w(") (6) FI(A(to(ccow, (1))))
(Case 1)
(19)
-6(00 (I(AA))) (CALE// (1)))
(20)
=w (o (w)) co (2)
( (()))))))
(Case 2)
(21)
(t) co
(22)
(V
(w,)) ! ( (o(w,
(8) (1)))))))
(Case 3)
(23)
-ALAAOO (WW)) (w ea (w
(24)
-ACCAClos(M) (1-1()))))
(25)
23

---

where the last equality holds the induction hypothesis. This proves that Cases 1-3 are identical. Lastly. based on the definition of the projection operation, we have:
w !1-1) = -((((!-11)
(26)
" (((((()))))
(27)
= -f(Allas (ELAtas(bh 10 (w)) co(w) co(w))))))) ()))))
(w))
(Case 3).
(28)
This proves that canonical variable denoising cases (Cases 4-5) are equivalent Case 3.
We validate the proof both qualitatively and quantitatively in applications with 1-to-1 projection: ambiguous image generation and arbitrary-size sized image generation in Section 3.4.2 and Section respectively, where all cases generate identical results.
Inpui 3D mesh
Edited 3D mesh (SyacTveedies)
nascar"
"A
graffits"
"A lanters"
Chinese style lantern"
golden
Figure 13: Qualitative results of 3D mesh texture editing, We edit the textures of the 3D meshes generated from Genies [T] using SyncTveedies.
(a) Pain-it
(b) SynTTwarkes
Figure 14: Diversity comparison. Optimization- based method Paint-it 622 (Left) and diffusion- synchronization--asse method, SyncTweedies (Right). SyncTweedies generates more diverse images.
24

---

E 3D Mesh Texture Editing and Diversity
In this section, we extend the 3D mesh texture generation from Section 5.1 and present a texture editing application, along with diversity comparison of SyncTweedies to the optimization-basee method Paint-it 622
E.1
3D Mesh Texture Editing
Despite the recent successes of 3D generation models [ 34). the textures of the generated 3D meshes often lack fine details. We utilize SyncTweedies edit the textures of the generated 3D meshes, and enhance the texture quality. Specifically, we use the 3D meshes generated from a text-to-3D model. Genie [
We follow SDEdit 36 to edit the textures of 3D meshes. We begin by adding noise at an intermediate time t' to the texture image of the 3D mesh and then perform reverse process starting from t'.
Implementation Details. We set the CFG weight 22 to 30 and t' to 0.8. For other settings, follow the 3D mesh texture generation experiment presented Section 5.1
Results. We present qualitative results 3D mesh texture editing in Figure The 3D meshes edited with SyncTweedies exhibit fine details, including graffiti the row paintings the lantern row 2, and the intricate shells the turtle in row 3.
E.2 Diversity of SyncTweedies
In Figure 14 we present qualitative results of 3D mesh texturing using the optimization-basae method (Paint-it [22 and SyncTweedies with different random seeds. SyncTweedi generates more diverse texture images compared to Paint-it.
F
Runtime and VRAM Usage Comparison
Table 7: A runtime comparison in 3D mesh texturing and 3D Gaussian splats texturing applica- tions. The best in each row highlighted by bold.
Diffusion
Finetuning.
Optim.
Iter. View Updating
Synchronization
Based
-Bsed
3D Mesh Texturing
SyacTveedies
Pain3D
Pa
TEX ure
Tos12 Tex
Runtime (minutes)
Case
1,83
2.65
21.95
1.54
13.10
Gaussian Splats Texturing
SyncTseedies
SDS
INZN
Case
10.56
85.50
37.93
Table 8: A VRAM usage comparison in 3D mesh texturing and 3D Gaussian splats texturing applications. The best in each row highlighted by bold.
Diffusion
Finctuning
Optim.
Iter. View Updating
Metric
Synchronization
-Based
-Based
3D Mesh Texturing
SyncTvoedies Case
PaintAD
Paint-it
TuXe
Text2Tex
VRAM Usage
(GiB)
6.49
9.15
28.36
10.66
10.92
3D Gaussian Splats Texturing
SyncTseodies
SDS
INZN
Case
9.30
6.87
10.38
25

---

As discussed in Section 4 one of the advantages of diffusion synchronization processes is the fast computational speed. We compare the runtime performance of SyncTweedies with optimization- based and iterative-view-uplee seepwwuudating-aase splats texturing. The quantitative results are presented in Table
methods in the 3D mesh texturing and the 3D Gaussian
In the 3D mesh texturing application. SyncTweedies shows faster running time than other baselines except TEXTure [44] which shows comparable running time. However, TEXTure [44) generates suboptimal texture outputs as observed in Table 3 and Figure The finetuning-based method Paint3D 663 shows comparable running time to SyncTweedies, but it shows inferior quality, as seen in Table B and Figure 6 Another fierative-view viriiive-iee-uudatinng based method, Text2Tex [1O]. improves quality texture image by integrating refinement module, but this comes at the cost of additional overhead in terms of running time. In contrast, SyncTweedies achieves running times that are 7 times faster than Text2Tex and even outperforms across all metrics as shown in Table 3 Lastly, SyncTweedies shows 11 times faster running time when compared to Paint-it 62] an optimization-bant based method.
In the 3D Gaussian splats texturing, SyncTweedies achieves the fastest running time. SyncTweedies is 3 times faster than the tleratie-vieew.updatin---ese 8 times faster than the optimization-basee only generates high-fidelity textures, but also excels other baselines in computational speed. We use the NVIDIA RTX A6000 for the runtime comparisons.
method IN2N [19, and
method, SDS [4]. This shows that SyncTweedies not
Additionally, in Table 8 baselines, where our SyncTweedies requires around 6-9 GiB of memory, making it suitable for most GPUs.
present results comparing the VRAM usage of SyncTweedies and the
G
User Study
We conduct user studies to evaluate the textures of the generated 3D Gaussian splats (26) through Amazon's Mechanical Turk. Following the methodology of Ritchie (45), participants were presented with input text prompts and randomly sampled output images generated by our method and the baseline methods. Participants are asked choose the most plausible image that aligns with the given text prompt. In Table 9 our results are the most preferred in the human evaluations compared to the other baselines.
Details on User Study. We conduct separate user studies comparing our method to diffusion- synchronization- based methods, optimization- based methods (SDS 41,, MVDream-SDDS (22). and wiw-iiiaeiie-uuaatiniinnn
method (IN2N [1)). For each user study, we use 20 images in shuffled order including five vigilance tasks. We collect survey responses only from participants who pass the vigilance tasks. Specifically, 94 out of 100 participants passed in the test with Case 5, 90 out of 100 passed with SDS 41. 95 out of 100 passed with MVDream-SID [52] and 92 out 100 passed with IN2N [19. Screenshots of the user study, including an example of vigilance tasks,
shown in Figure 15
Table 9: User study results in 3D Gaussian splats texturing application. SyncTweedies is the most preferred method over the baselines from human evaluators.
Baselines
Case 5
SDS 41
MVDream-SDS 52]
IN2N 19
Prefer Baseline (%)
33.56
41.33
12.21 87.79
40.05
Prefer SyncTweedies (%)
66.44
58.67
59.95
26

---

Triol ou prompt. procpp
images generated prompt, procapt
images generated provided, which of two images show better quality and aligmment with the text peonpp??
provided, which of two images show better quality and algiment with the text
peompe?
Prompt: photo white shop
Pronne: "Plasve choune of dag
imsgee
image
image
Right insage
(o) Main problem
(b) Vigilance takk
Figure 15: 3D Gaussian splats texturing user study screenshots. A screenshot of a main problem (left) and a vigilance task (right) is shown.
(1)
(1-1)
f
BSGO
6
(a)
Instance variable denoising trajectory
(b) Canonical variable denoising trajectory
(t)
(1-1)
fa
OF
gA
9+
9
w
Ef
RICO
(-1
(c) Instance variable denoising trajectory
(d) Canonical variable denoising trajectory 2
(t)
BElO
(1-13
(
(--13
9
g+
w
E#
(--
w
E6 Canonical variable denoising trajectory
(c) Instance variable denoising trajectory
(t)
(t-
(t-1)
f
g
E6
(g) Instance variable denoising trajectory
(h) Canonical variable denoising trajectory 4
Figure 16: Diagrams of diffusion synchronization processes. All feasible trajectories of the instance variable denoising process (left) and the canonical variable denoising process (right). Each row shares the same trajectory with different variables denoised.
27

---

H Analysis of Diffusion Synchronization Processes
As outlined in Section 3.4.4 we present comprehensive analysis of all possible diffusion synchro- nization processes, including the representative five diffusion synchronization processes introduced in Section 3.2 Following the main paper, we categorize diffusion synchronization processes into two types: the instance variable denoising process, where instance variables (w!") are denoised, and the canonical variable denoising process, which denoises a canonical variable 2 directly. Unlike the representative cases, other all feasible cases either take inconsistent inputs when computing Eg(:), (..)) and (..-) or conduct the aggregation A multiple times. Additionally, for more exhaustive analysis, we introduce another type of diffusion synchronization processes, named the combined variable denoising process, which denoises {w!") and ! together.
We present a total of 46 feasible cases for the instance variable denoising process, for the canonical variable denoising process, and an additional 6 representative cases for the combined variable denoising process. We provide instance variable denoising cases in Section H.2 and canonical variable denoising cases in Section H.3 Additionally, the six representative cases for the combined
variable denoising process are detailed Section H.4 We conduct quantitative comparison of all listed cases following the experiment setup outlined in Section 1 and the results are presented in
Section H.S
H.1 Overview
We provide the representative trajectories in Figure 16 where (a)-(b), (c)- (e)-(f), and (g)-(h) follow the same trajectory but differ in the denoising variable, either instance or canonical, respectively. In each denoising case, there are 2 = 4 possible trajectories determined by whether (:,)),)) and (.--) are computed in the canonical space or instance space. This is because among the three computation layers-ee(-). o( (...) and (.,)) -yly) the last two operations can be computed in both the canonical space and the instance space unlike noise prediction which is only available in the instance space. Table 10 summarizes the computation spaces of (1() (...) and (.,)),,) :). along with their corresponding trajectories.
Table 10: Computation space of each denoising trajectory. (; and (..-) can be computed in both instance space W. or canonical space Z, whereas noise prediction (-) can only be computed in the instance space.
(t.-)
(-,-)
Trijectory
Compulation space
Computation space
Trajectory
W
W.
Trajectory
W
Trajectory
Trajectory
W;
Next, we introduce an additional operator F; that synchronizes instance variables. This operator unprojects set of instance variables and averages them in the canonical space. Subsequently, the aggregated variables are reprojected to the instance space:
F((Ww))jnaa = f fi(A(19, (w,)),,,))))))
(29)
The red arrows in the diagrams of Figure 16 indiciae the potential incorporation of F, Thus, a total of 2 different cases can be derived from a trajectory marked by N red arrows, depending on whether F, is applied to each variable or not.
Lastly, review the five representative diffusion synchronization processes discussed in Section 3.2 along with two additional denoising processes: an instance variable denoising process that proceeds without synchronization (No Synchronization) and a canonical variable denoising process that
averages the outputs of eh(()) (..-) (Case 6):
No Synchronization : w (t-1) = vi(t) (w:!) ,000 (w!"" [eo(w(")))
Case (( (t-1) = xb(0) (w:! 6,00 Fi( ()))
Case 2 w (1-1) )) wi(!!(w,)) ( (w)!" Eo (w (0))))
28

---

Case 3 w/ (t-1) = (w (w(":((((((
Case ((t-1) 1) 1510 (200 ((bo(eo(((22))
Case 5 2((11) = (1000
Case 6 : ((t-1) (1 o' (fi(")))), Eo
Note that Case 3 and 6 are identical except for the initialization, which can be either {w: ( (T) For the independent instance variable denoising process (No Synchronization), synchronization is only applied at the end of the denoising process.
H.2 Instance Variable Denoising Process
Here, we explore all possible instance variable denoising processes. Here, the canonical space Z is employed to synchronize the outputs of co(-), ()(")(, .) and (1.-) in the instance spaces.
Following the trajectory shown part (a) of Figure 16 marked by five red arrows, there are a total of 2 32 possible denoising processes. This includes the independent instance variable denoising process (No Synchronization), where Fi not applied at any red arrow. Additionally, the three representative instance variable denoising processes. Cases 1-3, are also included, along with Cases 7-34 which are presented below:
Case 7 w! (((-)) (/ (w/) 200 (w (W. (o(F)((())))
Case 8 w. (1-1) 2/(1 (w:
FF(EE(FF(w (?)))(((()))))) ()))
Case 9 w.! (t-1) (w(" (F (o),(((
Case 10: w (t-1) 20(1)( (F( (
Case 11 w. (t-1) 20 (F((W (1) Fi(eo(W
Case 12 w. (t-1)
(F((W (1) F((Eo(F,
Case 13: w (t-1) = (w!
(0) Fi(o"( (1) (F(F[(
Case 14 w. (t-1) = (w!" (w! (0) Fi(co(W;
Case 15 w ((-1) w'(( (l
(1) Fi((o(Fii
Case 16 w! (1-11) ( (F
(1) co
Case 17 w (t-1) ( (F )
Case 18: w. ((-11) (w(! A(o (Fi(W ),,, (co
(
Case 19 w. (f-1) ( (0) F(o (F((w) (e
Case 20 w. (t-1) (1
Ea
Case 21: w. (t-1) (F,(WW
Fo(F((
Case 22 w, (t-1) (F,(W
F((eo(W
Case 23: w. (t-1) (F,(WW
(F(
Case 24 w (f-1) (1,
Case 25 w (t-1) (F
Case 26 w. (t-1) 1) Fe((! (w:
(((-))
Case 27 (1-) (F,( (0)
Case 28 w. (f-1) (F (T) F.F00 co (F, (w: (1)
Case 29 w(( (1-1) (()) (w,") Fe (w:
(1)
29

---

Case 30 w/ (t-1) (F(( F((e(FF((
Case 31 w; (t-1)
coll
Case 32 W, (t-1) =F((V((wW) (
Case 33 W, (t-1)
(F(( F ()))
Case 34 : w; ((-1) = Fw( F (0 (w! Fi
Similarly, four cases are derived from the trajectory 2 shown in part (c) of Figure 16 These correspond to Cases 35-38 below:
Case 35 w (f-1) (/(")((? (0) fi((")) (A(1
Case 36 w; (1-1) w/00 (,
Case 37 w!" (t-1) (3(1
Case 38 (((1)) (100 1:(0 (
The trajectory 3 shown in part (c) of Figure 16 accounts for two cases, corresponding to Cases 39-40 below:
Case 39 (((11)= S10110 (A()
((
Case 40 w. (t-1) (A(6)(((1)) (w.) (1) }}), (A(lg) ((
Lastly, the trajectory shown in part (g) of Figure 16 includes Cases 14-48 below:
Case 41 w((-1)
(1) EB (W (8) )))}))
Case 42 w. (t-1) (1
(F)
Case 43 w. (1-1) = S100000
Fi(co(
Case 44 w((-!)) fi(0 (A(9)) "4 F ((
Case 46 w. (t-1) 1 (o(
Case 45 w, (t-1)
Case 47 w. (((-1) 11(50)(((1))) (w (1) !1 ).Fcco(
Case 48 : w. (1-1)
F, (6
H.3
Canonical Variable Denoising Process
Here, we present all possible canonical variable denoising processes. Due the absence of noise prediction in the canonical space, a process first redirects canonical variable (t)) to the instance spaces where a subsequence of operations Eo(-), (,,-) and ()())) (.,-) are computed.
We exclude the application of F, to w + fi(z("), the variable remains unchanged after the operation. Therefore, applying F; to w! +- (z((0) for the inputs of cg(:), (,.(,,)) and (.,-) not considered.
Case 4 which belongs to the trajectory 3, is visualized in part (f) of Figure 16 Cases and 49 derive from the trajectory 4 which shown part (h) of Figure I6
Case 49 (t-)) (1 (z) (( F((eo(())))
In the trajectory 1, 22 cases are possible, as shown in part (b) of Figure 16 This includes Case along with Cases 50-52 below:
30

---

Case 50 z (--1)
(f Fi(e((
Case 51 : zz (t-1)
Case 52 z((-1)
(f( FFleL)
Lastly, trajectory 2, shown in part (d) of Figure T6 encompasses one possible case, corresponding to
Case 53:
Case 53 (t-1)) =
H.4 Combined Variable Denoising Process
In this section, we introduce combined variable denoising processes where both instance and canonical variables are denoised. This process synchronizes instance variables and a canonical variable by aggregating the unprojected instance variables and the canonical variable in the canonical space.
For clarity, we introduce additional operations below. (z(-) takes a variable in the canonical space 2 É 2, projects it into the instance spaces, predicts noises in those spaces, and aggregates them back in the canonical space after the unprojection. 4 (-) then computes Tweedie's formula 46 based the noise term computed by 8z()) (-).
62(2) A((g((e(ff(())))
(30)
+ 2 (z) の (z, 8(2)).
(31)
Similarly, given a set variables in the instance spaces (wi). J. the following operators aggregate the unprojected outputs of ()(()) (,, c.).) and (., in the canonical space:
(1) Wi ([w:)) = A(( (1) (w, co(w ()))
(9
(32)
6w: (1 ((wi)) = (( (H)))
(33)
w. ((wi)) = (o( co(w!")))
(34)
We present joint variable denoising cases on the representative cases discussed in Section H
Case 54 w("-!) (w(" .000 (L (( (2,
Case 55 w. (1-1) (5(1(( SAA(E ((w) ()) 1 (2(()))))
Case 56 w." f((A(E ([ ((,())), (t,(((-1))))
Case 57 (t-1) -500 (1 ((00 (1) A(10
(w:((w)) (6 ()))))
Case 58 z (t-1) (z(1) A(F 12 (1f((Z
$ W (tw, (I)
Case 59 24-1 A(( (L (() ())))
Cases 54-59 correspond to the combined variable denoising processes from Cases 1-6, respectively. In each of the above cases, highlight the terms already present in the original representative case in orange and newly added variable to be synchronized together in purple.
H.S Quantitative Results
In Table II
present the quantitative results of the 60 diffusion synchronization processes listed above. We follow the same toy experiment setup described in Section [3.41| and Section B. As outlined in Section H.T for all instance variable denoising processes, including the independent denoising case (No Synchronization) we perform the final synchronization at the end the denoising process. For n-to-1 projection, we utilize M 10 multiplane images done in Section 3.4.4
We report the quantitative results of all cases in Table TT The results align with the observations of Table I In the 1-to-1 projection scenario, most diffusion synchronization processes exhibit similar
31

---

performances. Except for Cases 55-56, the combined variable denoising processes (Cases 54-59) show suboptimal performances with FID [2]] scores over 100. This indicates that denoising either instance variables canonical variable is sufficient to produce satisfactory and consistent results.
When it comes the 1-to-n projection scenario, Cases 2 and 5 outperform the others, with some exceptions such as Cases 11 and 35. This trend is also consistent with the results in Section 334.3 highlighting the effectiveness synchronizing the outputs of Tweedie's ' formula [46] (.. even when compared to more complex diffusion synchronization processes.
Lastly, in the n-to-l projection scenario, Case (SyncTweedies) is the only one that outperforms the others across all metrics.
In conclusion, as shown in Table Case 2 (SyncTweedies) distinctly exhibits superior performance across various projection scenarios, outperforming even more convoluted cases.
Table 11: A quantitative comparison of all cases in ambiguous image generation. KID scaled by 10 For each column, we highlight the row whose value is within 95% of the best.
6 is
CLIP-A +
CLIP-C 42
FID 20
KID (
1-to-1 Projoction
Syne.
102.14
Case
30.26
64.44
85.55
31,95
Case (ByncTweedies)
30.35
Case
30.34
64.46
85.31
31.57
Case
30.28
64.47
85.37
32.44
Case
30.36
35.56
Cuse
30,3
64.49
84.75
31.29
Case
30.27
64.49
85.00 $5.03
32.08
29.40 46
62.17
91.48
44.21
Case 10
30.36
64.68
84.79
31.44
Case
30.31
64.48
81.81
32.26
Case 13
30.33
64.46
85.8 83
32.35
Case 15
30.35
64.63
85.6
32.17
Case
30.34
64.57
32.55
Case 17
30.32
64155
.666
32.3
Case 18
30.31
64.63
32.35
Case 20
29.91
63.48
92 18
38.44
30.3
Case 22
30.33
64.61
85 99
32.41
Case
30.31
64.59
85_17
31.77
Case 24
30.06
63.91
91.8 82
37.62
Case
30.3
64.46
85.41
32.22
Case 27
30.31
64.49
84.89
31.8
Case
30.35
Casc 29
30.33
64.55
13.93
32.29
Case
30.33
64.51
.3
31.72
Case
30.32
64.42
85.95
32.91
Case 32
30.33
64.5
85.77
32.35
Case 34
30.37
64.66
84.99
31.84
Case
30.36
Case 36
30.34
64.55
84_59
31.8
Case 37
30.33
64.63
21
31.84
Case 38
30.39
64.56
75
31.82
Case 39
30.31
64.55
32.67
64.55
Case 41
30.31
64.48
85.5 53
32.02
Case
30.35
64.47
Case
30.31
64.55
85.4
32.09
Case
30.3
64.51
86.
32.55
Case
30.32
64.42
.44
32.25
Case 46
30.34
64.51
85.5 59
32.67
Case 48
30.35
64,57
84.95
31.96
Case 49
30.3
64.48
.4.4
32.43
Case
30.31
64.45
86.49
32.97
Case
30.3
64.47
85.83
32.41
Case
30.35
64.62
86.05
32.44
Case 53
30.28
64.45
85.38
32.21
32

---

CLIP-A [
CLIP-C (42
FID 21
KID [6]
Case 54
21.27
50.03
422.05
491.69
Case 55
30.09
64,26
87.04
34.26
Case
30.33
26.26
Case 57
21.28
49,99
420.29
495.9
Case
26.88
60.05
116.9
60.24
Projection
No Syne.
27.64
56.97
132.6
107.41
Case
26.12
54,97
231.23
212.39
Case (ByncTweedess)
30.23
60.87
10:.73
76.25
Case
25.86
$4,29
254.74
2500.66
30.29
108.77
74.51
Case
30,0
60.55
117.64
84,84
Case
28.41
58.41
195.37
186.39
Case
25.79
54.21
271.32
278.01
Case
28.77
57.56
129.22
99.92
60.91
Case II
30.2
60.84
110.38
76.53
Case
29.92
60.8
121.93
Case 13
29.93
60.84
121.64
86.13
Case
29.3
$9.91
163.26
127.25
Case 15
29.53
60.29
158.83
140.3
Case 16
30.05
60.45
117.48
86,85 86
118.56
Case 18
30.18
60.77
113.1
79.59
Casc
30.17
60.79
117.68
85,37
Case
29.45
$9.41
119.8
87.59
Case
30.06
60.76
115.39
82.83
Case
30.02
60.58
16.08
83.42
Case 23
30.06
60.69
117.5
84.91
Case 25
29.94
60.45
118.35
86.34
Case
30.02
60.55
119.02
86.96
Case
29.92
60.45
118. 86
86.83
Case
30.01
60.52
119.1
86.91
Case
94
60.54
118.76
85.98
Case 30
29.97
60.52
120.49
89.38
29.95
Casc 32
30.02
60.62
119.03
86,73
Case
29.95
60.52
119.92
87.55
Case 34
29.96
60.54
119.57
87.29
Case 35
30.2
60.84
110.38
76.08
Case
92
60.8
121.93
86.99
Case 37
29.94
60.45
118.35
86.27
Case
30.02
119.0
Case
29.94
60.45
118.35
86.05
Case
30.02
60.55
11.002
87.07
Case
29.92
60.45
118 86
86.51
Case 42
30.01
60.52
119.1
86.46
Case
94
60.54
118.76
86.42
Case 44
29.97
60.52
120.49
89.04
119.19
Case
30.02
60.62
19.003
87.21
Case
29.95
60.52
119.92
87.91
Case
29.96
60.54
11..57
87.2
Case 49
29.32
59.98
162.41
126.53
118.17
Case 51
29.96
60.53
119.6
87.85
Case
30.0
60.62
118.17
85.66
Case
21.25
49.99
442.91
538.
Case
25.99
55.43
223.48
200.81
Case 56
25.9
55.22
229.8
210.01
Case 58
28.21
$8.01
151.01
122,97
152.2
n-to-1 Projection
No Syne.
28.26
61.75
111.11
$7.24
Cuse
21.28
49.94
405.82
496.98
Case (SyacTseedies)
29.56
63.1
96.3
40.91
Case
21.58
50.58
243.23
151.11
301.2
Case
21.09
50.04
289 389.82
213.45
22.28
50.11
329.11
299.11
Cuse
21.76
50.01
422.31
518.33
33

---

CLIP-A 1
CLIP-C C (42
FID 21
KID [6]
Case
21.72
50.0
426.69
530.85
Case
22.81
51.54
192.3
112.63
56.05
158.99
Case 11
25.67
54.72
288. 83
278.59
Case
25.11
53.61
260.91
223.48
Case
24.29
$2.09
344.55
379.52
Case 15
25.38
53.72
259.18
221.03
Case 16
27.93
58.75
198.
144.28
194.6
Case 18
25.32
53.76
315.95
329.26
Case
21.53
$0.71
236 09
157.04
Case
22.47
$2.48
189.73
104.92
Case
23.74
54.67
154.53
77.4
Case 23
22.75
53.44
174.95 95
87.28
Case 25
24.72
57.52
130.55
60.76
21.53
50.81
Case
21.53
50.4
246. 63
161.28
Case
21.44
30.9
253.72
157.43
Case
21.28
50.75
249.3
151.45
Case 30
21.58
30.84
249, 85
152.37
Case 32
21.89
$0.68
243.65
160.12
Case
22.22
117.23
Case 34
22.01
50.48
257.02
181.18
Case 35
25.72
54.78
289 98
279.62
Case
25.73
56.58
160.42
Case 37
24.79
57.46
131.86
61.08
Case 39
24.77
$7.45
33.03
$9.89
Case
21.58
50.88
212.99
122.21
Case
21.52
50.5
247.4
161.85
Case 42
21 46
50.86
253.6
157.02
Case 44
21.54
$0.85
251.18
154.49
237.56
Case 46
21.94
50.69
244.4
161.12
Case
22.27
$1.05
206.77
114.38
Case
22.05
50.47
258.46
174.1
Case 49
21.13
50.05
280.71
195. 28
Case 51
22.29
$0.04
354.31
332,85
Case 53
22.74
50.06
349.4
321.65
Case
20.77
50.06
419.46
495.55
Case
22.01
50.15
270.14
192.31
Case 56
21.8
50.1
291.59
217.05
Case 58
26.32
$8.49
124.28
66.34
26.52
108.4
34

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We discuss the claims made in the abstract and introduction throughout the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the last section of the main paper, we address the limitation of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
35

---

Justification:
All the results shown in the paper are obtained through experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide experiment setups and implementation details in the main paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
36

---

Answer: [Yes]
Justification:
The code is publicly released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeeee //nips.com htc/:/ubbb///gdddee///eeeeueisss ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide detailed experiment setups in the main paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Due our limited computational resources, unable report error
bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
37

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide runtime comparison of ours and other baselines the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This work conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss societal impacts in the last section of the main paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
38

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited all sources.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
39

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We provide detailed setups for our user study in the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
We conducted the user study with IRB approval.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "ZdCnVNUdvI", "reviewer": "Reviewer_KJxR", "summary": "This paper investigates content generation within a target space using pretrained diffusion models operating in projected subspaces. The authors analyze five variants of the DDIM procedure, performed separately in each subspace and aggregated in the target space using known projection and unprojection operators. They explore different sequences of projection and aggregation, concluding that the optimal approach is to aggregate each subspace's \"estimated x0 from xt\" (the output of Tweedie's formula).\n\nThe method is evaluated through texturing tasks involving depth, mesh, and 3DGS, each representing distinct projection and unprojection scenarios. The ablation study and comparisons against baselines demonstrate the robustness of the proposed aggregation stage.", "questions": "please see weaknesses.", "limitations": "See weaknesses -- the work in it sucrrent form is limited to known geometry", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper effectively breaks down the multi-DDIM process, clearly delineating distinct alternatives that differ in the sequencing of the projection and aggregation operations.  \n+ The identification of three different tasks, which span a range of projection and un-projection operations, enriches the analysis and allows for a clear identification of the strengths and weaknesses of each aggregation strategy. I also appreciated the toy experiments, which effectively mimic the 3D effect in a simpler 2D setup.  \n+ The final strategy advocated for in this work aligns with the standard DDIM, is straightforward to implement, requires no additional tuning, and does not incur extra computational costs.", "weaknesses": "- Missing Explanations: The figures are difficult to understand due to missing information. For example, Figure 3 does not explicitly state the experimental setup, including the definitions of the canonical space and the subspaces.\n- The formulation of cases 1-3 that operate in the subspace does not explain how the recovered signal in z is finally returned. This is related to my previous comment, where it seems Figure 3 shows individual subspace generations rather than the end goal/task.\n- The initialization process is not discussed in the main paper. It is unclear whether the subspaces share the same initial noise. If they do, how is noise being generated in z? This is particularly challenging when z is 3DGS.\n- In the appendix, it states that the 3DGS are optimized towards the predicted clean images. Is this optimization considered the unprojection operation \"g\"? If so, this should be explicitly stated and moved to the main paper. More details on this optimization process are needed, including whether it is performed until convergence at each denoising time during generation.\n- In lines 129-130, phi and psi are used in the target domain. Please explain how this is done properly, given that noise is not injected in that space. Why would the coefficients in eq (3) and (4) hold true?\n\n- The caption of Figure 1, “Diverse visual content generated by SyncTweedies,” and the abstract phrase \"generating diverse visual content\" are misleading. They suggest the generation of general 3D content, while the method, as mentioned in the limitations, relies on known mappings between the subspaces and the target space, limiting it to texturing tasks. The authors should tone down these statements.\n- If the mapping from the canonical space to the instance spaces is simple (say, linear or even the Idnetity mapping), it seems all \"cases\" perform similarly both quantitatively and qualitatively. Do these methods mathematically converge to the same method, or are there still differences? In the 1-1 projection example in Figure 3, it appears all methods produce the same result. Is this correct? Table 10A shows almost identical results for different methods, and in Table 1, the scores for the 1-1 cases are very similar.\n- The claim of being the first to propose this method and that it was previously \"overlooked\" is too bold. Guidance literature shows that guidance in \"\\hat{x_0}\" offers more stable optimization and results. While guidance differs from multi-diffusion, it is similar enough to make the analogy. The authors should discuss similarities and differences, referencing papers like \"Universal Guidance for Diffusion Models\" and motion diffusion papers such as MDM and Trace&Pace. Further, if in degenerate cases of f and g maps (e.g. if they are the identity function) some \"cases\" are indistinguishable, be mindful when attributing aggregation choices to these methods (e.g., MultiDiffusion). Methods requiring tuning, like DiffCollage, should not be dismissed as baselines. The main contribution of this paper is identifying the best \"place\" to aggregate, and tuning is somewhat orthogonal. If \"Case 2\" improves DiffCollage, this should be demonstrated.\n- The term \"1:1 case\" is confusing and suggests disjoint individual subspaces.\n- The integration process of 1:n should also be better explained. It may also be useful to add illustration figures. \n- Improve Figure 2 for clarity with a more detailed caption. Connecting the diagram to the notations introduced in the paper would also help readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your in-depth reviews and feedback. We find that some of the reviews have been addressed in the appendix but are missing in the main paper. Please understand that these details were moved to the appendix due to page limits. However, we agree that providing more details in the main paper will help in understanding our work, and we will move these details to the main paper for clarity.\n\n**Please refer to the global rebuttal for our answers to \"convergence to the same method in $1$-to-$1$ projection\" and \"comparison with finetuning-based methods.\"**\n\nFigure 3 experimental setup.\n---\nDue to page limits, we detailed the experimental setup of ambiguous image generation in L559-583 of the appendix. Ambiguous image generation is a special case where both canonical and instance spaces take the form of images with the same dimensionality. Here, we consider one image as the canonical space, and instance space samples are obtained by applying transformations to the canonical space sample. We acknowledge that Figure 3 was not presented with sufficient details. We will move the experimental setup details from the appendix to the main paper in the final version.\n\nHow $\\mathbf{z}$ is returned in Cases 1-3?\n---\nAs discussed in L529-531, instance variable denoising processes, including Cases 1-3, obtain the final canonical sample $\\mathbf{z}^{(0)}$ by aggregating fully denoised instance samples ${\\mathbf{w}^{(0)}_i}$ in RGB space.\n\nDetails on noise initialization.\n---\nThe details of the initialization scheme are provided in L517-521. For all experiments except 3DGS texturing, instance space samples are initialized by projecting the initial canonical space variable to each view, $\\mathbf{w}_i = f_i(\\mathbf{z}^{(T)})$, where $\\mathbf{z}^{(T)} \\sim \\mathcal{N}(\\mathbf{0}, \\mathbf{I})$. For 3DGS texturing, instance variable denoising cases initialize noises directly in the instance space to prevent the variance decrease issue discussed in the paper. On the other hand, canonical denoising cases have no other option but to initialize noise in the canonical space $\\mathbf{z}^{(T)} \\sim \\mathcal{N}(\\mathbf{0}, \\mathbf{I})$, which causes a variance decrease issue when projecting to the instance space. We will move the details from the appendix to the main text.\n\nIn 3DGS, is optimization considered as the unprojection operation?\n---\nYes, more precisely, a combination of unprojection and aggregation functions\u001cby optimization as outlined in L580-581. As discussed in L661-663, we only optimize the colors of 3D Gaussians while fixing the other parameters, such as opacity, positions, and covariance matrices. The optimization runs for 2,000 iterations with a learning rate of 0.025. We will include this further detail in the paper. \n\nHow Equations (3) and (4) can be used in the target space? Why would the coefficients in Equations (3) and (4) hold true?\n---\nAs shown in the \"convergence to the same method in $1$-to-$1$ projection\" paragraph in the global rebuttal, when the mappings are $1$-to-$1$, all cases become identical, and canonical variable denoising processes follow the exact same trajectory as the instance variable denoising processes. Hence the coefficients defined by the instance space diffusion model can also be used in Cases 4-5. However, as discussed in Sections 3.4.2 and 3.4.3, the same may not hold when the mappings are $1$-to-$n$, as this may violate Equations R1 and R2 in the global rebuttal. Additionally, $n$-to-$1$ mappings result in a variance decrease when projecting the canonical space sample, causing performance degradation in canonical variable denoising cases. For clarity, we will add these details in our main paper.\n\nFigure 1 caption.\n---\nThe phrase \"diverse visual content\" refers to the ambiguous images, arbitrary-sized images, depth-to-360-panoramas, and textures on mesh and 3DGS that SyncTweedies produces, as shown in Figures 1, 3, and A7. We will adjust the phrase accordingly to better convey the capabilities and limitations of ours.\n\nFirst to propose this method and that it was previously overlooked is too bold.\n---\nThe previous works mentioned by the reviewer also utilize $\\hat{\\mathbf{x}}^{(0)}$, similar to $\\texttt{SyncTweedies}$. However, we would like to clarify that the phrases \"first to propose\" (L40, L62) and \"unexplored approach\" (L64) in the main paper are meant to claim the first work to present a generalized diffusion synchronization framework and instance variable denoising process that averages the outputs of Tweedie's formula, not the first work utilizing the outputs of Tweedie's formula $\\hat{\\mathbf{x}}^{(0)}$. Nevertheless, we find that the related works utilizing $\\hat{\\mathbf{x}}^{(0)}$ will make our paper more comprehensive, and we will revise our related work accordingly.\n\nThe term \"1:1 case\" is confusing and suggests disjoint individual subspaces.\n---\nPlease note that the instance spaces are not disjoint spaces. The instance spaces are initialized with overlapping regions, and the term $1$-to-$1$ is used to indicate the property of one canonical space pixel mapping to one instance space pixel. We acknowledge that the term $1$-to-$1$ may sound confusing, and we will consider revising the term to better indicate the property such as pixel-aligned mapping.\n\nThe integration process of 1:n should also be better explained.\n---\nWe will add more details to the aggregation process of $1$-to-$n$ projection with an illustration figure. In $1$-to-$n$ projection, each instance space sample is unprojected to the canonical space resulting $N$ unprojected samples, $\\\\{g_i(\\mathbf{w}^{(t)}) \\\\} _{i=1}^N $. The canonical space sample $\\mathbf{z}^{(t)}$ is then obtained by aggregating the unprojected samples using a weighted sum, where the weights can be either a scalar $1/N$ or determined based on the visibility from each view $\\\\{ w_i \\\\} _{i=1}^N$. \n\nFigure 2 caption.\n---\nWe will add more details to the caption of Figure 2 with explanations of the notations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DfXH71snhJ", "reviewer": "Reviewer_TnCt", "summary": "This paper introduces SyncTweedies, a novel framework to generate diverse visual content such as ambiguous images, panoramas, mesh textures, and Gaussian splat textures. The method uses a synchronization process that averages outputs of Tweedie's formula across multiple instance spaces, eliminating the need for fine-tuning. The authors claim their method preserves the rich priors from large-scale datasets, enhancing generalizability. Experimental results show that SyncTweedies outperforms existing methods in various applications.", "questions": "(1) How does SyncTweedies handle scenarios where projection and unprojection functions between instance spaces and the canonical space are not perfectly invertible? What impact does this have on the quality of generated content?  \n\n(2) I am also wondering if the synchronization process can be optimized further to reduce computational overhead without compromising the quality of the generated content.", "limitations": "1. The computational complexity and scalability of SyncTweedies are not thoroughly discussed (Lines 271-290). The synchronization process involves multiple denoising steps and averaging operations, which can be computationally intensive. There is no detailed analysis of the time and space complexity of the method, making it challenging to evaluate its feasibility for large-scale or real-time applications.    \n\n2. The paper assumes a level of consistency and stability across instance spaces, which may not always be realistic (Lines 202-214). For example, in 3D mesh texturing, different views can have significantly different appearances due to factors like lighting, occlusion, and perspective distortion. SyncTweedies does not provide a thorough analysis of how it handles such variability. Robustness under varying conditions is crucial for practical applications", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) From my point of view, the framework's versatility is impressive, as it addresses various visual content generation tasks. This general applicability is a significant advantage over more specialized methods.  The use of Tweedie's formula for synchronizing diffusion processes across different instance spaces is genuinely novel. This methodological innovation could inspire new directions in the field of generative models. \n\n(2) SyncTweedies' ability to function without fine-tuning is a strong point. This feature ensures that the model retains its generalization capabilities, making it effective on diverse and previously unseen datasets. \n\n(3) Besides, the authors have conducted extensive experiments, comparing their method against several state-of-the-art techniques. These comparisons convincingly demonstrate the superior performance of SyncTweedies across multiple tasks.", "weaknesses": "(1) The method assumes that the projection and unprojection functions are accurate, but in practice, these operations can introduce errors, especially in complex transformations such as those required for 3D mesh texturing. The author does not address how SyncTweedies handles these projection errors, which can accumulate and degrade the quality of the generated content.   \n\n(2) The theoretical basis of SyncTweedies seems to implicitly assume that the data distributions in the instance and canonical spaces are well-aligned. However, this assumption may not hold in practice, especially when dealing with diverse and complex datasets. The paper does not explore the theoretical implications of misaligned data distributions and how they might affect the performance and stability of the synchronization process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our work such as \"the framework's versatility is impressive\", and \"the methodological innovation could inspire new directions in the field of generative models.\" Below, we answer your insightful comments.\n\nHow does $\\texttt{SyncTweedies}$ handle scenarios where mappings are not invertible?\n---\nWe would like to emphasize that one of our contributions is the analysis of diffusion synchronization cases with various mapping functions: $1$-to-$1$, $1$-to-$n$, and $n$-to-$1$ projections (Section 3.4.1, Section 3.4.2, and 3.4.3). The $1$-to-$n$ and $n$-to-$1$ projections correspond to cases where the mappings are not invertible. Our comprehensive analysis demonstrates that $\\texttt{SyncTweedies}$ outperforms other methods across all mapping scenarios, even when the mappings are not invertible. Additionally, we provide results of applications corresponding to each mapping scenario and their analysis: $1$-to-$1$ projection (arbitrary-sized image generation, Section A3), $1$-to-$n$ projection (3D mesh texturing, Section 5.1 and depth-to-360-panorama, Section 5.2), and $n$-to-$1$ projection (3DGS texturing, Section 5.3). \n\nAssumption of aligned distributions between canonical and instance spaces.\n---\nThank you for providing insightful feedback. As the reviewer has mentioned, we agree that there might be a misalignment of data distributions between instance and canonical spaces. For example, in 3D mesh texturing, the data distribution of the pretrained image diffusion model in the instance space is more diverse than the distribution of rendered 3D mesh images. Even with potential distribution misalignment, we demonstrate that $\\texttt{SyncTweedies}$ can be applied to diverse applications, outperforming both optimization-based and finetuning-based approaches. In future work, we plan to provide a theoretical analysis of misaligned data distributions.\n\nOptimizing synchronization process to reduce computational overhead without compromising the quality.\n---\nWhile the synchronization process in $1$-to-$1$ and $1$-to-$n$ projection cases introduces some computational overhead, the majority of the computational overhead comes from predicting noise for multiple instance space samples. Similar to previous work such as MVDream and MVDiffusion, in applications where the canonical space sample is expressed as a composition of instance space samples, noise prediction for multiple instance space samples is inevitable. One possible direction to reduce computational overhead would be to use fewer views without sacrificing output quality, which we leave for future work.\n\nTime and space complexity of $\\texttt{SyncTweedies}$.\n---\nPlease see Section A5 in the appendix for a time complexity comparison, where $\\texttt{SyncTweedies}$ demonstrates an 8-11 times faster running time compared to optimization-based methods and a comparable or 3-7 times faster running time compared to iterative-view-update-based methods. In Table R1 of **the attached PDF file**, we include results comparing the space complexity of other baselines and $\\texttt{SyncTweedies}$, where our method takes around 6-9 GiB of memory, making it suitable for most GPUs. We use NVIDIA RTX A6000 for computation comparisons.\n\nAnalysis of how $\\texttt{SyncTweedies}$ handles variability across views.\n---\nWe agree that introducing variability across views is an important factor. We demonstrate that our method can control variations across views by using varying prompts. In Figure R1 of **the attached PDF file**, we provide qualitative results of arbitrary-sized images generation using a uniform prompt and varying multiple prompts across views. The result shows that our method can generate canonical space samples with variations across views.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MQUxmla1rb", "reviewer": "Reviewer_XFCg", "summary": "SyncTweedie attempts to elucidate the design space for synchronized diffusion. Synchronized diffusion means to optimize some representation (e.g., 3D mesh) by jointly diffusing its lower-dimensional projections (e.g., 2D image). The paper analyzes existing synchronized diffusion approaches on different tasks under the same umbrella on a diagnostic benchmark, and provides an empirical insight that one underexplored design is the best choice. The paper then verifies the insight in a range of tasks, including panorama image generation and 3D texturing.", "questions": "- The paper mentions that the framework cannot jointly optimize geometry and texture for 3D generation, could the authors elaborate more on why there is such a limitation? Does it suggest that the aggregation function $A$ cannot be too under-constrained?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is enjoyable to read. The analysis of the 5 categories of diffusion is very clear. The 3 types of tasks (1-1, 1-N, N-1) are also nicely defined. There have indeed been extensive studies on different tasks based on a similar idea, this paper has done a nice job of reviewing them and elucidating the design space.\n2. The toy experiment is well-designed. It is a simple task, and the authors slightly modify it so it comprehensively covers all challenges in all types of tasks. The toy experiment is also diagnostic enough to provide empirical insights, which makes Case 2 (SyncTweedies) stands out.\n3. The experiments cover a range of tasks including arbitrary-sized image generation, panorama image generation, mesh texturing, and 3DGS texturing. I find these tasks sufficient to support the generalizability of the proposed approach.", "weaknesses": "1. The depth-to-360$^\\circ$ panorama result is blurry. In the meantime, the generated arbitrary-sized image seems to be fine. Is there any reason for that?\n2. It would be better if generation diversity could be discussed, which I think would be an advantage over SDS-based works. How would different noise initialization affect the generated results and how robust is the approach given different random seeds?\n3. Some implementation details are missing. \n     - The paper seems to miss the implementation details of the aggregation function for different tasks. For example, how is the view aggregation for 3DGS performed? Is it a gradient-based optimization or an analytical weighted sum as in MultiDiffusion[4]? This will also help to understand the runtime of the method.\n     - What are the number of projected views (i.e., $N$) for each task? How are the views sampled?\n\nMinor:\n- L95: tdeterministic -> deterministic\n- L594 refers to something in L628.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our work \"enjoyable to read, and the analysis is very clear, elucidating the design space.\" We will correct the typos accordingly.\n\nBlurry outputs of depth-to-360-panorama.\n---\n\nDepth-to-360-panorama involves Equirectangular projection which introduces more distortions than simple cropping in arbitrary-sized image generation task. We believe this can be mitigated using more precise mapping functions, which we leave for future work. \n\nDiversity of $\\texttt{SyncTweedies}$.\n---\nThank you for pointing out the advantage that the diversity of $\\texttt{SyncTweedies}$ can produce. In Figure R2 of **the attached PDF file**, we present qualitative results of 3D mesh texturing using the optimization-based method (Paint-it) and $\\texttt{SyncTweedies}$ with different random seeds. $\\texttt{SyncTweedies}$ generates more diverse texture images compared to Paint-it. We will add more results in the revised version. \n\nHow is the view aggregation for 3DGS performed?\n---\nIn 3DGS texturing, optimization functions as a combination of unprojection and aggregation. As discussed in L661-663, we only optimize the colors of 3D Gaussians while fixing the other parameters, such as opacity, positions, and covariance matrices. The optimization runs for 2,000 iterations with a learning rate of 0.025. We will include the details of optimization in the revised version. \n\nHow are the views sampled and what are the number of projected views $N$ for each task?\n---\nIn both $1$-to-$1$ and $1$-to-$n$ projections, the canonical space is represented as a single slab ($N=1$). For the $n$-to-$1$ projection experiment, however, the canonical space is represented by multiple slabs ($N=10$) to simulate the weighted sum rendering process.\n\nAs outlined in L587-593 of Section A2.2, in 3D mesh texturing, we use eight views sampled around the object at $45^\\circ$ azimuth intervals at $0^\\circ$ elevation, along with two additional views at $0^\\circ$ and $180^\\circ$ azimuths with $30^\\circ$ elevation. For 3DGS texturing, we use 50 views sampled from the Synthetic NeRF dataset, as described in L645-648 of Section A2.4.\n\nWhy $\\texttt{SyncTweedies}$ cannot optimize geometry and texture jointly?\n---\nOur framework focuses on cases where projection and unprojection mappings are defined between the canonical space and the instance spaces. When the geometry is not given, the mappings cannot be defined, limiting the application of our method. In future work, initiating the generation process with a coarse geometry provided by an off-the-shelf network and jointly updating the geometry and appearances would be an interesting direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers for their constructive and insightful comments. Some comments are addressed in the global rebuttal due to length constraints.

***Please refer to the attached PDF file for qualitative and quantitative results.***

[KJxR] Do cases mathematically converge to the same method when the mappings are $1$-to-$1$ projection?
---
Yes, it is mathematically guaranteed that Case 1-5 become identical when the mappings are $1$-to-$1$ and noises are initialized by projecting from the canonical space $\mathbf{w}_i^{(T)} = f_i(\mathbf{z}^{(T)})$, where $\mathbf{z}^{(T)} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$. Note that $\phi^{(t)}(\cdot, \cdot)$ and $\psi^{(t)}(\cdot, \cdot)$ are linear operations and commutative with other linear operation such as $f_i, \mathcal{A}$, and $g_i$. Assume the following conditions hold:

\\begin{align*}
    \\mathbf{z}^{(T)} &= \\mathcal{A}( \\{g_i( f_i(\\mathbf{z}^{(T)}))\\}), && \\text{(R1)} \\\\
    \\mathcal{A}(\\{g_i(\\mathbf{w}_i)\\}) &= \\mathcal{A}(\\{ g_i(f_i(\\mathcal{A}(\\{ g_j(\\mathbf{w}_j) \\}))) \\}) \\quad \\forall \\{\\mathbf{w}\\} _{i=1}^N. && \\text{(R2)}
\\end{align*}

Based on induction, we have:
\\begin{align*}
    \\mathbf{z}^{(t-1)}
    &= \\psi^{(t)}(\\mathbf{z}^{(t)}, \\phi^{(t)}(\\mathbf{z}^{(t)}, \\mathcal{A}( \\{ g_i(\\epsilon_\\theta(f_i((\\mathbf{z}^{(t)})))) \\}))) && \\text{(Case 4)}\\\\
    &= \\psi^{(t)}(\\mathbf{z}^{(t)}, \\phi^{(t)}(\\mathcal{A}( \\{ g_i( f_i(\\mathbf{z}^{(t)})) \\}), \\mathcal{A}( \\{ g_i(\\epsilon_\\theta(f_i(\\mathbf{z}^{(t)}))) \\}))) && \\\\
    &= \\psi^{(t)}(\\mathbf{z}^{(t)}, \\mathcal{A}( \\{ g_i(\\phi^{(t)}(f_i(\\mathbf{z}^{(t)}), \\epsilon_\\theta( f_i (\\mathbf{z}^{(t)})))) \\})) && \\text{(Case 5)}\\\\
    &= \\psi^{(t)}(\\mathcal{A}( \\{g_i( f_i(\\mathbf{z}^{(t)})) \\}), \\mathcal{A}(\\{ g_i( \\phi^{(t)}( f_i(\\mathbf{z}^{(t)}), \\epsilon_\\theta(f_i(\\mathbf{z}^{(t)})))) \\} )) && \\\\
    &= \\mathcal{A}( \\{ g_i(\\psi^{(t)}(f_i(\\mathbf{z}^{(t)}), \\phi^{(t)}( f_i(\\mathbf{z}^{(t)}), \\epsilon_\\theta(f_i(\\mathbf{z}^{(t)}))))) \\}) \\\\
    &= \\mathcal{A}(\\{ g_i( f_i( \\mathcal{A}(\\{ g_j(\\psi^{(t)}(f_j(\\mathbf{z}^{(t)}), \\phi^{(t)}(f_j(\\mathbf{z}^{(t)}), \\epsilon_\\theta(f_j(\\mathbf{z}^{(t)}))))) \\})))\\}) \\\\
    &= \\mathcal{A}(\\{ g_i( f_i(\\mathbf{z}^{(t-1)}))\\}), 
\\end{align*}
where the last equality holds the induction hypothesis. This proves that Cases 4-5 are identical.
For instance variable denoising cases we have:
\\begin{align}
\\mathbf{w}^{(t-1)}_i &= \\psi^{(t)}(\\mathbf{w}^{(t)}_i, \\phi^{(t)}(\\mathbf{w} ^{(t)}_i, f_i( \\mathcal{A}( \\{g_j(\\epsilon _\\theta(\\mathbf{w}^{(t)}_j))\\} )))) && \\text{(Case 1)}\\\\
&= \\psi^{(t)}(\\mathbf{w}^{(t)}_i, \\phi^{(t)}( f_i( \\mathcal{A}( \\{ g_j( \\mathbf{w}^{(t)}_j) \\})), f_i( \\mathcal{A}( \\{ g_j( \\epsilon _\\theta(\\mathbf{w}^{(t)}_j)) \\} )))) && \\\\
&= \\psi^{(t)}(\\mathbf{w}^{(t)}_i, f_i( \\mathcal{A}( \\{ g_j(\\phi^{(t)}(\\mathbf{w}^{(t)}_j, \\epsilon _\\theta(\\mathbf{w}^{(t)}_j))) \\} ))) && \\text{(Case 2)}\\\\
&= \\psi^{(t)}( f_i( \\mathcal{A}( \\{ g_j( \\mathbf{w}^{(t)}_j ) \\})), f_i( \\mathcal{A}( \\{ g_j(\\phi^{(t)}(\\mathbf{w}^{(t)}_j, \\epsilon _\\theta(\\mathbf{w}^{(t)}_j))) \\}))) && \\\\
&= f_i( \\mathcal{A}( \\{ g_j(\\psi^{(t)}(\\mathbf{w}^{(t)}_j, \\phi^{(t)}(\\mathbf{w}^{(t)}_j, \\epsilon _\\theta(\\mathbf{w}^{(t)}_j)))) \\} )) && \\text{(Case 3)} \\\\
&= f_i( \\mathcal{A}( \\{ g_j( f_j( \\mathcal{A}( \\{ g_k (\\psi^{(t)}(\\mathbf{w}^{(t)}_k, \\phi^{(t)}(\\mathbf{w}^{(t)}_k, \\epsilon _\\theta(\\mathbf{w}^{(t)}_k)))) \\}))) \\})) \\\\
&= f_i( \\mathcal{A}( \\{ g_j(\\mathbf{w}_j^{(t-1)}) \\})),
\\end{align}
where the last equality holds the induction hypothesis. This proves that Cases 1-3 are identical.

We validate our observations both qualitatively and quantitatively in our $1$-to-$1$ projection toy experiment and arbitrary-sized image generation application in Section 3.4 and Section A3, respectively, where all cases generate identical results. However, the same may not hold when the mappings are $1$-to-$n$ or $n$-to-$1$. We will add these details in the revised version. 

[KJxR] Methods requiring tuning should not be dismissed as baselines.
---
Even though a comparison with DiffCollage is not included since the code is not publicly released, we provide a comparison with other finetuning-based methods, such as Paint3D in 3D mesh texturing (Table 3) and MVDiffusion in depth-to-360-panorama generation tasks (Table 4) in the main paper. As discussed in L286-288 and L300-301 of the paper, $\texttt{SyncTweedies}$ outperforms the finetuned models since they are trained on a relatively small-scale dataset compared to the pretrained image diffusion models, which limits their applicability to unseen scenes. Additionally, based on the reviewer's comment, we experimented applying $\texttt{SyncTweedies}$ on a finetuned model, MVDiffusion. Qualitative results are presented in Figure R3 of **the attached PDF file**. We observe that a finetuned model with $\texttt{SyncTweedies}$ synchronization still fails to generalize to unseen domains, resulting in consistently inferior performance.

Meta-review: All reviewers recognized the quality of the work : 
- "The identification of three different tasks [...] enriches the analysis and allows for a clear identification of the strengths and weaknesses of each aggregation strategy."
- "the framework's versatility is impressive"
- "This methodological innovation could inspire new directions in the field of generative models."
- "extensive experiments"
- "The paper is enjoyable to read"
- "The toy experiment is well-designed."
- "The experiments cover a range of tasks..."

Several concerns have been raised by the reviewers but the discussion period has resolved those and all reviewers have positively judged the authors response.
Decision: Accept (poster)