SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Moreover, we notice that similar observations were proposed in Delta Denoising Score (DDS) [52], where they introduced an image-to-image translation method that is based on SDS, and the difference of the noise estimate from target prompt and that from source prompt are used. Our CSD can be combined with DDS by changing the noise difference term as follows:
$$
\Delta \boldsymbol {\mathcal {E}} _ {t} = \epsilon_ {\phi} (\mathbf {x} _ {t}; y _ {\mathrm {t g t}}, t) - \epsilon_ {\phi} (\tilde {\mathbf {x}} _ {t}; y _ {\mathrm {s r c}}, t),
$$
where  $\mathbf{x}$  and  $\tilde{\mathbf{x}}$  are target and source images,  $y_{\mathrm{tgt}}$  and  $y_{\mathrm{src}}$  are target and source prompts. However, we found that CSD-Edit with InstructPix2Pix is more amenable in editing real images as it does not require a source prompt. Finally, we remark that CSD-Edit can be applied to various text-to-image diffusion models such as ControlNet [17], which we leave for future work.
Setup. For the experiments with CSD-Edit, we use the publicly available pre-trained model of Instruct-Pix2Pix [14] by default. We perform CSD-Edit optimization on the output space of Stable Diffusion [4] autoencoder. We use SGD optimizer with step learning rate decay, without adding weight decay. We set  $t_{\mathrm{min}} = 0.2$  and  $t_{\mathrm{max}} = 0.5$ , where original SDS optimization for DreamFusion used  $t_{\mathrm{min}} = 0.2$  and  $t_{\mathrm{max}} = 0.98$ . This is because we do not generally require a large scale of noise in editing. We use the guidance scale  $\omega_y \in [3.0, 15.0]$  and image guidance scale  $\omega_s \in [1.5, 5.0]$ . We find that our approach is less sensitive to the choice of image guidance scale, yet a smaller image guidance scale is more sensitive to editing. All experiments are conducted on AMD EPYC 7V13 64-Core Processor and a single NVIDIA A100 80GB. Throughout the experiments, we use OpenCLIP [56] ViT-bigG-14 model for evaluation.
B.1 Panorama image editing
To edit a panorama image, we first encode into the Stable Diffusion latent space (i.e., downscale by 8), then use a stride size of 16 to obtain multiple patches. Then we select a  $B$  batch of patches to perform CSD-Edit. Note that we perform CSD-Edit and then normalize by the number of appearances as mentioned in Section 3.3. Note that our approach performs well even without using small batch size, e.g., for an image of resolution  $1920 \times 512$ , there are 12 patches and we use  $B = 4$ .
For experiments, we collect 32 panorama images and conduct 5 artistic stylizations: "turn into Van Gogh style painting", "turn into Pablo Picasso style painting", "turn into Andy Warhol style painting", "turn into oriental style painting", and "turn into Salvador Dali style painting". We use learning rate of 2.0 and image guidance scale of 1.5, and vary the guidance scale from 3.0 to 10.0.
B.2 Video editing
We edit video sequences in DAVIS 2017 [33] by sampling 24 frames at the resolution of  $1920 \times 1080$  from each sequence. Then, we resize all frames into  $512 \times 512$  resolution and encode all frames each using Stable Diffusion. We use learning rate  $[0.25, 2]$  and optimize them for  $[200, 500]$  iterations.
B.3 3D scene editing
Following Instruct-NeRF2NeRF [39], we first pretrain NeRF using the `nerfacto` model from NeRFS-tudio [57], training it for 30,000 steps. Next, we re-initialize the optimizer and finetune the pre-trained NeRF model with edited train views. In contrast to Instruct-NeRF2NeRF, which edits one train view with Instruct-Pix2Pix after every 10 steps of update, we edit a batch of train views (batch size of 16) with CSD-Edit after every 2000 steps of update. The batch is randomly selected among the train views without replacement.
<https://github.com/timothybrooks/instruct-pix2pix
C More experimental results
C.1 Compositional editing
Recent works have shown the ability of text-to-image diffusion models in compositional generation of images handling multiple prompts [58, 59]. Here, we show that CSD-Edit can extend this ability to compositional editing, even at panorama-scale images which require a particular ability to maintain far-range consistency. Specifically, we demonstrate that one can edit a panorama image to follow different prompts on different regions while keeping the overall context uncorrupted.
Given multiple textual prompts  $\{y_{k}\}_{k = 1}^{K}$ , the compositional noise estimate is given by
$$
\boldsymbol {\epsilon} _ {\phi} (\mathbf {x} _ {t}; \{y _ {k} \} _ {k = 1} ^ {K}, t) = \sum_ {k = 1} ^ {K} \alpha_ {k} \boldsymbol {\epsilon} _ {\phi} ^ {\omega} (\mathbf {x} _ {t}; y _ {k}, t),
$$
where  $\alpha_{k}$  are hyperparameters that regularize the effect of each prompt. When applying compositional generation to the panorama image editing, the challenge lies in obtaining image that is smooth and natural within the region where the different prompts are applied. To that end, for each patch of an image, we set  $\alpha_{k}$  to be the area of the overlapping region between the patch and region where prompt  $y_{k}$  is applied. Also, we normalize to assure  $\sum_{k} \alpha_{k} = 1$ . In Figure 10, we illustrate some examples on compositional editing of a panorama image. For instance, given an image, one can change into different weathers, different seasons, or different painting styles without leaving artifacts that hinder the spatial consistency of an image.
C.2 Text-to-3D generation with CSD
We explore the effectiveness of CSD in text-to-3D generation tasks following DreamFusion [26]. We train a coordinate MLP-based NeRF architecture from scratch using text-to-image diffusion models. Since the pixel-space diffusion model that DreamFusion used [26] is not publicly available, we used an open-source implementation of pixel-space text-to-image diffusion model. Given a set of text prompts, we run both DreamFusion and DreamFusion with CSD with a fixed seed. Our experiments in this section are based on Stable-DreamFusion [60], a public re-implementation of DreamFusion, given that currently the official implementation of DreamFusion is not available on public.
Setup. We use vanilla MLP based NeRF architecture [27] with 5 ResNet [61] blocks. Other regularizers such as shading, camera and light sampling are set as default in [60]. We use view-dependent prompting given the sampled azimuth angle and interpolate by the text embeddings. We use Adan [62] optimizer with learning rate warmup over 2000 steps from  $10^{-9}$  to  $2 \times 10^{-3}$  followed by cosine decay down to  $10^{-6}$ . We use batch size of 4 and optimize for 10000 steps in total, where most of the case sufficiently converged at 7000 to 8000 steps. For the base text-to-image diffusion model, we adopt DeepFloyd-IF-XL-v1.0 since we found it way better than the default choice of Stable Diffusion in a qualitative manner. While the original DreamFusion [26] used guidance scale of 100 for their experiments, we find that guidance scale of 20 works well for DeepFloyd. We selected 30 prompts used in DreamFusion gallery<sup>4</sup> and compare their generation results via DreamFusion from the standard SDS and those from our proposed CSD. We use one A100 (80GB) GPU for each experiment, and it takes  $\sim 5$  hours to conduct one experiment. For CSD implementation, we use LPIPS [42] as a distance of RBF kernel. Note that LPIPS gives more computational cost than the usual  $\ell_2$ -norm based RBF kernel. The LPIPS is computed between two rendered views of size  $64 \times 64$ . For the kernel bandwidth, we use  $h = \frac{\mathrm{med}^2}{\log B}$ , where  $\mathrm{med}$  is a median of the pairwise LPIPS distance between the views,  $B$  is the batch size.
For evaluation, we render the scene at the elevation at 30 degree and capture at every 30 degree of azimuth angle. Then we compute the CLIP image-text similarity between the rendered views and input prompts. We measure similarities for both textured views (RGB) and textureless depth views (Depth). We also report Frechet Inception Distance (FID) between the RGB images and ImageNet validation dataset to evaluate the quality and diversity of rendered images compared to natural images.
<sup>3</sup>https://github.com/deep-floyd/IF
<sup>4</sup>https://dreamfusion3d.github.io/gallery.html
Table 5: Text-to-3D. Quantitative comparison between CSD and SDS under on text-to-3D generation via DreamFusion [26]
CLIP Similarity Color ↑	CLIP Similarity Geo ↑	FID ↓
SDS [26]	0.437	0.322	259.4
CSD (Ours)	0.447	0.345	247.1
Results. In Table 5, we report the evaluation results of CSD on text-to-3D generation comparison to DreamFusion. Remark that CSD presents better CLIP image-text similarities in both RGB and Depth views. Also, CSD achieves lower FID score showing its better quality on generated samples. Since we used the same random seed in generating both CSD and DreamFusion, the shapes and colors are similar. However, the results show that CSD obtains finer details in its generations.
In Figure 14, we qualitatively compare the baseline DreamFusion (SDS) and ours. We empirically observe three advantages of using CSD over SDS. First, CSD provides better quality compared to SDS. SDS often suffers from Janus problem, where multiple faces appear in a 3D object. We found that CSD often resolves Janus problem by showing consistent information during training. See the first row of Figure 14. Second, CSD can give us better fine-detailed quality. The inconsistent score distillation often gives us blurry artifact or undesirable features left in the 3D object. CSD can handle this problem and results in higher-quality generation, e.g., Figure 14 second row. Lastly, CSD can be used for improving diversity. One problem of DreamFusion, as acclaimed by the authors, is that it lacks sample diversity. Thus, it often relies on changing random seeds, but it largely alters the output. On the other hand, we show that CSD can obtain alternative sample with only small details changed, e.g., Figure 14 third row. Even when SDS is successful, CSD can be used in generating diverse sample.
C.3 User study
Given the subjective nature of editing tasks, we additionally conduct subjective user studies, where we ask three questions in evaluating the editing quality of both CSD-Edit and baselines: the consistency of the edited results, frame-wise image instruction fidelity, and the editing quality. For each of the three studies, we asked 20 subjects to rank different methods. As shown in Table 6 and Table 7, our method, CSD-Edit, consistently outperforms other baselines achieving the best user preferences across all three aspects.
Table 6: User study on video editing.
Temporal Consistency ↓	Instruction Fidelity ↓	Editing Quality ↓
FateZero [22]	2.37	2.05	2.12
Pix2Video [25]	2.36	2.35	2.28
CSD-Edit (Ours)	1.27	1.6	1.6
Table 7: User study on 3D scene editing.
Temporal Consistency ↓	Instruction Fidelity ↓	Editing Quality ↓
IN2N [14]	1.61	1.69	1.71
CSD-Edit (Ours)	1.39	1.31	1.29
C.4 Computation time comparison
Regarding the computational efficiency of our method, we measure the computation times and compare them with the baselines of each task. All these evaluations were conducted on a single NVIDIA A100 80GB and AMD EPYC 7V13 64-Core Processor.
For panorama image editing experiments, we compare our method with the baseline where applying Instruct-Pix2Pix to patches cropped with overlapping (InstructPix2Pix + Overlapping Patches). Note that the computation time of both methods depends on the input image resolution, whereas we show that our method becomes more efficient as the resolution goes higher. Table 8 shows how the computation time (total time in seconds) differs by the size of the input image. Here, the baseline method requires computing noise estimates of every patch at each diffusion step, while our method only requires computing a minibatch of patches per iteration.
Table 8: Computation time comparison on panorama image editing.
Method	Resolution	Total time (sec.)
InstructPix2Pix + Overlapping Patches	1920×640	62
CSD-Edit (Ours)	1920×640	68
InstructPix2Pix + Overlapping Patches	3968×4352	487
CSD-Edit (Ours)	3968×4352	275
In video editing experiments, we measure the total computation time of our method and baseline methods. As shown in Table 9, our method, CSD-Edit, consistently outperforms the baselines, even taking the shortest time in editing ( $\times 3.3$  times faster than FateZero [22],  $\times 1.3$  times faster than Pix2Video [25]). Note that the computational efficiency of our method, CSD-Edit, on video editing can be easily improved by reducing the number of iterations for optimization while using a larger learning rate. Here, using a higher learning rate changes frames more radically, thus slightly degrading the frame consistency, however, the effect is meager as shown in CLIP Image consistency and LPIPS metrics.
Table 9: Computation time comparison on video editing.
Total time (sec).	CLIP Directional Similarity ↑	CLIP Image Consistency ↑	LPIPS ↓
FateZero [22]	192	0.312±0.003	0.948±0.001	0.264±0.002
Pix2Video [25]	77	0.229±0.001	0.948±0.001	0.282±0.001
CSD-Edit (Ours)	59	0.320±0.001	0.955±0.001	0.243±0.001
CSD-Edit (Ours)	423	0.319±0.002	0.957±0.001	0.235±0.001
In text-to-3D generation experiments, when directly comparing with DreamFusion [26](or SJC [63]), there is a slight increase in the computational cost due to the usage of LPIPS metric as distance for RBF kernel. For instance, it takes 1 hour to generate a 3D model with DreamFusion, while using CSD takes 84 minutes. We note that the increment is not substantial, however, ours yields better qualities with finer details, as shown in Table 5 and Figure 14 in the Appendix.
C.5 More analysis about the batch size
We denote  $\mathbf{N}$  as the total number of images (e.g., total patches of a panorama image or total frames of a video) and we update B minibatch of samples per iteration. Intuitively, using a large B would encourage more consistency among samples, but it is more computationally expensive. Also, we observed that using large batch sizes dilutes the effect of editing. Thus, the batch size controls the trade-off between computation time, editing quality, and preserving consistency. To further verify our choice of B, we provide additional ablation studies on the panorama image editing experiments. Given the same experimental setup as in Section 4.1 with fixed guidance scale=7.5, we swept over  $\mathrm{B} = 4, 8, 12$ , and measured the CLIP source-target image similarity, CLIP directional image-text similarity, and computation time (iteration per second for total 200 iterations, measured on single A100 40GB GPU). Table 10 demonstrates the effect of batch size.
Table 10: Ablation study about the batch size.
Batch size	Total time (iter / sec.)	CLIP Directional Similarity ↑	CLIP Image Similarity ↑
B=4	2.86	0.639	0.240
B=8	1.47	0.692	0.217
B=12	1.02	0.739	0.195
"Turn penguins into chickens"
(a) minibatch  $B = 3$
(b) minibatch  $B = 12$
Figure 9: Ablation study: the effect of batch size. One can control the diversity of generated output, e.g., the same penguins are changed to more diverse chickens, by choosing an appropriate batch size.
"Turn into sunny weather"
"Turn into rainy weather."
"Turn into snowy weather"
"Turn into spring"
"Turn into fall"
"Turn into Van Gogh style painting"
"Turn into Paul Gauguin style painting"
Figure 10: Compositional image editing. CSD-Edit demonstrates the ability to edit consistently and coherently across patches in panorama images. This provides the unique capability to manipulate each patch according to different instructions while maintaining the overall structure of the source image. Remarkably, CSD-Edit ensures a smooth transition between patches, even when different instructions are applied.
"Turn sheeps into wolves"
"Turn sheeps into kangaroos"
"Turn sheeps into polar bears"
"Turn sheeps into reindeers"
"Turn penguins into chickens"
"Turn penguins into bears"
"Turn penguins into pandas"
Figure 11: Object editing. CSD-Edit can edit many objects in a wide panorama image consistently in accordance with the given instruction while preserving the overall structure of source images.
"Turn penguins into sea lions"
"Turn a bear into a tiger"
Figure 12: Video editing. CSD-Edit demonstrates various editing from an object (e.g., tiger) to attributes (e.g., color) while providing consistent edits across frames and maintaining the overall structure of a source video.
"Give him a yellow T-shirt"
"Turn into Van Gogh style painting"
"Turn into Pablo Picasso style painting"
"Turn into Andy Warhol style painting"
"Turn into oriental style painting"
"Turn into Salvador Dali style painting"
CSD-Edit
Figure 13: Ablation study: SVGD and random noise. As illustrated, edits across different patches are not consistent without SVGD. Also, when using random noise as baseline noise, it loses the content and the detail of the source image.
CSD-Edit without SVGD
CSD-Edit with Random Noise
"a fox holding a videogame controller"
"a plush dragon toy"
"a beautiful dress made out of fruit, on a mannequin"
SDS
Figure 14: Text-to-3D generation examples. (First row) CSD helps to capture coherent geometry compared to using SDS. (Second row) CSD allows learning finer details than SDS. (Third row) CSD can provide diverse and high-quality samples without changing random seeds.
"a crocodile playing a drum set"
"Chichen Itza, aerial view"
"a squirrel in samurai armor wielding a katana"
SDS
CSD (Ours)
"Turn flowers into red roses"
"Turn it into an old vintage photo"
Figure 15: Limitations. (First row) CSD-Edit often manipulates undesirable contents due to the inherent inability of Instruct-Pix2Pix model. (Second row) CSD-Edit often produces artifacts on the image due to the patch-wise update.
As our method leverages pre-trained Instruct-Pix2Pix, it inherits the limitations of it such as undesirable changes to the image due to the biases. Also, as described in [14], Instruct-Pix2Pix is often unable to change viewpoints, isolate a specific object, or reorganize objects within the image.
When editing a high-resolution image by dividing it into patches, it often remains an artifact at the edge of the patches, especially at the corner side of an image. This is due to that the patches at the corner are less likely to be sampled during the optimization. See Figure 15 for examples.
When editing a video, the edited video often shows a flickering effect due to the inability of the Stable Diffusion autoencoder to compress the video. We believe that using CSD-Edit with video diffusion models trained on video datasets can possibly overcome this problem.