Boomerang: Local sampling on image manifolds using diffusion models
Authors: Lorenzo Luzi
Paul M Mayer
Josue Casco-Rodriguez
Ali Siahkoohi
Richard Baraniuk
Venue: Accepted by TMLR ()

Abstract: The inference stage of diffusion models can be seen as running a reverse-time diffusion stochastic differential equation, where samples from a Gaussian latent distribution are transformed into samples from a target distribution that usually reside on a low-dimensional manifold, e.g., an image manifold. The intermediate values between the initial latent space and the image manifold can be interpreted as noisy images, with the amount of noise determined by the forward diffusion process noise schedule. We utilize this interpretation to present Boomerang, an approach for local sampling of image manifolds exploiting the reverse diffusion process dynamics. As implied by its name, Boomerang local sampling involves adding noise to an input image, moving it closer to the latent space, and then mapping it back to the image manifold through a partial reverse diffusion process. Thus, Boomerang generates images on the manifold that are ``similar,'' but nonidentical, to the original input image. We can control the proximity of the generated images to the original by adjusting the amount of noise added. Furthermore, due to the stochastic nature of the partial reverse diffusion process in Boomerang, the generated images display a certain degree of stochasticity, allowing us to obtain ample local samples from the manifold without encountering any duplicates. Boomerang offers the flexibility to work seamlessly with any pretrained diffusion model, such as Stable Diffusion, without necessitating any adjustments to the reverse diffusion process. We present three applications for local sampling using Boomerang. First, we provide a framework for constructing privacy-preserving datasets having controllable degrees of anonymity. Second, we show that using Boomerang for data augmentation increases generalization performance and outperforms state-of-the-art synthetic data augmentation.  Lastly, we introduce a perceptual image enhancement framework powered by Boomerang, which enables resolution enhancement.

Published in Transactions on Machine Learning Research (11/2023)
Boomerang: Local sampling on image manifolds using diffusion models
Lorenzo Luzi Rice University
enzo@rice.co edu
Paul M. Mayer
pmmm3 edu
Rice Universiby
Josue Casco-Rodriguez
jc135Grice.ed edu
Ríce University
Ali Siahkoohi
alisk@rice edu
Rice University
Richard G. Baraniuk
richb@rice.ed edu
Rice University
Reviewed on OpenReview: https: openreview. net/ pee//fore forum? id- NYdTkkjWW
Abstract
The inference stage of diffusion models can be seen as running a reverse-time diffusion stochastic differential equation, where samples from a Gaussian latent distribution are trans- formed into samples from target distribution that usually reside on low-dimensional manifold, c.g., an image manifold. The intermediate values between the initial latent space and the image manifold can be interpreted noisy images, with the amount of noise de- termined by the forward diffusion process noise schedule. We utilize this interpretation to present Boomerang, an approach for local sampling of image manifolds exploiting the re- verse diffusson process dynamics. As implied by its name, Boomerang local sampling involves adding noise to an input image, moving closer to the latent space, and then mapping it back the image manifold through a partial reverse diffusion process. Thus, Boomerang generates images the manifold that are "similar," " but nonidentical, to the original input image. We can control the proximity of the generated images to the original by adjusting the amount of noise added. Furthermore, due to the stochastic nature of the partial reverse diffusion process in Boomerang, the generated images display certain degree of stochastic- ity, allowing us to obtain ample local samples from the manifold without encountering any duplicates. Boomerang offers the flexibility to work seamlessly with any pretrained diffu- sion model, such as Stable Diffusion, without nccessitating any adjustments the reverse diffusion process. We present three applications for local sampling using Boomerang. First, provide framework for constructing privacy-preserving datasets having controllable de- grees of anonymity. Second, we show that using Boomerang for data augmentation increases generalization performance and outperforms state-of-the-art synthetic data augmentation. Lastly, we introduce a perceptual image enhancement framework powered by Boomerang, which enables resolution enhancement.
1 Introduction
Generative models have seen a tremendous rise in popularity and applications over the past decade, ranging from image synthesis (Greić et al., 2021; Dhariwal & Nichol, 2021; Sauer al., 2022), audio generation (yan

---

Published in Transactions on Machine Learning Research (11/2023)
Back prompt = "person"
Back prompt "person"
Back prompt "person"
Back to t prompt "person"
Back prompt "cat"
Original image
Forward to 200
Forward to i 500
Forward to 700
Forward to 800
Figure 1: An example using Boomcrang via Stable Diffusion (Rombach et al., 2022). Starting from an original image 2t ~ p((oo); we add varying levels noise to the latent variables according the noise schedule of the forward diffusion process. Boomerang maps the noisy latent variables back to the image manifold by rumning the reverse diffusion process starting from the reverse step associated with the added noise out of T 1000. The resulting images are local samples from the image manifold, where the closeness is determined by the amount of added noise. Note how, as t approaches to T, the content of Boomerang- generated images strays further away from the starting image. While Boomerang, here is applied to the Stable Diffusion model, it is applicable to other types of diffusion models, e.g., denoising diffusion models (Ho et al., 2020). Additional images are provided in Appendix A.1. Boomerang Colab demo available https://colab. hww.oollab.errerrea.... rwwearch.google........ www.ge.coo//oo/eeeeeoo
den Oord et al., 2016; Klejsa et al., 2019; Kong et al., 2020; 2021), out-of-distribution data detection (Li et al., 2022; Dionelis et al., 2022), and reinforcement lcarning to drug synthesis (Kingma & Welling, 2014; Goodfellow et al., 2014; Bond-Taylor et al., 2021). One of the key benefits of generative models is that they can generate new samples using training samples from an unknown probability distribution. A family of generative models known as diffusion models have recently gained attention both the academic and public spotlights with the advent of Dall-E 2 (Ramesh et al., 2022), Imagen (Saharia al., 2022a), and Stable Diffusion (Rombach et al., 2022).
Generative models estimate the underlying probability distribution, manifold, of dataset by learning to sample from this distribution. Sampling with generative models involves mapping samples from a latent distribution, e.g., Gaussian, via the trained generative model to samples from the target distribution, which yields a global sampling mechanism. While global sampling useful for modeling the whole distribution, there are several important problems which require local sampling -hhe the ability to produce samples close a particular data point. The anonymization of data is one application of local sampling in which the identity of subjects in the datasct should be erased while maintaining data fidelity. Another application is data augmentation, which involves applying transformations onto copies of data points to produce new data points (Wong et al., 2016). A third application of local sampling is to remove noise from an image, especially in the cases of severe noise, where traditional denoising methods might fail (Kawar et al., 2021).
Despite their success in global sampling, GANs (Goodfellow al., 2014), Variational autoencoders (VAFs) (Kingma & Welling, 2014), and Normalizing Flows (NFs) (Rezende Mohamed, 2015) are not the best candidates for local sampling. GANs do not lend themselves well for local sampling: GAN inversion is required, and finding training the best methods for GAN inversion is difficult problem and ongoing area of research (Karras et al., 2020; Xia et al., 2022; Yu et al., 2020). VAEs and NEs, on the other hand, can both project data point x into latent vector z in their respective latent spaces and then re-project z back into the original data space, producing an estimate x' of the original data point. As such, VAEs and NEs can perform local sampling of data point æ by projecting a perturbed version of its corresponding latent vector z back into the original data space. While the straightforwart tractability of VAE- NF- based local sampling is attractive, VAEs and NFs are not stato-of-the-ar (SOTA) on popular tasks such image synthesis. For these reasons, turn to diffusion models, which SOTA (Dhariwal Nichol, 2021),
2

---

Published in Transactions on Machine Learning Research (11/2023)
to perform local sampling. For these reasons, propose local sampling with diffusion models, which are SOTA (Dhariwal & Nichol, 2021) and do not require latent space inversion.
We propose the Boomerang algorithm to enable local sampling of image manifolds using pretroined diffusion models. Boomerang earns its name from its principle mechanism -adding noise of a certain variance to push data away from the image manifold, and then using diffusion model to pull the noised data back onto the manifold. The variance of the noise is the only parameter in the algorithm, and governs how similar the new image is to the original image, as reported by Ho et al. (2020). We apply this technique to three applications: machine learning; (2) data augmentation; and (3) perceptual
(1) data anonymization for privacy-preservee enhancement for low resolution images. We show that the proposed local sampling technique is able to: (1a) anonymize entire datasets to varying degrees; (lb) trick facial recognition algorithms; and (1c) anonymize datasets while maintaining better classification accuracy when compared with SOTA synthetic datasets. For data augmentation we: (2a) obtaín higher classification accuracy when trained on the Boomerang- augmented dataset versus no augmentation at all; and (2b) outperform SOTA synthetic data augmentation. Finally, we show that Boomerang can be used for perceplual image enhancement. The images generated via local sampling: 3a) have better perceptual quality than those generated with competing methods; 3b) are generated faster than other deep-learning methods trained methods such as the Deep Image Prior (Ulyanov et al., 2020); and 3c) can be used for any desired upsampling factor without needing to train or fine-tune the network.
In Section we discuss the training framework of diffusion models, introducing the forward and reverse processes. In Section we introduce our proposed local sampling method- Boomerang and provide insights how the amount of added noise affects the locality of the resulting samples. Finally, describe three applications (Sections 4 to 6) that Boomerang be used without any modification to the diffusion model pretraining.
2 Diffusion models
Diffusion models sample from distribution by learning to reverse forward diffusion process that turns data from the training dataset into realizations of a Gaussian distribution (Ho et al., 2020). The forward diffusion process involves adding Gaussian noise to the input data, To ~ ~p(za), in T steps:
V1 B, + Ec: ~~~ ((0,00, t= 1=
1.... T, is the noise variance at step t, which is typically chosen beforehand (Song & Ermon, 2020). Since the transition from step / - to t is defined by a Gaussian distribution in the form of ((z,/x,,1) == WVVI- [VVI-B,,,-11 B,I), the distribution of x, conditioned on the clean input image on can expressed as a Gaussian distribution,
where Be E (0, 1). t= ...
q( = N N (1 a.) t= 1, I,
(2)
with 04 (1- During braining, diffusson models learn to reverse the forward process by starting 1 == T with a sample from the standard Gaussian distribution ør NN(,,) I). The reverse process is defined via a Markov chain over Øo æ1,..... : WT such that
2. fo(2,, f) m. Th NO0, 3,D), t= T.
(3)
In the above expression, fo(z,, f) is parameterized by neural network with weights Φ, and B,I denotes the covariances at step t. Equation (3) represents a chain with transition probabilities defined with Gaussian distributions with density
p=(0x-1/x;) = N(fol t),1), 1..,..,
(4)
The covariance 3,1 different stops can be also parameterized by neural networks, however, we follow Luhman & Luhman (2022) and choose 3, = 1-ae-1 1-a1 8r, that matches the forward posterior distribution when conditioned on the input image q( |,,, [.,, 20) (Ho et al., 2020).
3

---

Published in Transactions on Machine Learning Research (11/2023)
To ensure the Markov chain in Equation (3) reverses the forward process (Equation (1)), the parameters ø are updated such that the resulting image al step / = 0 via the reverse process represents sample from the target distribution p(2o). This can be enforced by maximizing with respect to & the likelihood (x of training samples where 24 represents the outcome of the reverse process at step = O. Unfortunately, the density Ps(arg) does not permit a closed-form expression. Instead, given the Gaussian transition probabilities defined in Equation (4); the joint distribution over all the T states can be factorized as,
Tol .T) p(ærr) Pa (((Tr)=N((..))
with all the terms on the right-hand-side of the equality having closed-form expressions. To obtain a tractable expression for training diffusion models, we treat æ1...., ær as latent variables and use the negative evidence lower bound (ELBO) expression for po(xu) as the loss function,
log Pe (x x8:) ( (cc 1)
[()) :- Bygts...
2 Ba(an)) |- log (o
After training, new global samples from Pa (x0) 2 p(æo) are generated by rumming the reverse process in Equation (3) starting from 2T ~ I). Due to the stochastic nature of the reverse process, particularly, the additive noise during each step, starting from two close initial noise vectors at step does not necessarily lead to close-by images on the image manifold. The next section describes our proposed method for local sampling on the image manifold.
3 Boomerang method
Our method, Boomerang, allows one to locally sample point an on an image manifold X' close point X' using pretrained diffusion model fg- Since we mainly interested in images, suppose that and z' are images on the image manifold X. We control how close to Do we want zá to be by setting the hyperparameter /Bomm- We perform the forward process of the diffusion model /oom times, from = 0 to = toom in Equation (1), and use fo to perform the reverse process from t = tBoo back to f = 0. If Boom T we perform the full forward diffusson and hence lose all information about æp; this simply equivalent globally sampling from the diffusion model. We denote this partial forward and reverse process B( Boom) a and call it Boomerang because an and ag are close for small Boom, which can be seen in Figure! 1.
When performing the forward process of Boomerang, it is not necessary to iteratively add noise fooo times. Instead, we simply calculate the corresponding Opooom and sample from Equation (2) once to avoid unnecessary computations. The reverse process must be done step by step however, which is where most of the computations lake place, much like regular (global) sampling of diffusion models. Nonetheless, sampling with Boomerang has significantly lower computational costs than global sampling; the time required for use Boomcrang to perform local sampling along with faster sampling schedules, e.g., sampling schedules that reduce sampling time by 90% (Kong & Ping, 2021) before Boomerang is applied. Pseudocode for the Boomerang algorithm is shown in Algorithm 1.
Boomcrang is approximately (Boem times the time for regular sampling. Morcover, T
We present a quantitative analysis to measure the variability of Boomerang-generaleel images as Boom is changed. As an expression of this variability, we consider the distribution of samples generated through the Boomerang procedure conditioned on the associated noisy input image step {Booms i.e., (0 (t ). According to Bayes' rule, we relate this distribution to the distribution of noisy images at step Boom of the forward process,
Po (ooom x P& ( Etooo 16))((2())
( (x))p(21))
(7)
= N (Votnoom 2/. (1 Oct I) pl p(xx)-
4

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm Boomerang local sampling, given diffusion model (x,
Input: 2o, Booom: {artEr {3r]1=1
Output: g/
Et e+ I0,
Boom Attoom 30 + V1 Ottooo 6
for (Boom: .... do
if t > 1 then
8.- 1-a
I-a; 13g
7 ~ NO,,
else
1=0 0
end if
x/-1 + fo(2/; +.m
end for
return z/
The second line in the expression above assumes that by training the diffusion model via the loss function in Equation (6), the model will be able reverse the diffusion process at cach step of the process. The last line the equation above follows from Equation (2) and the fact that p(ar)) = p(æo)) The latter can be understood by noting that p(x() is the distribution of images generated by the diffusion model when the reverse process Varpuen Xo vI O e, where the original images Xy are drawn from p(Xo) and e ~ I). The distribution of these noisy images is equivalent to (y/ttooom Xo, - Of I), with Xo ~ p(xo), which is equal to the forward diffusion process distribution at step Boom, denoted as (Xttooom (xo) (recall Equation (2)). Given that the diffusson model is well-trained, we can expect that its output matches the original image distribution regardless of which step the reverse process in initiated, as long as the same forward diffusion process noise schedule used, Equation (7) suggests that the density of Boomerang--enerater images is proportional to the density of a Gaussian distribution with covariance (1 - (ktgaom))) ) times the clean image density p(æo). In other words, the resulting density will have very small values far away from the mean of the Gaussian distribution. In addition, the high probability region of Po (tnoom) grows as 1 Ot becomes larger. This quantity monolonically increases as /Bom goes from one to T since 0a -I B:) and Bi E (0, 1). As result, we expect the variability in Boomerang generatted images to increase as we run Boomerang for larger fBoom steps.
is initiated at step Boom using noisy images obtained from Xiggom
Since Boomerang depends on a pretrained diffusion model fo: it does not require the user to have access to significant amount of computational resources or data. This makes Boomerang very accessible to practi- tioners and even everyday users who do not have specialized datasets or hardware to train a diffusion model for their specific problem. Boomerang requires that the practitioner find diffusion model that represents the desired image manifold. With the advent of diffusion models trained on diverse datasets, such as Sta- ble Diffusion (Rombach et al., 2022), finding such models is becoming less and less of a problem. Overall, our Boomerang method allows local sampling on image manifolds without requiring significant amounts of computational resources or data.
4 Application 1: Anonymization of data
Local sampling anonymizes data by replacing original data with close, but nonidentical, samples the learned data manifold. Overparameterized (i.c., deep) models are vulnerable to membership inference at- lacks (Shokri et al., 2017; Tan et al., 2022), which attempt to recover potentially sensitive data (e.g,, medical data, financial information, and private images) given only access to a model that was trained on said data. Boomerang local sampling enables privacy-preseree that are similar to real (i.e., sensitive) data, yet not the same. The degree of similarity to the real data can be coarsely controlled through the Boom parameter, however Boomerang not remove specific sen-
machine learning by generating data points
5

---

Published in Transactions on Machine Learning Research (11/2023)
Class
1
2
3
4
5
6
7
Original
Boomerang
Figure 2: Using Boomerang on CIFAR-10 to change the visual features of images. These images were created with FastDPM (Kong & Ping, 2021) using tBoom/ = 100
40 40%.
sitive attributes while retaining other attributes. We prove the effectiveness of Boomerang anonymization by anonymizing various datasets, quantitatively measuring the degree of anonymity for anonymized face datasets, and successfully training classifiers on fully anonymous datasets.
4.1 Data and models
To show the versatility of Boomerang anonymization, we apply it to several datasets such as the LFWPeo- ple (Huang et al., 2007), CelebA-HQ (Karras et al., 2018), CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), FFHQ (Karras et al., 2019), and ILSVRC2012 (ImageNet) (Russakovsky et al., 2015) datasets. For the ImageNet-200 experiments, we use 200-class subset of ImageNet. that we call ImageNet-200;; these are the 200 classes that correspond to Tiny ImageNet (Russakovsky et al., 2015). Furthermore, to show Boomerang can be applied independent of the specific diffusion model architecture, we apply Boomcrang to the Stable Diffusion (Rombach et al., 2022), Patched Diffusion (Luhman & Luhman, 2022),*, Denoising Likelihood Score Matching (DLSM) (Chao et al., 2022)?, and Da (Kong & Ping, 2021) models. We compare Goomerang-generatte data with purely syuthetic data from the SOTA StyleGAN-XL (Sauer et al., 2022) and DLSM models.
When generating Boomerang samples for data anonymization augmentation, we pick Boom so that the Boomerang samples look visually different than the original samples. With the FastDPM model we use tooom T = 40/100 = 40% with Patched Diffusion, we use tooom T = 75/250 = 30%; and with DLSM, we use Boom/ / 250/ 25%.
4.2 Anonymization
Boomerang can anonymize entire datasets to varying degrees controlled by the hyperparameter Buom, which coarsely defines the anonymization level. For example, we anonymize commonly used datasets of face images. Additionally, we anonymize natural images. Specifically, we define that natural image Ty is anonymized to a/ if the features of each image are visibly different such that an observer would guess that the two images of different objects (note that we do not control which features are being anonymized). For cach diffusion model, we pick Boom so that the anonymized images are different, but not drastically different from the original dataset images. Some examples of anonymized images are shown in Figures 2 to 4.
Using Boomcrang, we succossfully anonymize datasets of faces, such that established facial recognition net- works infer that the anonymized and original images are from different people. First, we apply Boomerang Stable Diffusion the LFWPeople and CelebA-HQ datasets several ratios of foomm (0.2, 0.5, 0.7. 0.5,0.7.00... 0.8). Random samples (Figure 4) qualitatively establish that sufficiently large values of Boom replace identifiable features of the original images. Meanwhile, witeset-wide evaluations via facial recognition network embed- ding distances show that the distributions of perceptual dissimilarity between the original and Boomerang- anonymized images shift upward as function of Boom (Figure 5 and Appendix A.1). In fact, the percentage of aooomeaan--nnoyyizan images that the facial recognition networks declare as originating from different people than the original datasel images approaches 100% as Boom increases. Therefore, we qualitatively
We this repository for Patched Diffusion.
We this repository for DLSM. We this repository models. Tt. distills the original 1000 diffusion steps down 100 a STEP DDPM sampler.

---

Published in Transactions on Machine Learning Research (11/2023)
Class
32
67
142
183
Original
Boomerang
Figure 3: Using Boomerang ImageNet-200 to change the visual features of images. These images were created with Patched Diffusion (Luhman & Luhman, 2022) using toom 'T 75/250 30%. The FID values for these images have been plotted in Figure 15 in the Appendix.
Original
Aanam 20%
7
Boom 50%
Boom 70%
Boom 80%
Figure 4: Nine randomly selected LFWPeople samples anonymized via Boomerang Stable Diffusion with the same random seed and with the same prompt: "picture person".
and quantitatively establish that Boomerang anonymization is an efficient method of anonymizing images by local sampling.
Since training on anonymous data enables mrivacy-preserving machine learning, we show in Table 1 that our local sampling method outperforms state-of-the-art generative models on the task of generating entirely anonymous datasets upon which to train classifier. Specifically, we compare Boomerang anonymization against the strongest alternative: image synthesis by StyleGAN-XL (SOTA for CIFAR-10 and ImageNet) and DLSM (SOTA for CIFAR-100). We observe across all tasks that purely synthetic data, even from SOTA models, cannot reach the quality of data anonymized via Boomerang local sampling-ever when the local sampling is done with diffusion models that are not. SOTA. For example, on ImageNet the synthetic data produces a test accuracy of 39.8%, while the caaneerang-nnoyyiiiin data produces test accuracy of 57.8%. This phenomenon makes sense because anonymization via local sampling in-between generating purely synthetic data and using real data. Thus, training on Boomerang-anonye data outperforms training on synthetic data.

---

Published in Transactions on Machine Learning Research (11/2023)
As Boomerang distance (Boom increases,
Increasing Boom also increases the
so does facial recognition embedding distance
percentage of anonymized images
LWF VGG-Face
LWF VGG-Face
Average Embedding Distance
LWF Faceuet
Image over the threshold (%)
100
LWF Facenet
0.8
CelebA VGG-Face
GolebA VGG-Face
CelebA Facenet
CelebA Facenet
Threshold
0.6
50
0.4
0.2
20
40
60
80
20
40
60
80
Bavm
(%)
(
7
T
Figure 5: Facial recognition embedding distances between Boomeranged images and the original images as function of Boom We use the VGG-Face and Facenet models (Wang & Deng, 2021) to calculate the embeddings; both models have a default minimum distance threshold of 0.4 declare that two images are of different people. The largest standard error 0.0017 for embedding distance and 0.21% for the number of images over the threshold. Figure 10 contains the full distribution of VGG-Face embodding distances.
Table 1: With respect to training CIFAR-10, CIFAR-100, ImageNet-2200, and ImageNet classifiers on anony- mous data, Boomerang- wawnerrann-nooymizeee data is a superior alternative to purely syuthetic data from the SOTA StyleGAN-XL and DLSM models.
Classification Task
Top-1 Test Accuracy
Top-5 Test
Training data
Accuracy
CIFAR-10
StyleGAN-XL generated data (SOTA synthetic)
81.5%
C-FAR-10
FastDPM Boomerang data (ours)
84.4%
CIFAR-10
CIFAR-10 data (no anonymization)
87.8%
CIFAR-100
DLSM generated data (SOTA synthetic) DLSM Boomerang data (ours)
26.9%
CIFAR-100
55.6%
CIFAR-100
CIFAR-100 data (no anonymization)
62.7%
ImageNet-200
StyleGAN-XL generated data (SOTA synthetic)
50.2%
73.0%
ImageNet-200
Patched Diffusion Boomerang data (ours)
61.8%
83.4%
ImageNet-200
ImageNet-200 data (no anonymization)
66.6%
85.6%
ImageNet
StyleGAN-XL, generated data (SOTA synthetic)
39.8%
62.1%
ImageNet
Patched Diffusion Boomerang data (ours)
57.8%
81.3%
ImageNet
ImageNet data (no anonymization)
63.3%
85.3%
In conclusion, we have established that data anonymization Boomerang can operate on entire datasets, successfully anonymize facial images, and is superior alternative to generating purely synthetic data for downstream tasks such image classification.
8

---

Published in Transactions on Machine Learning Research (11/2023)
Table 2: Shomerang-generaaee data for data augmentation increases test accuracy of CIFAR-100, ImageNet- 200, and ImageNet classification tasks. The purely sythetic data augmentation was done using state-of-the- art (SOTA) models: Denoising Likelihood Score Matching (DLSM) (Chao et al., 2022) and StyleGAN- XL (Sauer et al., 2022). For the Boomerang data, used DLSM and Patched Diffusion (Luhman & Luhman, 2022).
Classification Task
Top-1 Test Accuracy
Top-5 Test
Training data
Accuracy
CIFAR-100
CIFAR-100 only (no data augmentation)
62.7%
CIFAR-100
CIFAR-100 DLSM DA (SOTA synthetic)
56.0%
CIFAR-100
CIFAR- 100 DLSM Boomerang DA (ours)
63.6%
ImageNet-200
ImageNet-200 only (no data augmentation)
66.6%
85.6%
ImageNet-
ImageNet- + StyleGAN-XL DA (SOTA synthetic)
65.8%
85.5%
ImageNet-200
ImageNNet-200 + Patched Diffusson Boomerang DA (ours)
70.5%
88.3%
ImageNet
ImageNet only (no data augmentation)
63.3%
85.3%
ImageNet
ImageNet + StyleGAN-XT, DA (SOTA synthetic)
63.3%
85.3%
ImageNet
ImageNet + Patched Diffusion Boomerang DA (ours)
64.4%
86.0%
5 Application 2: Data augmentation
Data augmentation for image classification is essentially done by sampling points on the image manifold near the training data. Typical augmentation techniques include using random image flips, rotations, and crops, which exploit symmetry and translation invariance properties of images in order to create new data from the training set. Although there are many techniques for data augmentation, most involve modifying the training data in ways which make the new data resemble the original data while still being different, i.e., they attempt to perform local sampling on the image manifold. Since Boomerang can also locally sample on manifolds, we investigate if using Boomerang beneficial for data augmentation on classification tasks.
For our data augmentation experiments, we pick toom to be large enough produce visual differences between the original dataset and the Boomerang-generatee dataset, as shown in Figures 2 and 3; discussed Scction 4.1. Due to the intrinsic computational costs of diffusion models, we generate the augmented data before training instead of on-the-lly generation during training. We then randomly choose to use the training data or the Seomerang-generatee data with probability 0.5 at each epoch. We use ResNet- 18 (He et al., 2016) for our experiments. We use the same models and datascts as described in Scction 4.1.
To demonstrale the impact of data augmentation using Boomerang, we begin by comparing how it en- hances accuracy in classification tasks as compared to the case of no data augmentation. We find that using Boomcrang for data augmentation increases generalization performance over not using data augmentation at all for a wide range of datasets and classifier architectures. As shown on Table 2, using Boomerang data augmentation on C-0AR-100, TmageNet-2000, and ImageNet classification tasks increases test accuracy from 62.7% to 63.6%, from 66.6% to 70.5%, and from 63.3% to 64.4%, respectively. Applying Boomerang data augmentation requires only selecting a single hyperparameter, namely toom. and it does not make any ex- plicit assumptions about data invariances, making a versatile and general-purpel data augmentation tool. Additionally, Boomerang data augmentation can be combined with other data augmentation techniques, c.g., flips, crops, and rotations.
We also observe that using Boomerang for data augmentation increases generalization performance over using purcly synthetic data augmentation. Specifically, we see that using purely synthetic data augmentation does not appear to help generalization performance at all. The generalization performance is actually lower than or equal to the baseline for CIFAR-100, ImageNet-200 synthetic data augmentation (see Table 2). This somewhat surprising because the models that we used for the Boomerang data augmentation are not SOTA, with the exception of the DLSM model. Meanwhile, the models that. we use for the purely synthetic data augmentation are SOTA and thus compare to
and ImageNet classifications tasks with SOTA
9

---

Published in Transactions on Machine Learning Research (11/2023)
the hardest setting possible. For example, the reported FID (metric of image quality, lower is better) for the Patched Diffusion model we use for Boomerang on ImageNet is 8.57 whereas it is 2.3 for StyleGAN- XL Luhman & Luhman (2022), which is used for the synthetic data augmentation. Therefore, Boomerang data augmentation seems to be more beneficial than using purely synthetic data augmentation even if Boomerang is done with a lower performing model.
In conclusion, we showed that Boomerang data augmentation can be used increase generalization perfor- mance over augmentation and that it even beats synthetic data augmentations using SOTA models.
6 Application 3: Perceptual resolution enhancement for low-resolution images
As the final application of Boomerang local sampling, we propose perceptual resolution enhancement, (PRF) for images. This process aims to upsample low-rcsolution images with focus on perceptual quality even if traditional handcrafted quality measures such as peak signal-to-noise ratio (PSNR) are low. We begin by framing PRE local sampling problem, followed by two proposed perceptual resolution enhancement PRE methods using Boomerang.
6.1 Perceptual resolution enhancement via local sampling
Downsampled or otherwisc corrupted images belong to a different distribution than realistic or natural im- ages. For example, the dimension of ground-truth image Ptrue R2 is higher than that of its downsampled image Ida Rd. We frame PRE local sampling approach to restore Dds through: (1) obtaining rough approximation uup IR" of the ground-truth high-rosolution image using standard resolution enhancement techniques, and (2) restoring perceptual quality via applying Boomerang to Tup. By locally sampling around Tup R Ztrue, Boomerang produces a point on the manifold of high-resolution images near the ground-truth image Ttrue. The goal of PRE is to improve the perceptual quality of a corrupted or low-dimensional image ads without being forced match Fds at every pixel: i.c., unlike traditional super-resolution, PRE is allowed to modify the pixel values of the downsampled image instead of just interpolating between them.
6.2 Vanilla Boomerang perceptual resolution enhancement
We perform Boomerang PRE by:
1. Bringing the corrupted image into the ambient space of the desired result using some off-tho-shelf interpolation method (cubic, nearest-neighbor, etc). In all our experiments, we used lincar interpo- lation.
2. Selecting a Boom corresponding to the desired "radius" of the search space. As fngom - 1, we move from locally sampling around given point to sampling globally on the learned manifold.
3. Performing Boomerang described in Section
The resulting image is an enhanced version of the input image in the neighborhood of the ground-truth image. The parameter tooom corresponds to the strength of PRE applied, with larger values more appropriate for more corrupted images.
While others such as Saharia et al. (2022b) and Rombach et al. (2022) have used diffusion models for image enhancement (including super-resolution) Boomerang has two key advantages to existing diffusion-model. based methods. The first is that Boomerang's local sampling keeps the output "close" to the input the image manifold, i.c., we implicitly account for the geodesies of the manifold instcad of merely optimizing a metric in the Euclidean space. The second is that adjusting /Boom allows easy "tuning" of the PRE strength, controlling the resulting trade-off between image fidelity and perceptual quality. This means that the same pretrained network can perform percoptual enhancement on images with different levels of degradation or downsampling: one can choose larger value of Boom fill in more details of the final image. Furthermore, by varying Boom for different passes of the same input image, one can generate multiple images, each with
10

---

Published in Transactions on Machine Learning Research (11/2023)
different detail/variance trade-off. This can be seen in Figure 11 in Appendix A.2, wherein more aggressive enhancement, improves clarity at the cost of distance from the ground-truth image. Empirical tests showed that setting Boom R 100 on the Patched Diffusion Model (out of I' = 250) produced a good balance between sharpness and the features of the ground-truth image, as seen in Appendix A.2. Finally, the same image can be passed through Boomerang multiple times with a fixed BBoo to generate different candidate images.
Our subsequent PRE experiments with Boomerang all use the Patch Diffusion model (Luhman & Luhman, 2022). With Stable Diffusion (Rombach et al., 2022), empirical results showed that detail was not added to the low-resolution images except for large Boom. where generated images no longer strongly resembled the ground truth images. We believe this is because noise is added in the latent space with Stable Diffusion (as opposed to in the image space with Patched Diffusion). We conclude that the noise impacts which kinds of inverse problems Boomerang will be effective on.
Since Boomerang PRE emphasizes perceptual quality, we evaluate its performance on metrics correlating with human perceptual quality, such as Fréchet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPIPS) (Heusel et al., 2017; Zhang et al., 2018). FID measures the distance (lower is better) between the distributions of "real" images and generated images. LPIPS, on the other hand, compares deep embedding of corresponding patches between two images, and correlates well with human visual perception. We follow (Zhang et al., 2018)'s recommendation and use AlexNet as the backend for LPIPS. Unlike FID, LPIPS is an image-to-image metric. We compare Boomerang PRE with (1) classical super-resolution methods such as nearest interpolation, linear interpolation, and cubic interpolation; and (2) the Deep Image Prior (DIP) (Ulyanov et al., 2020), an untrained method that performs well on super- resolution tasks. We chose not to compare Boomerang to deep learning methods explicitly trained to do image enhancement or super-resolution, as Boomerang and DIP do not require training nor fine-tuning,
Boomerang provides superior performance to all competing methods on perceptual metrics such as LPIPS and FID, as seen in Table 3. Furthermore, the hyperparameter Boom provides a unique trade-off between data fidelity and perceptual quality, enabling cost-effective and controllable generation of perceptually enhanced images. We present the Boomerang perceptual enhancement. results in Table 3.
Another benefit of Boomerang is its flexibility: Boomerang can perform variable perceptual image enhance- ment at various degrees of degradation using the same pretrained diffusion model. This is shown in Figure 13, where we enhance images downsampled by 4x, 8x and even 16x by simply varying Boom. Although met hods like DIP and linear interpolation are also flexible to the extent they are applicable for images downsampled by different factors, many superresolution methods are not. In particular, many deep learning superresolu- tion methods require separale training or fine-tuning depending on the desired fidelity, which requires more compute and explicit model selection. In our examples, which used 1024 x 1024 images, we found Boom = 50 worked well with images downsampled by 4x while BBom = 100 worked best with images downsampled by 8x. Our intuition suggests that a larger value of Boom is more appropriate for more degraded images, and our empirical results are in agreement.
Furthermore, Boomerang PRE is relatively fast and computationally elficient. On 1024 x 1024 images, Boomerang takes 0.932 minutes with a standard deviation of 0.0321 minutes per image. On the other hand, DIP takes about 36.426 minutes with a standard of 0.867 minutes per image. All of these statistics were calculated using random 50 images. In the time it takes to perform PRE on one image with DIP, 39 images can be enhanced with Boomerang. Additionally, Boomerang benefits from diffusion acceleration techniques like batch processing and distillation, whereas DIP cannot because each image is its own optimization problem. Although classical methods, such as linear interpolation, are much faster than Boomcrang, they are not perceptually pleasing and were not specifically designed to do data-driven perceptual enhancement.
6.3 Cascaded Boomerang image enhancement
As Boom is increased, the variance of the generated images dramatically increases due to diffusson models' stochastic nature (noise is added to data at each step t, see Algorithm 1). As a result, increasing &Boom eventually causes the generated images to vary so much that they no longer resemble the input image at all
4These times are reported for single Nvidia GeForce GTX Titan X GPU
11

---

Published in Transactions on Machine Learning Research (11/2023)
Table 3: FID and LPIPS scores of different super-resolution methods (lower better). Each was calculated from 5000 images from the FFHQ datasct (Karras et al., 2019). Since LPIPS is an imago-to-image metric, these values are the average scores across the individual images. FID scores around 5 are within the common range between test and training splits of the same dataset. For Cascaded results, the number of cascades were selected based on the best images decided by human observer who chose the image that had the best perceptual quality.
Image Enhancement Method
LPIPS Score
FID
Nearest Interpolation
0.550
18.82
Linear Interpolation
0.449
20.98
Cubic Interpolation
0.443
16.60
Deep Image Prior
0.353
7.14
Boomerang with Boom 50
0.341
8.93
Boomerang with Boom 100
0.338
5.10
Boomerang with /Boo = 150
0.394
5.19
Boomerang Cascaded, Boom 50
0.351
4.47
(i.e., we move from local sampling global sampling). Even for modest values of Boom the large variance of added noise causes repeated PRE attempts to differ significantly. method to keep variance of the generated image manageable.
In this section, we propose a simple
Cascaded Boomerang describes repeated passes of a corrupted image through a diffusion network with a small value of Boom. as opposed to passing an image through Boomerang once with a large value of BBoom Tf we denote the Boomerang PRE method the previous section on an input image Xds to generate XD = BB (Xax), the method described here would be Xcascade = B,, (.. (B,, (xa+)))). We designate Reascade as the number of times we repeat Boomerang on the intermediate result. In addition to stabilizing independent image enhancement attempts, the cascade method allows users to iteratively choose the desired PRE detail-simply stop repeating the cascade process once the desired clarity is achieved. An example of cascaded perceptual enhancement with Boomerang is shown in Figure 6, with more examples shown in Figure 12 in Appendix A.2.
The cascade method achieved higher FID scores (worse) but lower LPIPS score (better) than vanilla Boomerang PRE. Since FID score calculates distances between distributions, and cascaded Bboomcrang has stabilizing effect on the resulting images, this is consistent with our observations. Despite higher LPIPS score, thought the subjective quality of the cascaded Boomerang PRE images were superior to the vanilla generated images. The cascaded Boomerang method allows for progressive PRE and multiple candidate generation beyond the vanilla Boomerang method.
Boomerang is a quick and efficient method of performing perceptual resolution enhancement with pretrained diffusion models. As we have shown, Boomerang allows the user to easily adjust the strength of the en- hancement by varying Boom. The same procedure and pretrained network can thus be used to perform perceptual enhancement at different strengths and for images any dimension smaller than the output dimension of the diffusion network (we used images with dimensions 1024 1024): for larger scaling factors, simply increase Boom or cascade the result until one achieves the desired fidelity. This also avoids the issue of needing to train different networks depending on the magnitude of enhancement desired or for given factor of downsampling. This reflocts the real-world reality that we often want to "enhance" images that may not conform to a specific scale factor or dimensionality. As with the previous applications, Boomerang's generated images look realistic yet Boomerang requires no additional fine-tuning or training.
7 Related work
The method that has the closest connection to our work terms of the underlying algorithmic procodure involving the reverse process-is SDEdit (Meng al., 2022). This method uses pretrained unconditional diffusion models to edit. generate photorealistic images from sketches non-natural images. While
12

---

Published in Transactions on Machine Learning Research (11/2023)
Initial 8x downsampled Image
Reascade
Reaacadee 8
Ground Truth
Figure 6: Cascaded Boomerang perceptual resolution cnhancement with Boomerang 50.
SDEdit, like Boomerang, utilizes the distinct characteristics of the reverse diffusion process to accomplish its goal, it differs fundamentally from Boomerang in that we focus on local sampling, opposed to image editing. Specifically, we start with a natural image and generate nearby natural images on the learned manifold, and we show its applicability to data anonymization, data augmentation, and image perceptual quality euhancement. Tn contrast, SDEdit lacks the ability for local sampling as it requires an input sketch generate variation of the original natural image. Additionally, the SDEdit algorithm exccutes the complete forward and reverse process with an altered noise schedule, whereas Boomerang solely carries out partial forward and reverse process without modifying the noise schedule, which makes Boomerang computationally efficient. Inspired by SDEdit, Haque et al. (2023) employ an image-conditioned diffusion model to edit scenes generated using NeRF (Mildenhall et al., 2021). The method shares the concept of starting the diffusion reverse process from an intermediate step, but differs from our approach in that it carries out unconditioned local sampling instead relying on taek-ip conditional diffusion model.
In am effort reduce the sampling cost in diffusion models, Zheng al. (2023) proposed initiating the reverse diffusion process from internediate step rather than from pure Gaussian noise. To generate
13

---

Published in Transactions on Machine Learning Research (11/2023)
samples from the diffusion model, the authors suggest utilizing another generative model, such as a VAE, to learn the distribution of noisy data at the intermediate step and employ the samples provided by this model for the reverse process. The methodology described differs from our approach as it does not involve local sampling. Instead, the authors employ a partial reverse diffusion process to expodite the sampling procedure. In their work, Chung et al. (2022) aim lo decrease the compulational costs of solving inverse problems that are implicitly regularized by diffusion models. To achieve this, the authors propose an iterative scheme that involves updating the estimate by taking gradient step using a data fidelity loss function, followed by projecting it into the range of a diffusion model. To expedite the projection, the authors run the reverse process from an intermediate noisy estimate obtained by partially forward diffusing the current estimate. In contrast, our approach for enhancing image perceptual quality differs from Chung et al. (2022) in that we compule the gradient step only once al the beginning, followed by projecting it onto to the range of the diffusion model. This allows us to perform local sampling without modifying the reverse process, while Chung et al. (2022) alters the reverse process, making it conditional.
Other related work modifies the reverse diffusion processes to perform data anonymization, augmentation; upsampling. Recently, Klemp et al. (2023) proposed technique for dataset anonymization utilizing pretrained diffusion models. Specifically, they remove sensitive components from an image and then use a pretrained Stable Diffusion impainting model to fill in the removed details. In comparison to their approach, our proposed method is more controllable (through the use of parameter /Boom), requires fewer iterations to execute (especially for low values of Boom), and does not require the diffusion model to be specifically trained for inpainting purposes. Trabucco et al. (2023) present a novel data augmentation methodology that utilizes pretrained text-lo-image diffusion model. While Boomerang is applicable to any generic diffusion model, their approach specifically necessitates the use of text-to-image model, and importantly, requires textual input to generate augmented images Kawar et al. (2022) propose novel approach that leverages pretrained unconditional diffusion models to remove JPEG artifacts. Specifically, their methodology involves modifying the reverse process by conditioning it on the JPEG-compresedd image. In another related work, Lugmayr et al. (2022) alter the reverse process to condition it on an image with occlusions to perform inpainting. Both of these methods for JPEG artifact removal and image inpainting utilize a partial reverse diffusion process; however, they modify the reverse process, whereas Boomerang does not make any modifications. The image editing methodology proposed by Ackermnann & Li (2022) presents a multi-stage upscaling process suitable for high-resolution images using the Blended Diffusion model (Avrahani et al., 2022). Specifically, this image editing approach involves passing a low-resolution image through an off-the-shell super- resolution model, followed by the addition of noise and reverse diffusion from an intermediate stage using the text- guided Blended diffusion model (Avrahami et al., 2022). While the usage of a partial reverse diffusion process is shared with Boomerang, this method is fundamentally different that it aims to perform image editing instead of local sampling.
Finally, the recently introduced consistency models (Song et al., 2023) aim to reduce the computational complexity of diffusion models during inference. In their few-step method, a network is trained to perform the reverse process using significantly coarse time discrctization. The last step in the few-step method, which maps an intermediate noisy image to the final image manifold, is, in fact, an instance of Boomerang. In addition, the ftw-step method provides a means to trade off compute for sample quality. While this resembles method for perceptual image quality enhancement, in
the trade-off that we describe in our Boomerang- which fROM trades off compute and data fidelity for perceptual quality, consistency models are essentially model distillation approach and do not perform local sampling.
8 Conclusions
We have introduced the Boomerang algorithm, which is a straightforward and computationally efficient method for performing local sampling on an image manifold using pretrained diffusion models. Given an image from the manifold, Boomerang generates images that are "close" to the original image by reversing the diffusion process starting from an intermediate diffusion step initialized by the diffused original image. The choice of intermodiate diffusion step Boom determines the degree of locality in Boomerang sampling, serving as a tuning parameter that can be adjusted based on the specific requirements of the downstream application. Boomerang can be run on a single GPU, without any re-training, modifications to the model,
14

---

Published in Transactions on Machine Learning Research (11/2023)
change in the reverse diffusion process. We showed the applicability of Boomerang various tasks, such anonymization, data augmentation, and perceptual enhancement. Future works include continued experiments of its efficacy for data augmentation, as well as applying the Boomerang algorithm to different domains of data, such as audio and text. Finally, recent work has shown that diffusion models can work with non-stochastic transforms instead of additive Gaussian noise (Bansal et al., 2022; Daras al., 2022), and evaluating the Boomerang algorithms with such diffusion models would provide further insight into the nature of local sampling using diffusion models.
Acknowledgement
This work was supported by NSF grants CCF-1911094, TIS-1838177, and IIS-1730574; ONR grants N00014- 18-12571, N00014-20-1-2534, and MURI N00014-20-1-2787; AFOSR grant FA99550-221100000; DOE grant DE-SC0020345; and a Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2047.
References
Johannes Ackermann and Minjun Li. High-resolution image editing via multi-stage blended diffusion, 2022.
URL https://arxiv.org// wttgg///////....o/////g///// 12965.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18208-18218, June 2022.
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint ar xrx::22008.095: 2022.
Sam Bond-Taylor Adam Leach, Yang Long, and Chris G. Willcocks. Deep generative modelling: A compara- tive review of VAEs, GANs, normalizing flows, energy- -baee and autoregressive models. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. doi: 10. 10.1109/TPAM.22011. 110000211111160000 16668.
Chen-Hao Chao, Wei-Fang Sun, Bo-Wun Cheng, Yi-Chen Lo, Chia-Che Chang, Yu-Lun Liu, Yu-Lin Chang, Chia-Ping Chen, and Chun-Yi Lee. Denoising likelihood score matching for conditional score-based dala generation. arXiv preprint Xrx:22200. 14206, 2022.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-ciifuss--fssstt Accelerating conditional diffusion models for inverse problems through stochastic contraction. In 2022 IEEE/CVE Conference on doi: 10.1109/CVPRR5268.
Computer Vision and Pattern Recognition (CVPR), pp. 1400-11412, 2022. 2022.01209.
Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G Dimakis, and Peyman Milanfar. Soft diffusion: Score matching for general corruptions. arXiv preprint #xXi:2209.054422 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In M. Ranzato, A. Boygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 8780-8794, 2021.
Nikolaos Dionelis, Sotirios A. Tsaftaris, and Mehrdad Yaghoobi. Omasgan: Out-of-distribbttio minimum anomaly score gan for anomaly detection. In 2022 Sensor Signal Processing for Defence Conference (SSPD), pp. 1-5, 2022. doi:
Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshma Bongio. Generative Adversarial Nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advonces in Neural Information Processing Systems, volume 27, 2014.
15

---

Published in Transactions on Machine Learning Research (11/2023)
Matej Grcié, Ivan Grubišić, and Siniša Segvié. Densely connected normalizing flows. In Advances in Neu- vol Information Processing Systems, volume 34, pp. 23968 23982, 2021. URL https://proceeddngss seaper.pdt.
pttrrips.ccpapper. httpa///e....e.......ooooooooaaa
Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct- NeRF2NeRF: Editing 3D scenes with instructions. arXiv preprint 2303. 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Martin Hensel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two lime-scale update rule converge to a local nash equilibrium. Advances in neural infor- mation processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Aduances in Newral Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical report, University of Massachusotts, Amhorst, 2007.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE /CVF conference on computer vision and pattern recognition, pp. 4401-4410 2019.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVI Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Stochastic image denoising by sampling from the posterior distribution. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 1875, 2021.
Bahjat Kawar, Jiaming Song, Stefano Ermon, and Michael Elad. JPEG artifact correction using denoising diffusion restoration models. In NewrIPS 2022 Workshop on Score- Methods, 2022. URL https: / //openreview net//forumm2dd=00W00779003
Diederik P Kingma and Max Welling, Auto eetto--ncoding variational bayes. International Conference on Learning Representations, 2014. doi: 10.48550/ 10.48550/ARXXV ARX.V.1312.1..II
Janusz Klejsa, Per Hedelin, Cong Zhou, Roy Fejgin, and Lars Villemoes. High-quality speech coding with sample RNN. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. (155- 2019. doi: N0.1109/ICASSP:20000 2019.8682433.
Marvin Klemp, Kevin Rösch, Royden Wagner, Jannik Quehl, and Martin Lauer. Ldfa: Latent diffusion face anonymization for self-driving applications. arXiv preprint 0rXiv:2302.08991 2023.
Jungil Kong, Jaehyeon Kim, and Jackyoung Bae. HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-11 17033, 2020.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DifWWac: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. URL https: / //openreview.co wtwpse/wweeeeooouueeeee xF0aaax5..8K2.....
16

---

Published in Transactions on Machine Learning Research (11/2023)
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Yewen Li, Chaojje Wang, Xiaobo Xia, Tongliang Liu, xin miao, and Bo An. distrrbiitt detection with an adaptive likelihood ratio on informative hierarchical vae. In Advances in Neural Information Processing Systems, volume 35, pp. 7383-7396, 2022.
Andreas Lugmaar, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models, 2022.
Troy Luhman and Erie Luhman. Improving diffussin model efficiency through patching. orXiv preprinl orXiv:2207.04316 2022.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022. URL https:// /oppneeveneeeeeeee /tt./
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF:: Representing scenes as neural radiance fields for view synthesis. Commun. ACM, 65(1):99 106, dec 2021. ISSN 0001-0782, doi: 10.1145/300 10.1145/3503255 /3032200.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint or orXiv2224.001 2022.
Danilo Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. In Francis Bach and David Blci (ods.), Proccedings of the 32nd International Conference on Machine Learning, volume 37, pp. 1530 1538, 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10684 10695, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, al. Photorealistic text-to- image diffusion models with deep language understanding. arXiv preprinl arXiv:2205. 1487, 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via itcrative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 14, 2022b. doi: 10.1109 TT00/TTPM..20023304406
Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN-XL: Scaling StyleGAN to large diverse datasets. Tn ACM SIGGRAPH 2022 Conference Proceedings, pp. 1 1-10, 2022.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine lcarning models. In 2017 IEEE Symposium Security and Privacy (SP), pp. 3-18, 2017. doi: 10.1109/SP.2017..41
Yang Song and Stelano Ermon. Improved techniques for training score-based generative models. In Advances in Neural Information Processing Systems, volume 33, pp.
2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and Tlya Sutskever. Consistency models, 2023. URL https: wwtp:/s////......o/////// 01469.
Jasper Tan, Blake Mason, Hamid Javadi, and Richard G Baraniuk. Parameters or privacy: A provable tradeoff between overparameterization and membership inference. ar preprint Xix:22202.012/33 2022.
17

---

Published in Transactions on Machine Learning Research (11/2023)
Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models, 2023.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. International Journal of Computer Vision 128(7):1867-1888, mar 2020. doi: (0.1007/111263-00-000:000 URL https: //doi.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukeuoglu. WaveNet: generative model for raw audio. In Proc. 9th ISCA Workshop Speech Syndhesis Workshop (SSW 9), pp. 125, 2016.
Mei Wang and Weihong Deng. Deep face recognition: survey. Newrocompuling, 429:215 244, 2021. doi: atticlle/p1///09922321001111
200.eccco..20000000...... URL https: https://www.scieeneeeee scieccdirrec.................... com/science/
Sebastien C. Wong, Adam Gatt, Victor Stamatescu, and Mark D. McDonnell. Understanding data augmen- tation for classification: When to warp? In 2016 International Conference on Digital Image Compuling: Techniques and Applications (DICTA), pp. 1-6, 2016. doi: 10.1109/ 10.1109/DICTI 201.TA..11...777099
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. GAN inversion: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry Davis, and Mario Fritz. Inclusive GAN: Improving data and minority coverage in generative models. In European Conference Computer Vision (ECCV), 2020. doi:
23.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effec- tiveness of deep features a pereeptual metric. CoRR, abs/1801.08 2018. URL http://arxiv.orr org/ 0tss/8801.03924.
Huangjie Zheng, Pengeheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models and diffusion-based adversarial anto-encoders. In The Fleventh International Conference on Learning Representations, 2023. URL http://pwrevve.e.etttttrrr/a//ae
18

---

Published in Transactions on Machine Learning Research (11/2023)
A Appendix
A.1 Goomerang-generraee images via the Stable Diffusion model
Here we present additional images created via the Boomerang method that indicate the evolution of the predicted image as we increase /Bom These images are generaled via the pretrained Stable Diffusion model (Rombach et al., 2022) where instead of adding noise to the image space during the forward process it added in the latent space. Figures to showcase this where the images on the bottom row show noisy latent variables whereas the ones the top row indicate the Boomerang predictions with increasing amounts of added noise from left to right, except for the rightmost image, which is created by using alternate prompt.
A.2 Vanilla Boomerang super-resolution
Here prosent the result of Boomerang perceptual resoltion enhancement described in Section 6.2. Figure 11 illustrates the results for image perceptual enhancement. The top-left image in this figure shows the low-resolution image, and the top-right and bottom-left images are the result of vanilla Boomerang perceptual enhancement with Boom = 100 and Boom = 150, respectively. When compared with the high- resolution image in the bottom-right corner of Figure 11 we observe that the resulting image with toom = 100 is plausible while the result with Boom 150 seems high-resolution, but is inconsistent compared to the ground-truth image.
B Trade-off between accuracy and Boomerang distance in data augmentation
The observed benefits of Boomerang in data augmentation presented Section 5 are for specific values of Bunm One can observe that there would be no benefit in picking سسس 0% because then no modification of the training data is performed and hence the augmented data will be the same the non-augmented data, are essentially just sampling purely synthetic images from the diffusion model and using that for data augmentation. As shown in Section 5, there exists a value of fa which is better than both of these extremes, In Figure 14 we show accuracy on the CIFAR-100 classification task for intermediate values of 1Beam demonstrating a tradcoff between T
i.e., the training set. On the other hand, if we use سسيا 100%,
accuracy and fooo 7
Back to 0 prompt "cat"
Back 0
Back to 0 prompt = "person"
Back to ( 0
Back to / 0 prompt = "person" "
Original image
prompt = "cat"
prompt "cat"
Forward, 1 500 Forward, 800
Forward, I 900
Figure 7: The Boomerang method using Stable Diffusion (T 1000), in Figure 1, with an image of a cat.
19

---

Published in Transactions on Machine Learning Research (11/2023)
Back to = prompt
Back to t 0 "bedroom"
Back to f = 0 prompt "bedroom"
Back to f = 0 "bathroom"
Original image
"bedroom"
prompt
prompt
Forward to t 200
Forward 500
Forward to f s00
The Boomerang method using Stable Diffusion (T 1000), as in Figure 1, with an image of a bedroom.
Figure 8:
Back to = 0 prompt "person" "
Back to f = 0
Back to t = 0
Back to = 0 prompt = "dog" "
Original image
prompt "person"
prompt = "person" "
Forward to 200
Forward
500
Forward
800
Figure 9: The Boomerang method using Stable Diffusion (T 1000), as in Figure 1, with an image of Albert Finstein.
20

---

Published in Transactions on Machine Learning Research (11/2023)
Distribution of LFWPeople Embedding Distances
Distribution of CelebA-HQ Embedding Distances
1,500
Boom/ /T
3,000
Boom / /T
20%
20%
50%
50%
Count
1,000
70%
Count
2,000
80%
80%
500
1,000
0
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
Embedding Distance
Embedding Distance
Figure 10:
Distribution of facial recognition embedding distances from VGG-Face.
Initial DS Image
Boomerang 100
Boomerang 150
Ground-truth Image
Figure 11: Boomerang perceptual resolution enhancement for different values of (Boom-
21

---

Published in Transactions on Machine Learning Research (11/2023)
DS Image
Hease
Dlass 8
Ground-truth Image
DS Image
Rease
nasc
Ground-truth Image
Figure 12: Cascaded Boomerang perceptual resolution enhancement. For the top row, notice how the best
quality image is seen after 6 cascade steps, with Boomerang = 50. After 8 cascades, details such as the teeth begin to be removed. On the bottom row, however, Pease = 8 provides better results and more detail compared to previous steps. This shows the value in introducing the cascade method: different images may have better results with more cascade steps than others.
22

---

Published in Transactions on Machine Learning Research (11/2023)
Ground Truth
4x Downsampling
Boomerang PRE, toom
8x Downsampling
Boomerang PRE, Boom 100
16x Downsampling
Boomerang PRE, toom 150
Figure 13: Perceptual resolution enhancement with Boomerang. Here, we show that by adjusting Boom, we can easily control the strength of the perceptual resolution enhancement applied, where larger values are more appropriate for lower resolution images.
23

---

Published in Transactions on Machine Learning Research (11/2023)
Data augmentation trade-off between accuracy
and Boom on CIFAR-100 classification
62
Accuracy (%)
60
58
56
0
20
60
80
100
(Boom (%)
Figure 14: The trade-off between accuracy and Sugua has a maximum beneficial peak somewhere ween 0% and 100% data augmentation. At 0% we have essentially no data augmentation because Boomerang does not modify the augmented training data. At 100% we augment with purely synthetic data, which is typically has lower quality than training data.
FID increases with /oam
4
LFWPeople
CelebA-HQ
3
FID
2
0
0
20
60
80
Boom
(%)
Figure 15: Using Boomerang with Stable Diffusion on the LFWPcople and CelebA-HQ datasets results larger FID values for larger values of Sagg This is in part because real images better than synthetic, even if uses excellent diffusion model such Stable Diffusion.
24

---

and Image which are Stable Published in Transactions on Machine Learning Research (11/2023)
= 200
t == 400
t 600
t t= 800
-=0
Boomerang
(original image)
noise level = 0
noise level 200
noise level 400
noise level 600
noise level 800
UnCLIP
guidance 15
guidance 10
guidance 5
guidance 3
guidance
Image Variations
Figure 16: Here we use the same original image from Figure to compare Boomerang local sampling (lop) with UnCLIP /https:/bbugiingface Variations hhttps:/hhuggingffacl http://ggggggg.eee..e/aaaaaaaaaa Diffusion models fine-tuned to accept CLIP guidance to produce variations of an input image. The same two random soeds are used across each method. For Boomerang and UnCLIP, we used the default guidance of 7.5 and the prompl "picture of a person". UnCLIP has noise level parameter that behaves similarly to Boom- Image Variations cannot accept prompts and thus only has guidance its input parameter. Unlike smoothly interpolate between locally sampling very closely and very distantly from the original data point while maintaining consistent image quality. For example, notice that neither CLIP guidance method can recover the original image.
either CLIP guidance method, Boomerang
25


Reviews:
[{"review_id": "t3FzL1Uq1n", "reviewer": "Reviewer_o4y3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n\n### Strengths\n- The technique is extremely simple and can be easily implemented by any practitioner to augment datasets and create similar-looking pictures without much overhead.\n- It is important to demonstrate the applications in a paper with a simple technique. The authors nailed two important applications: preserving privacy and data augmentation; both are extremely important use cases. \n- The empirical evidence for how this simple technique can generate samples that are close to the original image is competing. \n\n### Weaknesses\n\n- The biggest flaw I see is being unable to control any part of the generation. In the sense that you can not control what part of the image is being anonymized. Similarly, you can not control how (or if) each cascading boomerang enhances the image resolution; it will produce a similar image to the output of the previous cascade; however, you can not guide the generation towards enhanced resolution. Therefore, anonymization and super-resolution applications feel more like a stretch to me.\n\n- Can the authors precisely clarify what are the changes between the anonymization and data augmentation experiments? In particular, if you took the data you used from augmentation experiments and trained only on augmented data, is the performance the same as the anonymized data? \n\n- How do you decide if the image is being anonymized or prepared for data augmentation? Is the augmented data always anonymized? Is the anonymized data always augmented? \n\n- Also, there are no error bars in any experiment. he absence of error bars is generally a concern. Could the authors comment on the lack of variance estimates? Isn't the data generation process stochastic and should lead to multiple answers no matter where you apply Boomerang? \n\n- While data augmentation is the primary application, it is fairly limited by being unable to control what the diffusion model would generate. If someone wants to use Boomerang to create data, then adjusting the $t_{\\mathrm{Boom}}$ parameter has a huge cost. You first sample a huge dataset with $t_{\\mathrm{Boom}} = 100$. You train and check the performance. It did not work great. So, you do that again with $t_{\\mathrm{Boom}} = 200$, and so on. \n\n- Also, what is the synthetic data in Table 2? Is this the synthetic data generated from some pre-trained diffusion model? How do we know this generated synthetic data is relevant to the classification problem? This is more of curiosity than criticism. I want to understand the comparison. \n\n- In Table 3, you mention that the number of cascades was selected based on the \"best images.\" What are the best images? Who decides the best images?\n\n- In Section 6.2, the last paragraph, you talk about the time it takes to generate an image with Boomerang. On what computing architecture are you reporting these numbers? \n\n- In the perceptual resolution enhancement, can the authors comment on what fraction of the images enhanced using boomerang falls below the minimum distance threshold from Figure 5? Can this be a good metric to demonstrate super-resolution? \n\n- The authors do a good job of not complicating a simple procedure. Well, for the most part. The purpose behind equation 7 was unclear. In the text, you state that the probability of $p(x_0')$ is the same as $p(x_0)$. Why is that true? Also, what is the inference from this small analysis? That the variability increases from increasing $t_{\\mathrm{Boom}}$? Don't we already know this without the confusing math?\n\n\n\nOverall, I feel that this paper is born out of a simple and beautiful observation. However, I wonder if this beautiful observation is as important as the authors claim. Clarifying the above questions can probably convince me otherwise.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, authors propose a simple technique to do \"local sampling\" of images from a pre-trained diffusion model where the term \"local sampling\" roughly translates to sampling images related to a given image. The idea is very simple: instead of inverting an image all the way to noise, you stop mid-way and return to the image manifold (giving the approach its name--Boomerang.) Since we return mid-way, the image does not completely lose its perceptual quality; however, since the generative process is stochastic, the generated image differs from the original image. Importantly, the simple idea can be implemented with minimal changes. The authors employ this simple technique for constructing privacy-preserving datasets, augmenting datasets, and enhancing image resolution. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses section. ", "broader_impact_concerns": "The most important concern is in terms of privacy-preserving applications. As far as I can see, there is no discussion on ensuring that the proposed method can preserve the user's privacy. Consider this scenario: a person deploys a system based on this technique where they take photos of users and \"anonymize\" them. However, they did not boomerang for long enough, and now the identity of the person in the photo is still revealed. \n\nMy concern may seem unwarranted to the authors. However, this technique is so simple and so easy to employ that anybody with access to a GPU can run this \"anonymization\" scheme. So, I believe there has to be a rigorous discussion about the claims of anonymity. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SighxRkq9f", "reviewer": "Reviewer_3PNA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the idea is very simple, yet effective. The applications of the framework are really important. Data anonymization becomes more and more important, especially as it becomes clear that diffusion models that are trained on LAION (such as Stable Diffusion) can memorize training images. The synthetic data application is also very important to avoid overfitting the dataset (in the case of small datasets) and to potentially mitigate biases in the model (e.g. by balancing different classes in the dataset). Finally, I really like the idea of using a pre-trained diffusion model to perform image enhancement without further training. \n\nMy main concern with this work is the novelty of the proposed method. I believe that the trick of producing image variations by adding noise and then denoising has been known to the community of people working with Stable Diffusion and other foundation models. For example, this feature seems to be supported out-of-the-box in this well-known open-source repo: [https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/2918](https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/2918). \n\nAdditionally, there are other ways to perform image variations too, e.g. by getting the CLIP embedding of the input image and then using it to guide the forward process. Adjusting the guidance scale has a similar effect to controlling the amount of noise added. I don't see any comparison with this method. Again, producing image variations seems to be something that the practitioners can already do and there are tools to do so, e.g. see [here](https://huggingface.co/spaces/lambdalabs/stable-diffusion-image-variations). \n\nFinally, an alternative way to produce image variations is to use Dreambooth or some other framework for personalized text-to-image generation. I understand that this framework requires some additional work (finetuning the model) but the cost of this seems to be trivial with LoRA and other similar tricks widely used by the practitioners. \n\nMy overall comment is that I get the sense that producing image variations has been a topic of interest, at the very least to the community of practitioners, and I believe that the authors do not establish well enough the connection to the existing efforts. I tried to follow the references in these open-source implementations and it seems that many of them credit the SDEedit paper, which is very similar to the proposed framework. I appreciated the discussion of the differences with the SDEedit in the main paper, but I am not convinced that the idea is significantly different. The authors mention that in the SDEedit paper the authors run the whole reverse process with a modified noise schedule. However, the initialization of both methods is the same (a noisy version of the reference image) and even Boomerang can be though as SDEEdit with a modified noise schedule that spends no time in the first part of the reverse SDE. At the very least, I think that a comparison with SDEedit is required.\n\nAnother weakness of the paper is that for each one of the applications there are alternative ways to achieve them and there is no thorough comparison to these approaches. For example, there have been many recent works about diffusion models and synthetic data augmentation. It would be useful to thoroughly explain the differences to such methods and provide numerical comparisons. Further, data anonymization can be achieved by projecting an image to the latent space of a GAN (image inversion) and then perturbing this latent. Finally, the data anonymization might not work if the image to be anonymized was in the training set of the foundation model.\n\n\n\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a framework, termed Boomerang, that can be used to generate variations of a given image using a pre-trained diffusion model. The algorithm works by adding some amount of noise to the input image and then denoising the image using the reverse SDE initialized at the intermediate noise level. One can control how close the generated images will be to the original image by changing the amount of noise added. The authors show how this framework can be used to perform data augmentation, anonymization and image enhancement. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My main concerns were listed in the Section above. The main thing I would like to see is a discussion and a potential comparison with alternative methods for image variations that are widely used by the community, including SDEEdit and the image variations using CLIP guidance.", "broader_impact_concerns": "I do not have any ethical concerns that are specific to this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FKYMqoZxav", "reviewer": "Reviewer_T5Zh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) The presented method is simple and intuitive. Moreover, having a single hyperparameter and compatibility with pretrained-diffusion models makes adoption easy for practitioners.\n\n2) The authors do a nice job of empirically validating the application of the method across a number of important application use-cases. More specifically, the data augmentation results in Section 5 demonstrate that augmenting datasets with purely synthetically generated samples might be suboptimal which is interesting.\n\nWeaknesses: See Requested Changes", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present Boomerang: a method for locally sampling from the data manifold using pre-trained diffusion models. More specifically, local sampling is achieved by noising the input data using the diffusion forward process and using the noisy sample as an initialization for the reverse process to generate the final sample. The degree of variability between the input and the final sample can be controlled by the amount of noise and is a hyperparameter of the method. The authors demonstrate the effectiveness of the method in a number of applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested Changes:\n\n1) For the qualitative results in Fig. 4, it looks like the visual quality of the samples becomes worse as the noise added in the input image increases (for instance at 70% some blurry artifacts are evident). It would be nice if the authors can include some quantitative results (in terms of FID) comparing the quality of the original images with the generated samples at different noising scales (or t_boom values).\n\n2) For the results in Section 5, I think it might be worth including a experiment demonstrating the impact of augmenting with different t_boom on the classification accuracy for a small dataset like CIFAR-10/100.\n\n3) For the super-resolution results, is there a general guideline for selecting t_boom for different downsampling factors? Intuitively, I would expect t_boom to be larger for larger downsampling factors since high-frequency details are added towards the end of the reverse process.\n\n4) It is common to use deterministic samplers like DDIM, DPM-Solver etc. for fast diffusion model sampling. It might be worth adding an experiment demonstrating the robustness of the proposed method within existing experimental setup (for ex: super-resolution) in the context of these samplers.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two of the three reviewers acknowledged that the author response and paper revision has addressed most of the concerns raised in the original reviews and are leaning towards acceptance.  

One outstanding concern from the third reviewer is about the applicability of the method for anonymization applications and lack of sufficient experimental support for this - which I agree with. In particular it is not clear how the proposed method can anonymize sensitive attribute while retaining other attributes of interest since the only controllability it allows is through controlling the variance of the noise added to the data. I agree with this concern and suggest the author revise the paper to clearly describe the limitations of the method when used for the anonymization task.
Decision: Accept (poster)