Learning Image Priors Through Patch-Based Diffusion Models for Solving Inverse Problems
Authors: Jason Hu
Bowen Song
Xiaojian Xu
Liyue Shen
Jeffrey A Fessler
Venue: NeurIPS 2024 poster (2024)
Keywords: reconstruction
computed tomography
deblurring
superresolution
Primary area: diffusion_based_models

Abstract: Diffusion models can learn strong image priors from underlying data distribution and use them to solve inverse problems,
but the training process is computationally expensive and requires lots of data.
Such bottlenecks prevent most existing works from being feasible for high-dimensional and high-resolution data such as 3D images.
This paper proposes a method to learn an efficient data prior for the entire image by training diffusion models only on patches of images.
Specifically, we propose a patch-based position-aware diffusion inverse solver, called PaDIS, where we obtain the score function of the whole image through scores of patches and their positional encoding and utilize this as the prior for solving inverse problems.
First of all, we show that this diffusion model achieves an improved memory efficiency and data efficiency
while still maintaining the  capability to generate entire images via positional encoding.
Additionally, the proposed PaDIS model is highly flexible and can be plugged in with different diffusion inverse solvers (DIS).
We demonstrate that the proposed PaDIS approach enables solving various inverse problems in both natural and medical image domains, including CT reconstruction, deblurring, and superresolution, given only patch-based priors.
Notably, PaDIS outperforms previous DIS methods trained on entire image priors in the case of limited training data, demonstrating the data efficiency of our proposed approach by learning patch-based prior.

Learning Image Priors through Patch-based Diffusion Models for Solving Inverse Problems
Jason Hu
Bowen Song
Xiaojian Xu
Liyue Shen
Jeffrey A. Fessler
Department of Electrical and Computer Engineering
University of Michigan
Ann Arbor, MI 48109
jjashu, bowenbw, xjxu, liyues, fessler) @umich edu
Abstract
Diffusion models can learn strong image priors from underlying data distributions and use them to solve inverse problems, but the training process computationally expensive and requires lots of data. Such bottlenecks prevent most existing works from being feasible for nigh-dimensional and high-resolution data such as 3D images. This paper proposes method to learn efficient data prior for the entire image by training diffusion models only on patches of images. Specifically, we propose patch-based position-aware diffusion inverse solver, called PaDIS, where we obtain the score function the whole image through scores patches and their positional encoding and use this the prior for solving inverse problems. We show that this diffusion model achieves improved memory efficiency and data efficiency while still maintaining the ability to generate entire images via positional encoding. Additionally, the proposed PaDIS model is highly flexible and can be paired with different diffusion inverse solvers (DIS). We demonstrate that the proposed PaDIS approach enables solving various inverse problems in both natural and medical image domains, including CT reconstruction, deblurring, and superresolution, given only patch-based priors. Notably, PaDIS outperforms previous DIS methods trained on entire image priors in the case of limited training data, demonstrating the data efficiency of our proposed approach by learning patch-based prior. Code: https: / github...... com/ sundeco/ PaDIS
Introduction
Diffusion models learn the prior of an underlying data distribution and can use the prior generate new images [1-3]. By starting with clean training images and gradually adding higher levels of noise, eventually obtaining images that are indistinguishable from pure noise, the score function of the image distribution, denoted s(x) = V log p(x)) can be learned by neural network. The reverse process (sampling or generation) then starts with pure noise and uses the learned score function to iteratively remove noise, ending with a clean image sampled from the underlying distribution p(æ).
Inverse problems are ubiquitous in image processing, and aim reconstruct an image from measurement where unknown noise. By Bayes rule, p( yy proportional to p(x) p(y)(=) Hence, to recover æ, is important to have a good estimate of the prior p(x), particularly when contains far less information than æ. Diffusion models are known for their ability learn a strong prior, so there is a growing literature on using them to solve inverse problems 8].
A(x) + A represents a forward operator, and represents random
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Training
X position
position
position
position
Noise
X position
Denoising score matching
UNet
Y position
Training the proposed Patch Diffusion Inverse Solver (PaDIS) method. Different sized patches are used in each training iteration. The X and Y position arrays have the same size as the patch and consist of the normalized X and Y coordinates of each pixel of the patch.
Figure 1:
However, diffusion models require large quantities training data and vast computational power be able generate high resolution images; Song al. [2] and Ho al. [3] used several days to weeks of training on over a million training images in the ImageNet [9] and LSUN [10] datasets to generate 256 256 images. This high cost motivates the research on improved training efficiency for diffusion models, such as tine-tuning an existing checkpoint the required training time and data. However, this strategy restricts the range of usable network architectures and requires the existence of a pretrained network, which limits the range of applications. Besides the demanding training data and computational cost, diffusion models also struggle in very large-scale problems, such as very high resolution images 3D images. To address these challenges, latent diffusion models [13, 14] have been proposed to learn the image distribution in a smaller latent space, but it is difficult to solve general inverse problems in the latent space [15]. Patch-base diffusion models have also been proposed to reduce computational cost. For example, Wang et al. [16] trained on patches of images, but for image generation, ultimately still relied on computing the score function of the whole image at once. Ding et [17] used patches the feature space, requiring additional encoding and decoding step. For 3D volumes, the most common method involves breaking up the volume into 2D slices [12]. [18]. These methods add regularizers between slices to enforce consistency during sampling. and thus do not provide self-contained method for computing the score function the whole volume. These application-speciie strategies make diffieult to adapt these methods general purpose inverse problem solvers using diffusion models [5], [19]. [7]. [20].
different dataset [11, 12] to reduce
Our proposed method tackles these challenges in unified way by training diffusion models on patches of images, as opposed to whole images (see Fig. 1). We provide location the randomly extracted patch to the network to help it learn global image properties. Since each training image contains many patches, the required size of the training dataset is greatly reduced, from the millions usually needed to generate high quality images to only couple thousand even several hundred (see Tab. 5). The required memory and training time also reduced because it is never necessary to backpropagate the whole image through the network. Our proposed method allows for flexible network architecture and only requires it to accept images of any size, a property true of many UNets [3], there is much more flexibility in the architecture design than fine-tuning methods.
At inference time (see Fig. 2), by first zero padding the image, the proposed approach partitions it into patches in many different ways (see Fig. 3), eliminating the boundary artifacts between patches that would appear if non-overlapping patches were used. We develop a method to express the distribution of the whole image in terms of the patch distribution that is learned by the proposed patch- patch-based

---

Reconstruction
DPS step
step
Messurement
Messurement
output
UNet
UNet
C-T
Figure 2: Overview of reconstruction process for the proposed Patch Diffusion Inverse Solver (PaDIS) method. Starting at T, each iteration we choose a random partition of the zero padded image and use the neural network trained on patches to get the score function of the entire image. Due to shifting patch locations, the output image has no boundary artifacts.
network. By incorporating positional information of patches, this framework allows compute the score function of the whole image without ever inputting the whole image into the network. Unlike previous patch-based works that may be task-specific [21-23], the prior defined by our approach may be treated in black box manner and then paired with any other stochastic differential equation (SDE) solver sample from the prior, or with any general purpose inverse problem solver to perform image reconstruction. We conduct experiments on multiple datasets and different inverse problems and demonstrate that the proposed method is able synthesize the patches produce reasonably realistic images and very accurate reconstructions for inverse problems. Furthermore, PaDIS provides a promising avenue for which generation and inverse problem solving very large and high dimensional images may be tackled in the future.
In summary, main contributions
follows:
We provide a theoretical framework whereby score function of a high-resolution high- dimensional image is learned purely through the score function of its patches.
The proposed method greatly reduces the amount of memory and training data needed compared to traditional diffusion models.
The trained network has great flexibility and can be used with many existing sampling algo- rithms and is the first patch-based model that can solve inverse problems in unsupervised manner.
We perform experiments on a variety of inverse problems to show superior image quality over whole image diffusion model methods while being far less resource heavy.
2 Background and Related Work
Diffusion models. Diffusion models consist of defining forward stochastic differential equation (SDE) that adds noise clean image [2]: for / [0, [), (1) E Rd,, we have
dx (B(t)/2) x dt + p(t))
(1)
where B(t) is the noise variance schedule of the process. The distribution of xx()) is the data distribution and the distribution x(T') is (approximately) standard Gaussian. Then, image generation is done through the reverse SDE [24]:
dx = (-3(t)/2 V()) log):(x:)) dt + V Btt)dw..
(2)
By training neural network to learn the score function V.,, log (xx.), one can start with noise and run the reverse SDE to obtain samples from the Iearned data distribution.

---

To reduce the computational burden, latent diffusion models [13] have been proposed. aiming to perform the diffusion process in a much smaller latent space, allowing for faster training and sampling. However, that method requires a pretrained encoder and decoder [25] for fixed dataset, so it must be retrained for different datasets, and it still requires large amounts of training data. Patch-based diffusion models [16, 17] focus on image generation while training only on patches. Supervised patch-based diffusion methods [23, 26] are task specific and do not learn an unconditional image prior that can be applied to all inverse problems. Other patch-based methods [27-29] learn an unconditional image prior but require the whole image as an input during inference time. Finally, work has been done to perform sampling faster [14, 30, 31], which is unrelated to the training process.
Solving inverse problems. For most real-world inverse problems, the partial measurement y is corrupted and incomplete, so the mapping from æ to y is many-to-one, even in the absence of noise, making it impossible to exactly recover œ. Hence, it is necessary to enforce a prior on æ. Traditionally, methods such as total variation (TV) [32] and wavelet transform [33] have been used enforce image sparsity. To capture more global image information, low-rank methods are also popular [34-37]. These methods frequently involve solving an optimization problem that simultaneously enforces data consistency and the image prior.
In recent years, data-driven methods have risen in popularity in signal and image processing [38-42]. In particular, for solving inverse problems, when large amounts training data is available, a learned prior can be much stronger than the handcrafted priors used in traditional methods. For instance, plug and play and regularized by denoising methods [43-49] involve pretraining a denoiser and applying it reconstruction time to iteratively recover the image. These methods have the advantage over supervised deep learning methods such as [50-53] that the same denoiser may be applied to solve a wide variety of inverse problems.
Diffusion models serve as an even stronger prior as they can generate entire images from pure noise. Most methods that use diffusion models to solve inverse problems involve writing the problem as a conditional generation problem [54-56] or as a posterior sampling problem (--,,7), In the former case, the network requires the measurement y (or an appropriate resized transformation of y) during training time. Thus, for these task-specific trained models, at reconstruction time, that network useful only for solving that specific inverse problem. In contrast, for the posterior sampling framework, the network learns an unconditional image prior for æ that can help solve any inverse problem related without retraining. Our proposed method may be paired with most existing posterior sampling algorithms [5-7, 58).
3 Methods
To be able to solve large 2D imaging problems as well as 3D and 4D inverse problems, our goal is to develop a model for p(z) that does not require inputting the entire image æ into any neural network. We would like to simply partition x into nonoverlapping patches, learn the distribution of each of the patches, and then piece them together obtain the distribution of æ. However, this would result in boundary artifacts between the patches. Directly using overlapping patches would result in sections of the image covered by multiple patches be updated multiple times, which inconsistent with the theory of diffusion models. Ideally, we would like to use nonoverlapping patches to update æ, but with variety of different patch tiling schemes so that boundaries between patches do not remain the same through the entire diffusion process.
To accomplish this task, zero pad wo and learn the distribution of the resulting zero padded image. More precisely, consider an N x N image Dy and let < P < N be an integer denoting the patch size and let = [N/P]. Then æo could be covered with a (k + 1) x (k + 1) nonoverlapping patch grid but that would result in (A + 1)P - N additional rows and columns for the patches. Hence, we zero pad vy on all four sides by M = (k- + I)P- N to form a new image with N + 2M rows and columns. With slight abuse of notation, we also denote this larger image by x. Let i, be positive integers between and M inclusive. Fig. illustrates that we may partition æ into (k + 1)² + 1
regions as follows: (k + 1)² of the regions consist of evenly chopping up the square consisting of rows i through + N + and columns j through j + + P into k + by l: + grid, with each such partition being P x P, and the last region consists of the remaining bordering part of x that not included in the first (k- + 1)2 regions. This last region will always be entirely zeros, and the (k + 1)2 square patches fully cover the central N x N image.
4

---

Each pair of integers i and ý correspond to one possible partition, so when we let and j range through all the possible values, our proposal is to model the distribution of æ as the following product distribution:
KJ=M
(k+1)²
p = II Pig.B(Bis.B) =J=1
II (B:g)))) r=1
where Di represents the aforementioned bordering region æ that depends on the specific values of and j. Pi is the probability distribution of that region, æi.j.r is the rth P P patch when using the partitioning scheme corresponding to the values of i and j. Pi.j.r is the probability distribution of that region, and Z is appropriate scaling factor. Generative models based on products of patch probabilities have a long history in the Markov random field literature; see $A.6.
The score function of the entire image follows directly from (3):
Σ ( (...)())...))))
V log)
i.j=1 $t..B(Zi.j.))
(4)
Thus, we have expressed the score function of the whole image æ as sums of scores of patches Rijj.r and the border Ri The former be learned through score matching in $3.1. For the latter, by construction, if x is a zero padded image then Li,i is everywhere zero. Thus, for all x, (( E( Of T] everywhere zero, where D represents the denoiser function. Then the computation of its score function is trivial by Tweedie's 's formula [59].
Importantly, unlike previous papers like [16] and [17], our method can compute the
function of the entire image through only inputs
of patches to the neural network. This makes it
Randen pa8ch
possible to learn a black box function for score functions of large images, where directly train- ing a diffusion model would be infeasible due to memory and time constraints. Furthermore, $4 shows that in the case of limited data, the large number of patches present each training image allows the patch-based model to learn model for the underlying distribution that per- forms better than whole image models.
N
N+ZM
3.1 Patch-wise training
Figure 3: Schematic for zero padding and parti- tioning image into patches
Following the work in [16] and [14], train a denoiser using the score matching approach. Our neural network Dy(æ,, σ2) accepts the noisy image æ and a noise level de, and is trained through the following loss function:
c, =x
(5)
By Tweedie's 's formula [59], the score function is given by sa σi) = (Do(z, 01) 2))o// Here, we want to learn the score function of patches Fiji.r we apply (5) to patches of x instead of entire image. Following [16), we extract patches randomly from the zero-padded image æ. To incorporate the positional information of the patch, we first define the positional array the size N + 2M 2D array consisting of the I positions of each pixel of the image scaled between and the y positional array is similarly defined. We then extract the corresponding patches of these two positional arrays and concatenate them along the channel dimension of the noisy image patch as inputs into the network, nothing that noise is not added to the positional arrays.
When directly applying (5), it would suffice fix the patch size and then train using size P patches exclusively. However, [16] found that by training with patches of varying sizes, training time can be reduced and the neural network learns cross-region dependencies better. Hence, we train both with patches of size P and also patches of smaller size, where the size is chosen according stochastic scheduling scheme. By using the UNet architecture [3], the same network can take images of different sizes input. The Appendix provides details of the experiments.

---

3.2 Sampling and reconstruction
The proposed patch-based based method for learning p(z) may be paired with any method that would
otherwise be used for sampling with full image diffusion model, such as Langevin dynamics [1]
and DDPM [3], as well as acceleration methods such as second-order solvers [14] and DDIM [30].
At training time, the network only receives patches of the image as input, along with the positional
information of the patch. Nevertheless, we show that when the number of sampling iterations is
sufficiently large, the proposed method is still capable of generating reasonably realistic appearing entire images. However, our main goal is solving large-scale inverse problems, not image generation.
Computing s(x) via (4) would require summing over the score functions of all (A: + 1)² patches total of 2 times (corresponding to the M² different ways of selecting and j). This method would be prohibitively slow due to the size of 2 instead, for each iteration randomly choose integers i and j between and M and estimate s s(x) using just that corresponding term of the outer summation.
For solving inverse problems with diffusion models, there are general algorithms e.g., [19] and [5], well as more model-specific algo- rithms, e.g., [6] and [7]. Here, we demonstrate that our method applies to broad range of in- verse problems, and opt to use generalizable methods that do not rely on any special prop- erties (such as the singular value decomposi- tion of the system matrix as in [6]. [7]) of the forward operator. We found that DPS [5] in conjunction with Langevin dynamics generally yielded the most stable and high quality results, so we use this approach as our central algo- rithm. Similar to [5], we chose the stepsize to be c/Ily A(D(x))) knowledge, this is the first work that learns fully patch-basee diffusion prior and applies it to solve inverse problems; we call our method Patch Diffusion Inverse Solver (PaDIS). Com- puting the score functions of the patches at each iteration is easily parallelized, allowing for fast sampling and reconstruction. Alg. 1 shows the pseudocode for the main image reconstruction algorithm; the appendix shows the pseudocode for the other implemented algorithms.
Algorithm 1 Patch Diffusion Inverse Solver (PaDIS)
Require: d1 < 02 ... < <0r, 0, 0. P. P.M.y
Initialize x o()
for I = T 1 do
Sample : (N0 r)
Set a, = o²
Randomly select integers i E [L
For all 1 < r S (k+1)², extract patch æiji
Compute D j... = (a o,)
To the best of our
Set Si.j.r = (Di,,r Bij. /
Apply (4) to get 8 s(x, oo
Set
V. n. ly A()
Set æ to x + 2 s + Vaiz
end for
Finally, we comment on some high-level similarities between our proposed method and [18]: in both cases, the image in question partitioned into smaller parts multiple ways. In [18], one of the partitions consists of 2D slices in the x-y direction, and the other partition consists of 2D slices in the x-z direction, whereas for our method, each of the partitions consists of (K + 1)2 square patches and the outer border region. For both methods, the score functions of each of the parts are learned independently during training. Then for each sampling iteration, both methods involve choosing one of the partitions, computing the score functions for each of the parts independently. and then updating the entire image by updating the parts separately. For our approach, the zero- pardoogpddng method allows for many possible partitions of the image and eliminates boundary artifacts between patches.
4 Experiments
Experimental setup. For the main CT experiments, we used the AAPM 2016 CT challenge data [60] that consists of 10 3D volumes. We cropped the data in the Z-direction select 256 slices and then rescaled the images in the XY-plane to have size 256 x 256. Finally, we used the XY slices from 9 of the volumes to define 2304 training images, and used 25 of the slices from the tenth volume as test data. For the deblurring and superresolution experiments, we used a 3000 image subset of the CelebA-HQ dataset [61] (with each image being of size 256 x 256) for training to demonstrate that the proposed method works well in cases with limited training data. We preprocessed the data by dividing all the RGB values by 255 so that all the pixel values were between and 1. The test data was a randomly selected subset of 25 of the remaining images. In all cases, we report the average

---

metrics across the test images: peak SNR (PSNR) in dB, and structural similarity metric (SSIM) [62]. For the colored images, these metrics were computed in the RGB domain.
For the main patch-based networks, we trained mostly with patch size of 56 x 56 to allow the target image be completely covered by 5 patches in each direction while minimizing the amount of zero padding needed. We used the network architecture in [14] for both the patch-based networks and whole image networks. All networks were trained on PyTorch using the Adam optimizer with 2 A40 GPUs. The Appendix provides full details of the hyperparameters.
Image generation. Our proposed method is able to learn a reasonable prior for whole images, despite never being trained any whole images. Fig. 4 shows generation results for the CT dataset using three different methods. The top row used the network trained on whole images; the middle row used the method of [16] except that the entire image is never supplied the network either at training or sampling time; the bottom row used the proposed method. The middle row shows that the positional encoding does ensure reasonably appropriate generated patches at each location. However, simply generating each of the patches independently and then naively assembling them together leads obvious boundary artifacts due to lack of consistency between patches. Our proposed method solves this problem by using overlapping patches via random patch grid shifts, leading to generated images with continuity throughout.
Figure 4: Unconditional generation of CT images. Top row: generation with a network trained on whole image; middle row: patch-only version of [16]; bottom row: proposed PaDIS method.
Inverse problems. We tested the proposed method on a variety of different inverse problems: CT reconstruction, deblurring, and superresolution. For the forward and backward projectors in CT reconstruction, we used the implementation provided by [63]. We performed two sparse view CT (SVCT) experiments: one using 8 projection views, and one using 20 projection views. Both of these were done using a parallel-beam forward projector where the detector size was 512 pixels. For the deblurring experiments, we used a uniform blur kernel of size 9 x and added white Gaussian noise with a = 0.01 where the clean image was scaled between 0 and 1. For the superresolution experiments, we used a scaling factor of 4 with downsampling by averaging and added white Gaussian noise with = 0.01. DPS has shown to benefit significantly from using higher number of neural function evaluations (NFEs) [5], so we use 1000 NFEs for all of the diffusion model experiments. Appendix A.7 discusses this further.
For the comparison methods, we trained a diffusion model on entire images using the same denoising score matching method shown in Section 3.1. The inference process was identical to that of the patch-based method, with the exception that the score function of the image each iteration was computed directly using the neural network, as opposed to needing to break up the zero -added image into patches. We also compared with two traditional methods: applying a simple baseline and reconstrueting via the total variation regularizer (ADMM-TV), For CT. the baseline was obtained by applying the filtered back-projection method to the measurement y. For deblurring, the baseline was simply equal to the blurred image. For superresolution, the baseline was obtained by upsampling the low resolution image and using nearest neighbor interpolation. The implementation of ADMM-TV can be found in [64]. We also implemented two plug and play (PnP) methods: PnP-ADMM [42]
7

---

and regularization by denoising (RED) [46]. We trained denoising CNNs on each of the datasets following [65] and then used them to solve the inverse problems.
For further comparison of diffusion model methods, we implemented different sampling and data consistency algorithms and applied them in conjunction with our patch-based prior. In particular, compared with Langevin dynamics [I], predictor-correcto sampling [19], and variation exploding DDNM (VE-DDNM) [7]. For all these sampling methods, we used the variation exploding framework for consistency and to avoid retraining the network. We also compared with two other ways of assembling priors of patches to form the prior of an entire image: patch averaging [23] and patch stitching [66]. App. A.5 contains pseudocode for these comparison algorithms.
Table 1 shows the main inverse problem solving results. The best results were obtained after training the patch-based models for around 12 hours, while the whole image models needed to be trained for 24-36 hours, demonstrating a significant improvement in training time. Furthermore, when averaging the metrics across the test dataset, our proposed method outperformed the whole image method in terms of PSNR and SSIM for all the inverse problems. The score functions of all the patches can be computed in parallel for each iteration, so the reconstruction times for these methods were very similar (both around 5 minutes per image). The whole-image results could be more favorable if more training data were used. See data-size study in App. A.3.
We also ran ablation studies examining the effect of various parameters of the proposed method. Namely, we studied the usage of different patch sizes during reconstruction, varying dataset sizes, importance of positional encoding for patches, and different sampling and inverse problem solving algorithms. The results of these studies are in App. A.2.
Table I: Comparison of quantitative results on three different inverse problems. Results are averages across all images the test dataset. Best results are in bold.
CT, 20
Views
CT, 8 Views
Deblurring PSNR1
Superresolution
Method
PSNR
SSIM
PSNR
SSIM
SSIM
PSNR1
SSIM
Baseline
24.93
0.595
21.39
0.415
24.54
0.688
25.86
0.739
ADMM-TV
26.82
0.724
23.09
0.555
28.22
0.792
25.66
0.745
PnP-ADMM [42]
26.86
0.607
22.39
0.489
28.82
0.818
26.61
0.785
PnP-RED [46]
27.99
0.622
23.08
0.441
29.91
0.867
26.36
0.766
Whole image diffusion
32.84
0.835
25.74
0.706
30.19
0.853
29.17
0.827
Langevin dynamics [1]
33.03
0.846
27.03
0.689
30.60
0.867
26.83
0.744
Predictor-corrector [19]
32.35
0.820
23.65
0.546
28.42
0.724
26.97
0.685
VE-DDNM [7]
31.98
0.861
27.71
0.759
26.01
0.727
Patch Averaging [23]
33.35
0.850
28.43
0.765
29.41
0.847
27.67
0.802
Patch Stitching [66]
32.87
0,837
26.71
0.710
29.69
0.849
27.50
0.780
PaDIS (Ours)
33.57
0.854
29.48
0.767
30.80
0.870
29.47
0.846
In addition to the main inverse problems shown in Table 1, we also experiments three additional inverse problems to demonstrate the effectiveness of our method on a wide variety of inverse problems: 60 view CT reconstruction, fan beam reconstruction using 180 views, and deblurring with uniform kernel of size 17 x 17. For the deblurring experiment, we again added Gaussian noise with level 0.01 to the measurement. Table shows the quantitative results of these experiments and Figure 10 shows the visual results.
Table 2: Extra inverse problem experiments. Results averages across all images in the test dataset.
Best results are in bold.
CT, 60 Views
CT, Fan Beam SSIM
Heavy Deblurring PSNR
Method
PSNRT
SSIM
PSNR1
SSIM
Baseline
25.89
0.746
20.07
0.521
21.14
0.569
ADMM-TV
30.93
0.833
25.78
0.719
26.03
0.724
Whole image diffusion
35.83
0.894
26.89
0.835
28.35
0.808
PaDIS (Ours)
39.28
0.941
29.91
0.932
28.91
0.818
In the bottom of Figure 5, some artifacts are present in the reconstructions obtained by the diffusion
model methods, although they are more apparent in the whole image model than with PaDIS. The measurements are very compressed in this case, so it is very difficult for any model to obtain diagnostic-quality reconstructions; the baselines perform significantly worse in terms of quantitative
8

---

FBP
ADMM-TV
Whole image
PaDIS
Ground truth
1200
800
Figure 5: Results of CT reconstruction. 60 views used for the top two rows, 20 views used for the bottom two rows. To better show contrast between organs, we use modified Hounsfield units (HU) in the top figure, while the same scale the images were trained in the bottom figure.
metrics and exhibit severe blurring. In clinical settings, patient diagnosis are typically performed with CT scans consisting of hundreds of views. The top of Figure 5 shows that when 60 views used, our proposed method yields a much better reconstruction without artifacts. Nevertheless, we show the potential of our proposed methods reconstruct images from very sparse views with decent image quality, which could be potentially used for applications such as patient positioning.
Measurement
ADMM-TV
Whole image
PaDIS
Ground truth
AMON
AMON
AMO
Figure 6: Results of deblurring with Gaussian noise (o 0.01).

---

Whole image
Ground
ANOH
ANOH
AMON
AMOH
Figure Results of superresolution with Gaussian noise (a 0.01).
Finally, since the original AAPM dataset contained CT images of resolution 512 x 512, we ran 60 view CT reconstruction experiments with these higher resolution images. Due the lack of data, did not train whole-image model on these higher resolution images. We used zero padding width of 64 and largest patch size of 64 x 64 for training. Figure shows the visual results of these experiments. Hence, proposed methods can obtain high quality reconstructions for both different inverse problems and also for different image sizes.
Table 3: Results of 60 view CT reconstruction with 512 512 images. Results are averages across all images the test dataset. Best results are in bold.
Method
FBP
ADMM-TV
PaDIS
PSNR 1 28.38
29.48
36.93
SSIM t
0.699
0.788
0.899
FBP
ADMM-TV
PaDIs
Ground truth
1400
800
Figure 8: Results of 60 view CT reconstruction on 512 x 512 images using modified HU units.
Conclusion
In this work, we presented method of using score-based diffusion models to learn image priors through solely the patches of the image, combined with suitable position encoding. Simulation results demonstrated how the method can be used to solve a variety of inverse problems. Extensive experiments showed that under conditions of limited training data, the proposed method outperforms methods involving whole image diffusion models. In the future, more work could be done on higher quality image generation using a multi-scaled resolution approach [67, 68), using acceleration methods for faster reconstruction, and higher dimensional image reconstruction. Image priors like those proposed this paper have the potential to benefit society by reducing X-ray dose in CT scans. Generative models have the risk of inducing hallucinations and being used for disinformation.
Acknowledgments and Disclosure of Funding
Work supported in part by a grant from the Michigan Institute for Computational Discovery and Engineering (MICDE) and gift from KLA.
10

---

References
[1] Y. Song and S. Ermon. "Generative Modeling by Estimating Gradients of the Data Distribution". In: Advances in Neural Information Processing Systems. Vol. 32. 2019.
Y. Song. J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. "Seore-Basse Bassed Generative Modeling through Stochastic Differential Equations". In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
[3] J. Ho, A. Jain, and P. Abbeel. "Denoising Diffusion Probabilistic Models". In: 33 (2020). Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, pp. 6840-6851,
[4] H. Chung, B. Sim, D. Ryu, and J. C. Ye. Improving Diffusion Models for Inverse Problems using Manifold Constraints. 2022.
[5] H. Chung. J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye. "Diffusion Posterior Sampling for General Noisy Inverse Problems". In: The Eleventh International Conference on Learning Representations. 2023.
[6] B. Kawar, M. Elad, S. Ermon, and J. Song. Denoising Diffusion Restoration Models. 2022.
[7] Wang. Yu, and Zhang, Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model.
2022.
[8] B. Kawar, G. Vaksman, and M. Elad. SNIPS: Solving Noisy Inverse Problems Stochastically. 2021.
[9] O. Russakovsky et al. ImageNet Large Scale Visual Recognition Challenge. 2015.
[10] F. Yu, A. Seff, Y. Zhang, S. Song. T. Funkhouser, and J. Xiao. LSUN: Construction of a Large-scale
Image Dataset using Deep Learning with Humans the Loop. 2016.
[11] T. Moon, M. Choi, G. Lee, J.-W. Ha, and J. Lee. "Fine-tuning Diffusion Models with Limited Data". In: NeurlPS 2022 Workshop on Score-Basee Baree Methods. 2022.
[12] H. Chung, D. Ryu, M. T. McCann, M. L., Klasky, and J. C. Ye. Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models. 2022.
[13] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-Resolution Image Synthesis with Latent Diffussoo Models. 2022.
[14] T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the Design Space of Diffusion- Baseed Generative Models. 2022.
[15] B. Song, S. M. Kwon, Z. Zhang, X. Hu, Q. Qu, and L. Shen, Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency. 2023.
[16] Z. Wang, Y. Jiang, H. Zheng, P. Wang, P. He, 7. Wang, W. Chen, and M. 7hou, Patch Diffusion: Faster and More Data- Ffficieni Training of Diffusiion Models. 2023.
[17] Z. Ding, M. Zhang. J. Wu, and Z. Tu. Patched Denoising Diffusion Models For Righ-Resolution Image Synthesis. 2023.
[18] S. Lee, H. Chung, M. Park, J. Park, W.-S. Ryu, and J. C. Ye. Improving 3D Imaging with Pre- Pre-Trained Perpendicular 2D Diffusion Models. 2023.
[19] H. Chung and J. C. Ye. "Score-based diffusion models for accelerated MRI". In: Medical Image Analysis 80 (2022), P. 102479. DOE: https: / /doi. org/ 10. 1016// media a 2022. 102479.
[20] Y. Song. L. Shen, L. Xing, and S. Ermon. "Solving Inverse Problems in Medical Imaging with Score-Based Generative Models", In: International Conference on Learning Representations. 2022.
[21] W. Xia. W. Cong. and G. Wang. Patch-Based Denoising Diffusion Probabilistic Model for Sparse-Viee CT Reconstruction. 2022.
[22] W. Xia, H. W. Tseng, C. Niu, W. Cong, X. Zhang, S. Liu, R. Ning. S. Vedantham, and G. Wang. Parallel Diffusion Model-based -bsed Sparse-view Cone-bean Breast CT. 2024.
[23] O. Ozdenizei and R. Legenstein. "Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models". In: IEEE Transactions on Pattern Analysis & Machine Intelligence 45.08 (Aug. 2023), pp. 10346 10356, DOI: 10. 1109/ 0 9/19////PMMM TAMMII 2 2023. 3238179.
[24] D. Anderson. "Reverse-time diffusion equation models", In: Stochastic Processes and their Applica- tions 12.3 (1982), pp.
DOL: https: http://oo..aao orgg10 1011 // 0000033444141 4149 (82) 90051-9 5. [25] Esser, R. Rombach, and B. Ommer. Taming Transformers for Higg-RReoluttin Image Synthesis. 2021. [26] W. Xia, C. Niu, W. Cong, and G. Wang. Sub-volume-bate denoising diffusion probabilistic model for cone-beam CT reconstruction from incomplete data. 2023.
[27] Bieder, J. Wolleb, A. Durrer, R. Sandkuehler, and C. Cattin. "Denoising diffusion models for memory-efficient processing of 3D medical images". In: Proc. Mach. Learning Res. 227 (2024), 552-67.
[28]
T. Luhman and E. Luhman. Improving diffusion model efficiency through patching. 2022.
[29]
W. Peebles and S. Xie. "Scalable Diffusion Models with Transformers". In: arXiv.org (2023). DOr:
10. 48550/ 1550//rxiv... 2212. 09748.
[30]
J. Song, C. Meng, and S. Ermon. Denoising Diffusion Implicit Models. 2022.
[31] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. 2022.

---

[32]
P. J. Huber. Robust statistics. New York: Wiley, 1981.
[33]
I. Daubechies, M. Defrise, and C. De Mol. "An iterative thresholding algorithm for linear inverse problems with a sparsity constraint". In: Comm, Pure Appl. Math. 57.11 (Nov. 2004), 1413-57. DOI:
10. 1000//cppaa.... 20042.
[34]
W. Ren, X. Cao, J. Pan, X. Guo, W. Zuo, and M.-H. Yang. "Image Deblurring via Enhanced Low-Rank Prior". In: IEEE Transactions on Image Processing 25.7 (2016), pp. 3426 03076.337..03 DOI: 10. 1109/TIP.
2016 5. 25 62.
[35]
Li et al. "An Efficient Iterative Cerebral Perfusion CT Reconstruction via Low-Rank Tensor Decompo- sition With Spatial-Tempone Temporal Total Variation Regularization". In: IEEE Transactions on Medical Imaging
38.2 (2019), pp. 160-370. DOI: 10. 1109/ 1109/MMI.... 2018. 8. 2865198.
[36] A. Spantini, A. Solonen, T. Cui, J. Martin, L. Tenorio, and Y. Marzouk. Optimal low-rank approximations
of Bayesian linear inverse problems. 2015.
[37] J. Assländer, M. A. Cloos, F. Knoll, D. K. Sodickson, J. Hennig, and R. Lattanzi. "Low rank alternating direction method of multipliers reconstruction for MR fingerprinting". In: Magnetic Resonance in Medicine 79.1 (Mar. 2017), pp. 83-96. DOI: 10. 1-111002/mmmm 26629.
[38] J. Liu, Y. Sun, X. Xu, and U. S. Kamilov. Image Restoration using Total Variation Regularized Deep Image Prior. 2018.
[39] Z. Li. X. Xu, Hu, J. Fessler, and Y. Dewaraja. "Reducing SPECT acquisition time by predicting missing projections with single-scan supe-supe coordinate based learning". In: Journal of Nuclear Medicine suppllmemnn 1 (2023), P1014-PI014,
[40] Hu, B. T.-W. Lin, J. H. Vega, and N. R.-L. Tsiang. "Predictive Models of Driver Deceleration and Acceleration Responses to Lead Vehicle Cutting In and Out". In: Transportation Research Record 2677.5
(2023), pp. 02-102. DOI: 10. 1177/03611 98 22222222277
[41] X. Xu, W. Gan, S. V. V. N. Kothapalli, D. A. Yablonskiy, and U. S. Kamilov. CORRECT: A Deep Unfolding Frumework for Correete Quantitative R2* Mapping. 2022.
[42] X. Xu, J. Liu, Y. Sun, B. Wohlberg, and U. S. Kamilov. "Boosting the Performance of Plug-and-Play: Priors via Denoiser Scaling". In: 54th Asilomar Conf. Signals, Systems, and Computers. 2020, pp. 1305-1312. DOI: 10. 11/9 TEEECONF' 594 2020. 9443410.
[43] Y. Sun, Z. Wu, X. Xu, B. E. Wohlberg, and U. Kamilov. "Scalable Plug-andPPla ADMM with Conver- gence Guarantees", " In: IEEE Transactions on Computational Imaging (July 2021). DOT: 10. 1109/
TCI. 20222.21.00 30 3404062
[44] X. Xu. Y. Sun. Liu, B. Wohlberg, and U. Kamilov. "Provable Convergence Plug-and- and-Play Priors With MMSE Denoisers". In: IEEE Signal Processing Letters 27 (2020), pp. 1280-1284, DOI:
10. 11..119///0 1011999sppp... Lsp. 2020. 3006390.
[45] J. Liu, X. Xu, W. Gan, S. Shoushtari, and U. Kamilov. "Online Deep Equilibrium Learning for Reg- ularization by Denoising". - In: Advances in Neural Information Processing Systems. Ed. by A. H. Oh,
A. Agarwal, D. Belgrave, and K. Cho. 2022.
[46] Y. Hu, J. Liu, X. Xu, and U. S. Kamilov, Monotonically Convergent Regularization by Demoising 2022.
[47] J. Liu. Y. Sun, W. Gan, X. Xu, B. Wohlberg, and U. S. Kamilov. "Stochastic Deep Unfolding for Imaging Inverse Problems". In: ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2021, pp. 13999 DOI: 10 1109/ CCASSP39728 2021 9414332.
[48] Cascarano, A. Benfenati. U. S. Kamilov, and X. Xu. Constrained Regularization by Denoising with Automatic Parameter Selection. 2024.
[49] A. H. Al-Shabili, X. Xu, Selesnick, and U. S. Kamilov, Bregman Plug-and-Plee Priors. 2022.
[50] K. Jin, M. T. McCann, E. Froustey, and M. Unser. "Deep convolutional neural network for inverse problems in imaging". In: JEEE Transactions on Image Processing 26.9 (2017), pp. 4509-4505544 4522.
[51] A. Lahiri, G. Maliakal, M. L. Klasky, A. Fessler, and S. Ravishankar. "Sparse-view cone beam CT reconstruction using data-consistent supervised and adversarial learning from scarce training data". In:
IEEE Transactions on Computational Imaging 9 (2023). pp. 13-28
[52] M. Sonogashira, M. Shonai, and M. liyama. -Hiih-Rcsolutton Bathymetry by Deep-Leaming-- -aasd Image Superresolution" In: PloS One 15.7 (2020), =0235487-20235487. DOI: 10. 1371 jourraa pone. 0235487.
[53] E. Whang, D. McAllister, A. Reddy, A. Kohli, and L. Waller. "SeidelNet: An Aberration-Infonn Informed Deep Learning Model for Spatially Varying Deblurring". In: SPIE. Vol. 12438. 2023, 124380Y-124380Y- DO1: 1.07/111/11 2650416.
[54] M. Delbracio and P. Milanfar. Inversion by Direct Iteration: An Alternative to Denoising Diffission for Image Restoration. 2024.
[55] G.-H. Liu, A. Vahdat, D.-A. Huang, E, A. Theodorou, W. Nie, and A. Anandkumar. FSB: Image-to-Image Schrödinger Bridge. 2023.
12

---

[56] HL Chung, J. Kim, and J. C. Ye. Direct Diffision Bridge using Duta Consistency for Inverse Problems. 2023.
[57] G. Cardoso, J. E. Idrissi, S. L.. Corff, and E. Moulines. Monte Carlo guided Diffusion for Bayesion linear inverse problems. 2023.
[58] H. Chung, B. Sim, Ryu, and C. Ye. "Improving Diffusion Models for Inverse Problems using Manifold Constraints", In: Advances Neural Information Processing Systems. Vol. 35. 2022, pp. 25683- 25696.
[59]
Efron. "Tweedie's formula and selection bias". In: Journal of the American Statistical Association 106.496 (2011). pp. (602-1614.
[60]
"Results of 2016 Low Dose Grand Challenge". English (US). In: Medical physics 44.10 (Oct. 2017), e339-e352. DOE: 10. 1002 /mp 12345.
H. McCollough
[61]
Liu, P. Luo, Wang, and X. Tang. "Deep Learning Face Attributes the Wild". In: Proceedings of International Conference on Computer Vision (ICCV). Dec. 2015.
[62]
Z. Wang. A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. "Image quality assessment: From error visibility to structural similarity". In: IEEE Transactions on Image Processing 13,4 (Apr. 2004), pp. 600 612.
[63]
O. D. Team. ODL: Operator Discretization Library. httpps: ooollloopppp github io edl
gui wetp:/eomtttt_geeeeeeeeeeeee
ht.l. Accessed: April 2024. 2022.
[64] T. Hong, L. Hermandez-Garcia, and J. A. Fessler. "A Complex Quasi-Newton Proximal Method for Image
Reconstruction in Compressed Sensing MRI". In: JEEE Transactions Computational Imaging 10 (2024), pp. 372-384. DOI: 1119/ttttiiii 202...33644390
[65] O. Ronneberger, P. Fischer, and T. Brox. "U-Net: Convolutional Networks for Biomedical Image Seg- mentation". In: ArXiv abs/1505.0453 (2015).
[66] J. L. Rumberger. X. Yu, P. Hirsch, M. Dohmen, V. E. Guarino, A. Mokarian, L. Mais, J. Funke, and Kainmueller. How Shift Equivariance Impucts Metric Learning for Instance Segmentation. 2021. [67] J. Gu, S. Zhai, Y. Zhang, M. Susskind, and N. Jaitly. "Matryoshka diffusion models". In: Proc. Intl. Conf. on Learning Representations. 2024.
[68] He, Yang, H. Chen, Cun, Xia, Y. Zhang, X. Wang, R. He, Q. Chen, and Shan. "ScaleCrafter: tuning-fire free higher- resolution visual generation with diffusion models", In: Proc. Intl. Conf. on Learning Representations. 2024.
[69] H. Chung, B. Sim, and J. C. Ye. Come Clome-Closer-Dipp Closser-Difiiss-Fasttee Accelerating Conditional Diffusion Models for Inverse Problems through Stochastie Contraction, 2022.
[70] J. Besag. "Spatial Interaction and the Statistical Analysis of Lattice Systems", In: Journal of the Royal Statistical Society 36.2 2 (1974), pp. 192-236.
13

---

A Appendix supplemental material
This is the appendix for the paper "Learning Image Priors through Patch-based Diffusion Models for Solving Inverse Problems," presented at NeurIPS 2024.
A.1 Additional inverse problem solving experiments
Figures A.I, A.2, A.3, and A.4 show additional inverse problem solving results.
Fig, A.I shows additional example slices for CT reconstruction from 20 views.
Fig, A.2 shows additional example slices for CT reconstruction from 8 views.
Fig. A.3 shows additional examples of image deblurring of face images.
Fig, A.4 shows additional examples of superresolution of face images.
14

---

FBP
ADMM-TV
Whole image
PaDIS
Ground truth
3
Figure A.I: Additional results of 20 view CT reconstruction for 7 different test slices.
15

---

FBP
ADMM-TV
Whole image
PaDIS
Ground truth
Figure A.2: Additional results of 8 view CT reconstruction for 7 different test slices.
16

---

Measurement
ADMM-TV
Whole image
PaDIS
Ground truth
VO
VO
VO
VO
Figure A.3: Additional results of deblurring with Gaussian noise (a 0.01).
17

---

Measurement
ADMM-TV
Whole image
PaDIS
Ground truth
OO
VO
VO
VO
Figure A.4: Additional results of superresolution with Gaussian noise (a 0.01).
18

---

A.2 Ablation studies
We performed four ablation studies to evaluate the impact of different parameters on the performance of our proposed method. Similar to Table 1, we ran the experiments all the images in the test dataset and computed the average metric. Section A.3 shows visualizations of these studies.
Effect of patch size. We investigated the effect of the patch size P used at reconstruction time for the 20-view CT reconstruction problem. We continued to augment the training with smaller patch sizes when possible as to be consistent with the main experiments (patch size of 56 but also trained with patch sizes of 32 and 16), while using the same neural network architecture. Different amounts of zero padding were needed for each of the experiments per (3). App. A.4 provides the full details. At reconstruction time, the same patch size was used throughout the entire algorithm. Using a "patch size" of 256 corresponds training a diffusion model on the whole image (without zero padding).
Table 4 shows that careful selection of the patch size is required to obtain the best results for a given training set size. If the patch size is too small, the network has trouble capturing global information across the image. Although the positional information helps in this regard, there may be some inconsistencies between patches, so the learned image prior is suboptimal although the patch priors may be learned well. At the other extreme, very large patch sizes and the whole image diffusion model require more memory train and run. The image quality drops in this case limited training data prevents the network from learning the patch prior well.
Table 4: Effect of patch size P CT reconstruction
Table 5: Dataset size effect on CT reconstruction
Dataset size
Patches PSNRT
Whole image
P PSNR1
SSIM
SSIM
PSNR1
SSIM 1
8
32.57
0.844
144
32.28
0.841
29.12
0.804
16
32.57
0,829
288
32.43
0.837
31.09
0.829
32
32.72
0.853
576
33.03
0.846
31.81
0,835
56
33.57
0.854
1152
33.01
0.849
31.36
0.834
96
33.36
0.854
2304
33.57
0.854
32.84
0.835
256
32.84
0.835
Effect of training dataset size. A key motivation of this work is large-scale inverse problems having limited training data. To investigate the effects of using small datasets on our proposed method, compared to standard whole image models, we trained networks on random subsets of the CT dataset. Table 5 summarizes the results. Crucially, although the reconstruction quality tends to drop as the dataset size decreased for both the patch-based model and the whole image model, the drop is much more sharp and noticeable for the whole image model, partieularly when the dataset very small. This behavior is consistent with the observations of previous works where large datasets consisting of many thousands of images were used train traditional diffusion models from scratch.
Effect of positional encoding. High quality image generation via patch-based models that lack positional encoding information would be impossible, as no global information about the image could be learned at all. We demonstrate that positional information is also crucial for solving inverse problems with patch-based models. We examined the results of performing CT reconstruction for trained networks without positional encoding encoding. According to [69], when solving inverse problems in some settings, it can be beneficial to initialize the image with some baseline image instead of with pure noise (as is traditionally done). To allow the network that did not learn positional information to possibly use a better initialization with patches roughly in the correct positions, we also ran experiments by initializing with the baseline. Table 6 shows that in both cases, the network completely failed to learn the patch-based prior and the reconstructed results were very low quality. Hence, positional information is crucial to learning the whole image prior well.
input compared to networks with positional
Sampling methods. One benefit of our proposed method it provides a black box image prior for the entire image that can be computed purely through neural network operations image
19

---

patches. We demonstrate the versatility of this method by pairing a variety of different sampling and inverse problem solving algorithms with our patch-based based image prior, along with comparisons with a whole-image prior. The implemented sampling methods include Langevin dynamics [1] with gradient descent term for enforcing data fidelity step and the predictor-correcto method for solving SDEs [19]. Since we observed better stability and results with Langevin dynamics, we also combined this sampling method with nullspace methods that rely on hard constraints [7] and DPS [5]. To use the same neural network checkpoint across these implementations, variance exploding SDE [2] method as the backbone for both training and reconstruction. DPS [5] and DDNM [7] were originally implemented with networks trained under the VP-SDE framework; here, we implemented those methods with the VE-SDE framework. Table shows that generally. VE-DPS performed the best and that the patch-based method consistently outperformed the whole image method. However, the patch-based method still obtained reasonable results for all the implemented methods, showing that the learned image prior is indeed flexible enough be paired with variety of sampling algorithms. App. A.4 provides more details about the implemented algorithms.
used the
Table 7: Dataset size effect on CT reconstruction
Table 6: Positional encoding effect for CT reconstruction
Method
Patch-based
Whole image
Metric
PSNR1
SSIM f
PSNRT
SSIM f
PSNR
SSIM
Langevin dynamics
33.03
0.846
30.92
0.813
no position enc.
23.25
0.459
Predictor-corrector
32.35
0.820
18.95
0.149
no position+init
24.51
0.518
with position
33.57
0.854
VE-DDNM
31.98
0.861
29.49
0.830
VE-DPS
33.57
0.854
32.84
0.835
20

---

A.3 Ablation study images
Figure A.5 shows the results of applying PaDIS to two example test images with different patch sizes. The main results, i.., those shown in Table 1, used P == 56. For some of the other patch sizes, some artifacts can be seen in the images. Namely, the smooth parts of the image become riddled with "fake" features for small patch sizes and some of the sharp features become more blurred. The fake features in the right half of the image in the top row are especially apparent when applying the whole-image model. The runtime for different patch sizes were fairly similar, with P = 8 taking notably longer than the others due to the large number of patches required. The image size for these experiments was small enough that the score function of all the patches could be computed in parallel; however, for larger scale problems such as high resolution 2D images or 3D images, large patch sizes become infeasible due to memory constraints.
P=8
P= 16
P=56
P=96
Whole image
Ground truth
Figure A.5: Results of PaDIS for 20 view CT reconstruction with different sized patches.
Figure A.6 shows the results of applying our proposed method and the whole image diffusion model to 20-view CT reconstruction for varying sizes of the training dataset. The image quality for PaDIS remains visually consistent as the size of the training dataset shrinks, as each image contains thousands of patches which helps avoid overfitting and memorization. However, the drop in quality for the whole image model is much more visible: in particular, the sharp features of the image are lost and the image becomes blurry. Hence, for applications where data even more limited, such as medical imaging, our method can potentially have greater benefit.
Size= 144
Size=288
Size=576
Size=1 1 1152
Size=2304
Ground truth
Figure A.6: Results for 20 view CT reconstruction with different dataset sizes. Top row shows recon performed by PaDIS; bottom row shows recon performed with the whole-image model.
21

---

Figure A.7 demonstrates the importance of adding positional encoding information into the patch- based network on two different images. When positional information is not included, the network simply learns a mixture of all patches, resulting in very blurry image with many artifacts resulting from the data fidelity term. Even when better initialization of the image is provided, the same blurriness remains.
No position
FBP init
With position
Ground truth
Figure A.7: Results of PaDIS for 20 view CT reconstruction for different positional encoding methods.
22

---

Figure A.8 shows the results of using our proposed method compared with the whole- diffusion model with different sampling and inverse problem solving algorithms. The predictor-correcte algorithm fails completely when using the whole-image model, indicating that this model could not be well-trained this limited data setting. Quantitatively, DPS performs the hest for PaDIS; visually, all of the methods obtain reasonable results, although some more minor artifacts are present in the first four methods. Nevertheless, this shows that the patch-based prior is flexible and can be used with variety of existing algorithms.
Predictor-corrector
Langevin
DDNM
DPS
Ground truth
Figure A.8: Results PaDIS for 20 view CT reconstruction using different sampling and inverse problem solving algorithms. Top row is with PaDIS and bottom row is with the whole-image model.
A.4 Experiment parameters
We trained the patch-based networks and whole-image networks following [14]. Since images scaled between 0 and 1, we chose a maximum noise level of = 40 and a minimum noise level of 0.002. We used the same UNet architecture for all the patch-based networks consisting of base channel multiplier size of 128 and I, 2, 2, and 2 channels per resolution for the four layers. We also used dropout connections with a probability of 0.05 and exponential moving average for weight decay with a half life of 500K patches to avoid overfitting. Finally, the learning rate was chosen to be 2 10-1 and the batch size for the main patch size was 128, although batch sizes of 256 and 512 were used for the two smaller patch sizes. The entire model had around 60 million weights. For the whole image model, we kept all the parameters the same, but increased the number of channels per resolution in the fourth layer to 4 so that the model had around I 11 million weights. The batch size in this case was
For image generation and solving inverse problems, we used a geometrically spaced descending noise level that was fine tuned to optimize the performance for each type of problem. We used the same set of parameters for the patch-based model and whole image model, as follows:
CT with 20 views: Omax = 10, miin = 0.002
CT with 8 views: Tmax = 10, min = 0.003
Deblurring: Tmax 40, o min 0.005
Superresolution: Onax 40, ouin 0.01.
The ADMM-TV method for linear inverse problems consists of solving the optimization problem
argmax 22 -
Axl TV()),
(A.1)
where TVxx)) represents the L1 norm total variation of æ. and the problem is solved with the alternating direction method of multipliers. For CT reconstruction, deblurring, and superresolution, we chose À to be 0.001, 0.002, and 0.006 respectively.
23

---

Ablation study details. For each patch size, we trained with the main patch size along with smaller patches whenever possible. However, since we did not modify the network architecture, and the architecture consists of downsampling the image 3 times by factor of 2, it was necessary for the input dimension to be a multiple of Furthermore, we followed a patch scheduling method similar to that of the main experiments unless otherwise noted. Finally, to avoid excessive zero padding, for larger patch sizes, we used patch sizes that were smaller than the next power such that the main image could still be fully covered by the same number of patches. The details are as follows.
P 8: This trained only with this patch size
smaller sizes could be used.
P = 16: Trained with patch sizes of 8 and 16 with probabilities of 0.3 and 0.7 respectively.
= 32: Trained with patch sizes of 8, 16, and 32 with probabilities of 0.2, 0.3, and 0.5 respectively.
P 56: Trained with patch sizes of 16, 32, and 56 with probabilities of 0.2, 0.3, and 0.5 respectively. Zero padding width was set 5 56 256 24.
P P= 96: Trained with patch sizes of 32, 64, and 96 with probabilities of 0.2, 0.3, and 0.5 respectively. Zero padding width was set 3 96 256 32.
A.5 Comparison algorithms
This section provides pscudocode for the implemented alternative sampling algorithms whose results shown in Table 7. Here, for brevity, show the versions using the whole-image diffusion model; the versions with our proposed method are readily implemented by computing s = s(x, o,) through the procedure illustrated in Alg. 1.
Algorithm A.1 Image Recon via Langevin Dynamies
Require: σi < 02 < ... or. > 0, > y
Initialize x: ~ (N(
ior
Sample = (0, 1)
Set a; = 6 of
Apply neural network to get D Dz(x,
Set 8 = (D - oo?
Set æ to x + GA (y - A(=))
Set to + 2 + va²z
end for Return x.
Algorithm A.2 Image Recon via Predictor-Correctte Sampling
Require: < <g <.., < <or, 0, G 0. r.
Initialize x ~ NN000
T
Set x: to x: (a - 0 )0 (2, 01t1)
Set æ to r + G (y A(x))
Sample z (),
Set to + V off1 07
Sample z ~ No.
Set 2r =a(a,o)]]|
|||||
Set s = so [ o4))
Set to x + 6, + v /2,,
Set x: to x + GAA (y - A(x))
end for
Return æ.
In cases, we used the same noise schedule as the main 20 view CT reconstruction experiment. For Langevin dynamics and DDNM, set 1; the final results were not sensitive with respect to this
24

---

Algorithm A.3 DDNM
Require: <2 < <
<or, e > 0, Gi y
Initialize ~ ~N(00,
for :
Sample
Set a; = c. - o;
Apply neural network to get D o,(
Set D A'y + D A²A(D)
Set s D x
Set z: to x + a + V
end for Return æ.
parameter. For Langevin dynamics and predictor-corrector sampling, took = - A(x))12 similar to the step size selection of DPS. Following the work [19], we chose r = 0.16 for PC- sampling. The same parameters
used for the patch-based and whole image methods.
Table 8 shows the average runtimes of each of the implemented methods when averaged across the test dataset for 20 view CT reconstruction.
Table 8: Average runtimes of different methods across images in the test dataset for view CT recon.
Method
Runtime (s) /
Baseline
0.1
ADMM-TV
PnP-ADMM
8
PnP-RED
22
Whole image diffusion
172
Langevin dynamics
98
Predictor-corrector
189
VE-DDNM
105
PaDIS (VE-DPS)
195
25

---

A.6 Markov random field interpretation
Markov random fields (MRF) are tool used to represent certain image distributions and are par- ticularly applicable to patch- patch-based diffusion models. Describing the connection between MRF and this work requires some notation: let = {I, E S} denote the random field, where the index S denotes the sites. The neighborhood system N is defined as . = (N, :8 S). A model for E X a MRF on with respect to the neighhorhood system N if
p( - (xN.), Vo X, E
(A.2)
Therefore, the distribution of each site (normally chosen to be pixel) conditioned on the rest of the pixels depends only on the neighboring pixels.
By the Hammersley- chiiffffrr Clifford theorem [70], such MRF satisfying p(x) > 0 everywhere can also be rewritten as p(æ) = z -U(2), where is a normalizing constant. In this case U(x) is called the energy function and has the form Uxx) = Eeec Vz(æ:), which is a sum of clique potentials (x())
over all all possible cliques. Thus, the score function for MRF model is:
s(x) V logp(x) = VU = VV.(m))
(A.3)
If we let the neighborhood system be the patches of the image, then Ve corresponds the clique potential for the eth patch of an image, and VV((z)) denotes the score function of that patch. Denoting by Ge the wide binary matrix that extracts the pixels corresponding to the eth patch from the whole image, we define Vx = V(G.., c²), where e* denotes the positional encoding method used for the cth patch, and now we simply have one (patch) clique function V. Finally, the overall score function under this model becomes
8(x) = (v.a,G))
(A.4)
where sv c²) - VVV(O, is the shared score function of each of the patches with positional encoding input.
In this work, approximate sv with neural network parameterized by 0, and we use denoising score matching to train the network via the loss function
L(0)= EI
(A.5)
(A.6)
This derivation makes no assumptions on the patches; in particular, this method train the score function would still hold if the patches overlapped for each iteration. However, such overlap would make costly to train the network, as the loss function would need be propagated through the sum over all patches every training iteration. We circumvent this problem by using non-overlapping patches (within a given reconstruction iteration; i.e., our approach only uses patches that "overlap" only across different iterations). For non oon-overlapping patches, the sum can be rewritten as follows:
L(0)
G +.
G.G.e/oo
(A.7)
=
Be- Erandom ellsy (G.z, 0) - G.c/of12
(A.8)
This loss function is much easier to compute, as we can now randomly select patches and perform denoising score matching on the individual patches, opposed to considering the entire image at once, and is equivalent to (A.5).
26

---

A.7 Acceleration methods
Although diffusion models are capable of generating high quality images, the iterative generative process typically requiring around 1000 neural function evaluations (NFEs) [2. 3] is a major disadvan- tage. In recent years, significant work has been done improve sampling speed of diffusion models [14,30 30, 31J. To reuse the same trained network, one can first derive an algebraic relationship between the score function s(r, a) (readily computed using the denoiser network ((r, o) trained via (5)) and the residual function c(x,, t) which is approximated with a neural network in papers such as [3]. The score matching network S2 0) learns map æ + ae to 2; whereas the residual network learns to map 2, = Vonzo v1 - orE to e where 6 (0 T). Hence, we may input va: Eo + </-ai as the noisy image into the denoising score matching network that the correct output becomes Zp. Then the corresponding noise level is o; <T-ai Finally, the outputs of the network must be scaled via (ax) -e(x)/o. Thus, by using this transformation, the network trained via (5) may also be applied to sampling algorithms requiring 6g.
Vau
vor
Using these ideas, we implemented the EDM sampler, second-order solver for SDEs, according to [14], which can produce high fidelity images in 18 iterations (equating to 36 NFEs as each iteration requires two NFEs). We also implemented DDIM [30] using 50 sampling steps. Figure A.9 shows the results using these methods with the proposed patch-based prior along with Langevin dynamics with 300 NFEs. The EDM sampler produces images that have clear boundary artifacts and the images from the DDIM method also have some discontinuous parts. This behavior is due to the stochastic method of computing the patch-based prior: according to Algorithm we randomly choose integers i and with which to partition the zero padded image and compute the score function according to this partition. Hence, accelerated sampling algorithms that attempt remove large amounts of noise at each step tend to fare worse at removing boundary artifacts. This limitation of our method makes difficult accelerated sampling algorithms, is direction for future research.
Figure A.9: Generation of CT images with various acceleration algorithms. Top row shows generation with the EDM sampler [14], middle row uses DDIM [30], bottom row our proposed method.
27

---

A.8 Additional figures Figure A.I0 shows the visual results of the extra inverse problems in Table 2.
1200
800
Figure A.10: Results of extra inverse problem experiments. From top to bottom: 60 view CT, fan beam CT. heavy deblurring. From left to right: baseline, ADMM-TV, whole image diffusion, PaDIS, ground truth.
To further explore the different methods of assembling patches to form the whole image. we looked unconditionally generated CT images according the methods of [23] and [66]. Unlike with our proposed method, for these two methods, the patch locations are fixed throughout all of the timesteps. Furthermore, overlapping patches must used to avoid boundary artifacts. The main difference the way in which the methods handle the overlapping pixels: [66] overrides the overlapped areas with the new patch update while [23] averuges over the overlapped area. We use the same network checkpoint trained CT images the main experiments and patch size of 56 with overlap of for the experiments. Figure A. shows the generated images: while both methods are able to avoid boundary artifacts, the overall structure of the generations of lower quality than the images generated by our proposed method (see Table 4). This suggests that our proposed method most effectively combines the patch priors to form prior for the entire image.
Figure A.11: Unconditionally generated CT images using patch stitching [66] for the top row and patch averaging [23] for the bottom row. Compare this to the images generated by our proposed method in Figure 4.
28

---

Figures A.12 and A.13 show the PSNR of each individual image in the test dataset when using the whole-image model versus our proposed method. The plots show that our method exhibited reasonably consistent performance improvements over the test dataset.
PSNR Comparison for Deblurring
Test images
36
34
32
PaDIS
30
28
Whole image
Figure 12: Comparison between PSNR of deblurring between whole-image model and proposed method for each image in the test dataset.
PSNR Comparison for Superresolution
Test images
34
32
PaDIS
30
28
Whole image
Figure A.13; Comparison between PSNR of superresolution between whole-image model and proposed method for each image the test dataset.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction contain claims that are expounded upon in the remainder of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The conclusion briefly states a couple of the limitations of the work. The appendix (and particular section A.7) delves into these limitations more.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer: [Yes]
Justification: The assumptions that are made throughout Section are clearly stated. The paper does not contain any theorems, but Section A.6 provides theoretical explanation for some of the algorithms used in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The pseudocode for the algorithms as well as hyperparameter selection and datasets used are completely outlined in the main body and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
31

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have made the code used to reproduce the main experimental results available and the link is provided in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https: sttops:/nnigg htt////iis..ee cc/
public collic//e gui des /CodeSuomii ssi assionooliio Pillicyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nipps cc/ wtnpsscccpbbicc..ooo coublic/guides CodeooeSSuu CoodSSubmls Submi ssiisssinn cesssonPoliee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details of the experiments have been provided in Section and A.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not provide error bars for the main results because the results consist of average across many different images with varying quantitative metrics, so the uncertainty would not be meaningful. It would be computationally expensive to run the algorithms for the same images multiple times obtain uncertainty measurements.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources and runtime for the experiments are specified in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http//euurpp.ee//ee/e http.//pc..ccctttscssceeeeeeeee.
Answer: [Yes]
Justification: We have read the Code of Ethics and checked that the research conducted in the paper conforms to it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the socictal benefits (reduced CT dose) and risks (hallucinations and disinformation) in the Conclusion section.
Guidelines:
33

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
The models used in the paper can generate images, but the datasets used have been checked be safe. Since the models can only generate images similar the datasets on which they have been trained, the images that can be generated should also conform to this safety.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all works and datasets that this paper uses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
34

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers saperswiiee thooe com/ dat datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper describes the data and methods used, but we have not had the time curate the code and data into shared repo.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
35

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "BGrGd30CiS", "reviewer": "Reviewer_mcvX", "summary": "The authors present an approach for tile-based training and prediction of diffusion models applied for inverse problem posterior sampling.\nThe core idea is that training is done with random patches and during generation the authors use a differently shifted non-overlapping tiling grid for each iteration of the process.\nThe authors additionally provide the x- and y-pixel-coordinates to the network by encoding them in extra channels and concatenating them.\nTheir approach allows them to generate images without Stichting artefacts.\nThe random tiling during training can be seen as data augmentation and the authors show that this enables their method to be trained on smaller datasets.", "questions": "It is not correct that using overlapping tiles would be \"*inconsistent with the theory of diffusion models*\".\nIn general, a tiling scheme for unets can be implemented, such that the stitched tiles are identical to the result of processing the image as a whole.\nThis can be achieved by using overlapping tiles (with the correct shift) and disregarding the areas close to the border in the outputs that are influence by padding. I believe a discussion of this can be found in [1].\nWhat would prevent us from applying this approach with a diffusion model in each step? The network could still be trained using patches.\n\nWhile such a tiling approach would cost additional computation time, since it requires overlapping patches, it should theoretically produce guaranteed stitch artefact-free outputs.\nDo the authors agree, that this well established approach should ideally be a baseline or at least be discussed?\nThe authors could show that their method produces comparable results at reduced computational cost.\n\n[1]: Rumberger, Josef Lorenz, et al. \"How shift equivariance impacts metric learning for instance segmentation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "limitations": "I am missing a discussion on the cost of the proposed patch scheme regarding the ability of the network to model long range correlations. See weaknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Improving the memory requirement for diffusion model is an important problem. Many applications need to process large images in a coherent way, while avoiding stitching artefacts.\n* I appreciate the fact that the proposed training scheme enables training with less data. And that the authors validate this in an experiment.\n* The authors show that their method does not depend on the particular smapling scheme or network architecture. I appreciate the generality of the approach.", "weaknesses": "* My main criticism is regarding the motivation of the problem. \nThe authors write: \n\"*Directly using overlapping patches would result in sections of the image covered by multiple patches to be updated multiple times, which is inconsistent with the theory of diffusion models.*\" The question of how tiling and stitching can be applied for unets, such that the result is equivalent to processing the image as a whole has been explored before. See questions section for details.\n\n* The proposed tiling comes at a cost: It limits the range of correlations that can be captured by the diffusion model. \nThis is visible in Figure 4, where the generated images show no stitching artefacts, but also produce nonsensical large scale anatomy. \nI am missing a discussion of this aspect.\nIn the posterior samples, the effect is not visible, since the input image contains enough information such that long range correlations are not relevant.\nI believe this would be a problem for inverse problems where the input image contains less information, i.e., when the noise is very severe, or for super resolution with a more extreme resolution factor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. \n\n**Comment: Tiling scheme for unets can be implemented with overlapping patches**\n\nWe implemented the method provided in the above reference [1] while using the same trained network, hyperparameters, and DPS for inverse problem solving. The network consisted of 3 layers of upsampling/downsampling and each layer involved pooling with a factor of 2. The largest patches that the network was trained on had size 56. To satisfy the hypotheses of the paper, the overlap between patches was to be a multiple of 8; we set it equal to 8 to minimize the number of patches for image partitioning. Hence each 256x256 image was divided into 36 overlapping patches, compared to 25 for PaDIS. \n\nTable 6 shows the results of using this method under the name Patching Stitching. For the shown inverse problems, the method obtains reasonable results but performs worse than our proposed method. Despite this, the reconstructed images did not appear to exhibit any boundary artifacts along patch boundaries, showing that the method of [1] worked in that aspect. The increased number of patches necessary for that approach increased the runtime of the reconstruction algorithm by approximately 30%. \n\nWe also examined the unconditionally generated images using the method of [1] which are shown in Figure B.2. Although these images exhibited relatively smooth features without any clear boundary artifacts between patches (unlike the clear artifacts visible in the middle row of Figure 4 generated by naive patch stitching), the overall structure was highly inconsistent with the CT images in the dataset. This is in contrast with the images generated by PaDIS as shown in the bottom row of Figure 4. Therefore, although [1] can result in smooth images, the lack of patch shifting means that the learned image prior differs from the one in Eq. (3), resulting in unrealistic looking generated images, which indicated the underlying data distribution cannot be well captured. The significantly worse prior learned by this method is reflected in the worse results for inverse problem solving and especially when the measurements are highly compressed.\n\nThere are two aspects of the UNet used in our application that likely caused that patch stitching method to fail to learn the prior well. Firstly, our diffusion UNet takes in an additional scalar input indicating the noise level of the noisy image being input into the network. This scalar input is processed through a sinusoidal positional encoding before being embedded into the layers of the UNet through an attention mechanism. Secondly, our network takes in the positional embedding of the location of the patch via concatenation along the channel dimension. Thus, our network learns the score function of patches while also incorporating the location of the patch, making it different from traditional UNets used for segmentation [1]. \n\n**Comment: Patches limit the ability of the network to learn long range correlations**\n\nLearning longer range correlations within an image is assisted by the method of using positional encoding of patches as an input to the network: it allows our network to learn a different distribution of patches at different locations in the image. Thus, provided that the location of the central object to be imaged is in a relatively consistent location (as is the case for CT scans or human faces), the network can learn that, for instance, the spine of the CT scan is typically around the middle bottom section of the image. Such learning is consistent with the generated results in Figure 4. Nevertheless, generating whole images with realistic large scale anatomy is challenging, and we show the generation results to demonstrate that they appear somewhat reasonable while emphasizing the focus on solving inverse problems.\n\nFigure 5 contains some examples of reconstructed images from 8 view CT, a very compressed sensing problem. Normally, to reconstruct a 256x256 image would require (pi/2*256)=402 views, so the compression is a factor of 50. In this case, the images show reasonable large scale anatomy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "en2dShY8oI", "reviewer": "Reviewer_YkyD", "summary": "This manuscript discusses diffusion models for inverse problems. The authors discuss using image patches of the image to improve computational bottlenecks and overcoming the lack of sufficient data in training appropriate surrogate neural network priors for the inversion task. The authors discuss details of their proposed method and illustrate the advantages of their methods on various tasks including CT, deblurring, and superresolution.", "questions": "See Weaknesses.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This manuscript is well-written and structured, making it easy for readers to follow the presented ideas. The authors ground their work in existing literature and reference relevant papers in this field.  Data-driven approaches for inverse problems have shown significant advances and this manuscript contributes to this field.", "weaknesses": "This work is partly incremental and heavily relies on various previous and cited publication, e.g., [12,18,19]. Furthermore,  I assume computational costs for the solution of the inverse problem are extremely large since just a stochastic gradient approach must be utilized in the inversion process (see algorithm 1). These are typically prohibitive for large-scale inverse problems removing the advantages of the learned data-driven prior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. \n\n**Comment: Work is partly incremental, heavily relies on cited publications**\n\nThe papers [12] and [18] apply diffusion models to solve 3D reconstruction problems, whereas the proposed method performs experiments on 2D reconstruction problems using a patch-based prior. [19] uses the predictor-corrector method for solving medical imaging problems, which we compared to in Table 5. We revised the labels in Table 5 to make this more clear.  The method of diffusion inverse problem solving most closely related to our method is DPS [5]. However, the most significant contribution of the proposed method is a patch-based image prior that requires only patch inputs to a neural network and can be paired with any diffusion inverse solving algorithm, as illustrated in Table 5. \n\n**Comment: Computational costs are large and requires a stochastic gradient approach**\n\nWe acknowledge that the computational costs of the proposed method will exceed that of more traditional reconstruction methods, as is true for almost all diffusion based methods, as a tradeoff for achieving better image quality. However, the computational cost of the proposed method is similar to other diffusion based methods. The average reconstruction time per image in seconds for the methods in Tables 1 and 5 are shown below for 20 view CT. Notably, our proposed method takes only slightly more time than the approach using diffusion models trained on entire images while greatly reducing the memory needed and improving the result.  We will add this table to the supplement.\n\nBaseline: 0.1\n\nADMM-TV: 0.7\n\nWhole image diffusion: 172\n\nPaDIS (VE-DPS): 195\n\nLangevin dynamics: 98\n\nPredictor-corrector: 189\n\nVE-DDNM: 105\n\nThe stochastic approach taken in Algorithm 1 allows the runtime of the algorithm to be similar to other diffusion methods while eliminating artifacts that would otherwise persist between boundaries of patches. The approach is similar to the one taken in the paper below: instead of computing the score function multiple times each iteration, stochastically choose one of them to compute each iteration. Over the course of hundreds of iterations throughout the reconstruction process, the stochastic approximation of the score function becomes more accurate. Furthermore, this approach does not sacrifice the advantages of using this data-driven prior, as Figure 4 shows that our proposed method can still be used to unconditionally generate fairly realistic images.\n\nS. Lee, H. Chung, M. Park, J. Park, W.-S. Ryu, and J. C. Ye. “Improving 3D imaging with pre-trained perpendicular 2D diffusion models”. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023, pp. 10710–10720.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UHHo9bd77V", "reviewer": "Reviewer_P2vb", "summary": "This paper proposes a patch-based diffusion model for inverse problems, such as CT reconstruction and natural image deblurring. This method divides images into patches, reducing the size of the input data fed into the network, thereby decreasing memory consumption.", "questions": "1. I suggest the authors make more improvements for image restoration tasks (inverse problems) to control the reliability of the generated images and reduce artifacts.\n2. it is necessary to compare with more SOTA methods to obtain a more comprehensive evaluation.", "limitations": "1. Due to the severe artifacts observed in the visual results (CT reconstruction) and the limited number of comparison methods, the experimental results are hardly convincing.\n2. The innovation is very limited. It is based on existing diffusion models and merely applied to the domain of inverse problems (image restoration tasks).", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This method provides a feasible approach for dividing and merging patches to reduce boundary artifacts for diffusion models.", "weaknesses": "1. The innovation is limited. The main contribution of this paper is the application of patch diffusion [1] to inverse problems. However, the experimental results are neither promising nor convincing.\n\n[1] Wang, Zhendong, et al. \"Patch diffusion: Faster and more data-efficient training of diffusion models.\" Advances in neural information processing systems 36 (2024).\n\n2. The CT images presented by the author in the paper and appendix are very blurry and not displayed with the correct window level and width, making it impossible to discern imaging details and lacking clinical significance.\n\n3. Even though the CT images are very blurry, it is still evident that all the reconstructed CT images exhibit significant numbers of image artifacts (incorrect organ structures) compared to the ground truth. This is completely unacceptable for medical images.\n\n4. The comparison methods are very limited, with only ADMM-TV, which is a very old method. There are numerous methods for CT reconstruction [2-4], natural image deblurring [5] and super-resolution [6] that the author did not compare with at all.\n\n[2] Shen, Liyue, John Pauly, and Lei Xing. \"NeRP: implicit neural representation learning with prior embedding for sparsely sampled image reconstruction.\" IEEE Transactions on Neural Networks and Learning Systems 35.1 (2022): 770-782.\n\n[3] Wu, Qing, et al. \"Self-supervised coordinate projection network for sparse-view computed tomography.\" IEEE Transactions on Computational Imaging 9 (2023): 517-529.\n\n[4] Chung, Hyungjin, et al. \"Solving 3d inverse problems using pre-trained 2d diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[5] Tang, Xiaole, et al. \"Uncertainty-aware unsupervised image deblurring with deep residual prior.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n[6] Wang, Longguang, et al. \"Unsupervised degradation representation learning for blind super-resolution.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. \n\n**Comment: Innovation is limited**\n\nThe main innovation of our method is a method to formulate a diffusion based image prior from solely the patches of the image. Diffusion models are known for requiring a large amount of memory for training and inference and extending them for large scale images is a challenging problem. This work illustrates how image priors for very large images can be learned by learning priors of patches, a much less memory intensive task. This is unlike the work of [1], which ultimately still requires training on whole images as well as inputting the whole image into the network to generate images. We also demonstrate in Table 3 that our proposed method can learn a reasonable prior for dataset sizes much smaller than is normally used for training diffusion models and the advantage over whole image diffusion models becomes more pronounced for small datasets. Finally, unlike previous patch-based diffusion model papers such as [16] and [17] that can only be used for generating images, we show how our method learns a full image prior from only image patch training that can be coupled with most diffusion inverse problem solving algorithms, which is not a trivial extension based on [16] and [17]. \n\n**Comment: CT images are blurry and displayed with the wrong window level**\n\nWe displayed CT images corresponding to our new CT experiments in Figure B.1 using a narrower window size of 800 to 1200 HU and with higher resolution. These images show better contrast between different organs and are more useful for obtaining a clinical diagnosis. In the revision, we will redisplay all the CT reconstruction images with this higher level of contrast. \n\n**Comment: CT images contain significant number of image artifacts**\n\nThe authors acknowledge that the images obtained by the generative models investigated including the proposed method for 20 view CT reconstruction show some hallucinations and artifacts. This is a natural consequence of using extreme compressed sensing with ultra-sparse views: normally, to reconstruct a 256x256 image requires (pi/2*256)=402 views, so for the 20 view experiments, the measurements have been compressed by a factor of 20. Due to this lack of information, it is very hard for any model to perform a diagnostic-quality reconstruction, though our proposed method (and the other diffusion model methods, to a lesser extent) are able to partially fill in this information through learning a strong image prior. The alternative methods that do not learn a prior perform significantly worse in terms of the shown metrics and exhibit severe blurring and artifacts. In clinical settings, it is much more common to perform patient diagnosis with CT scans consisting of hundreds of views. To illustrate this point, we perform experiments with 60 view CT, where our proposed method is able to obtain excellent quality images as shown in Figure B.1: essentially no artifacts are visible. (We show the potential of our proposed method to reconstruct images with ultra-sparse views with a decent image quality, which can be potentially used for other clinical applications such as patient positioning.)\n\n**Comment: Comparison methods are very limited**\n\n[4] is a method that applies 2D diffusion models to solve 3D inverse problems including CT reconstruction, whereas our proposed method applies for 2D inverse problems, so it cannot directly be applied. However, the sampling algorithm that is used is the predictor-corrector sampling algorithm, which we initially compared to in Table 5 and now added a more comprehensive comparison in Table 6. This sampler did not perform as well as the one chosen for PaDIS (DPS) by the quantitative metrics. \n\n[2] and [3] are self-supervised methods for CT reconstruction, which means that network is trained during reconstruction time, substantially slowing down the algorithm. [5] is a deep image prior method that also requires network training at inference time. Furthermore, [2], [3], [5], and [6] are all problem specific methods for which a generalization to the other types of inverse problems would be nontrivial. Our proposed method, along with most of the methods we compared to in Table 6, are easily generalizable methods that can solve a wide variety of inverse problems. Furthermore, the training process for our algorithm need only happen once per dataset, and no network training is required during reconstruction time. Due to these fundamental differences between our method and [2], [3], [5], [6], we believe that comparisons with those methods would not be fair.\n\nNevertheless, to provide a more complete evaluation of our method, we included additional comparisons with plug and play (PnP) methods and other diffusion inverse solvers in Table 6. These methods share the similarity with our method in that network training only needs to be done once per dataset, and the same trained network can be used for different types of inverse problems, allowing for greater flexibility. Table 6 consists of an expanded comparison between various methods. We implemented various diffusion inverse solving methods [1], [7], [19] in conjunction with our patch-based prior. We included two additional patch based methods from [23] and [69] where we applied [23] in an unsupervised way by using the same unsupervised network trained in our proposed method and adding a DPS step during reconstruction. We also implemented two plug and play (PnP) methods by first training denoisers on CT images and the CelebA dataset and then applying these denoisers in an unsupervised way to solve the inverse problems. Optimal hyperparameters for all these methods were found through searching. \n\nIn all cases, our proposed method outperformed the comparison methods. In the revision, we will add visual examples of these methods. These comparisons with several other SOTA methods strengthen the evaluation of our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gx7MD7BCW7", "reviewer": "Reviewer_L95h", "summary": "In this work, the authors propose a novel method for learning efficient data priors for entire images by training diffusion models only on image patches. During inference the authors introduce a patch-based position-aware diffusion inverse solver, which obtains the score function for the whole image through scores of individual patches and their positional encoding, using this as the prior for solving inverse problems. Multiple experiments on CT data as well as on the CELEBA dataset are conducted.", "questions": "I would be interested as to which number of data-samples the patch-based approach outperforms the Vanilla one?\nHow would the proposed approach perform with different forward operators, i.e. different inverse problems on the same datasets? \nWhy do the presented CT reconstructions exhibit such significant hallucinations, and how can this issue be addressed?", "limitations": "Although sparsely addressed in the conclusions, the authors do not give limitations about the proposed approach at all. \nEspecially, the proposed approach seems to have problems concerning hallucinations in the reconstructed CT images.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The idea of including positional information of patches into the diffusive reconstruction process seems novel and promising. Indeed, the authors demonstrate that their proposed approach can compute the score function for entire images without needing to feed the whole image through the network.", "weaknesses": "While the patch-based reconstruction for diffusion models appears promising, the evaluation and experiments presented in the paper require further extension to justify publication. Additional evaluations should be conducted against other models, reconstruction methods, inverse problems, and datasets. Furthermore, a sensitivity analysis should be included to examine how the proposed approach performs with different forward operators. Additionally, many of the presented CT reconstructions exhibit significant hallucinations, which is particularly concerning in the context of medical imaging. Overall, my decision is influenced by the weak evaluation of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. \n\n**Comment: Additional evaluations should be conducted against other inverse problems, sensitivity analysis should be included with different forward operators**\n\nWe conducted more experiments with different forward models: namely 60 view parallel beam CT, 180 view fan beam CT, and deblurring with a larger kernel of size 19x19. The results are shown in Table 7 and further demonstrate that our proposed method outperforms various SOTA methods for a large variety of forward models. Additionally, Table 6 contains a comparison with a wider variety of inverse problem solving methods. These comparisons with several other SOTA methods strengthen the evaluation of our method.\n\n**Comment: CT reconstructions exhibit significant hallucinations**\n\nThe authors acknowledge that the images obtained by the generative models investigated including the proposed method for 20 view CT reconstruction show some hallucinations and artifacts. This is a natural consequence of using extreme compressed sensing with ultra-sparse views: normally, to reconstruct a 256x256 image requires (pi/2*256)=402 views, so for the 20 view experiments, the measurements have been compressed by a factor of 20. Due to this lack of information, it is very hard for any model to perform a diagnostic-quality reconstruction, though our proposed method (and the other diffusion model methods, to a lesser extent) are able to partially fill in this information through learning a strong image prior. The alternative methods that do not learn a prior perform significantly worse in terms of the shown metrics and exhibit severe blurring and artifacts. In clinical settings, it is much more common to perform patient diagnosis with CT scans consisting of hundreds of views. To illustrate this point, we perform experiments with 60 view CT, where our proposed method is able to obtain excellent quality images as shown in Figure B.1: essentially no artifacts are visible. (We show the potential of our proposed method to reconstruct images with ultra-sparse views with a decent image quality, which can be potentially used for other clinical applications such as patient positioning.)\n\n**Comment: What is the number of samples for which the proposed method is better**\n\nWhen looking at PSNR, the number of samples out of the 25 test samples in which the patch-based approach outperformed the vanilla one is as follows: 23 for 20 view CT, 25 for 8 view CT, 20 for deblurring, 16 for superresolution.  We will add this information to the supplement.\n\n**Comment: Authors do not give limitations of the proposed method**\n\nOne limitation of the proposed approach (and all diffusion approaches) is that they tend to be slower than optimization based approaches, plug and play methods, and model-based learning methods. Another limitation of generative modeling approaches is the potential to hallucinate, particularly when the measurements are very compressed and contain little information. This is a limitation of most generative models, as illustrated by the visual examples from the whole image diffusion model for sparse view CT. This problem can be resolved by obtaining more projection views in a CT scan, depending on the application needs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PJgNBRzf67", "reviewer": "Reviewer_wfxx", "summary": "This work introduces a patch-based diffusion modeling approach to efficiently learn image priors that can be used to solve inverse problems. Particularly the model maintains memory and data efficiency due to the patch-based operating scheme. Experiments are demonstrated in both natural and medical image domains to solve various inverse problems (e.g., CT reconstruction, deblurring, etc.) based on priors learned via patches.", "questions": "- Details regarding how the evaluation metrics PSNR and SSIM are calculated are missing (i.e., in RGB domain, or via the luminance in YCbCr domain)?\n\n- The authors' justification on using non-overlapping patches during training is somewhat not clear to me. What exactly is the negative impact of using overlapping patches during training this model?\n\n- In Figure 4, results obtained with [16] are surprisingly bad. Is the implementation modified, or did the authors implement themselves? Is it the best case scenario for this method's performance? Also on a different note, did the authors perhaps try using the patch-based diffusion approach from [23] in an unsupervised manner in this simulation?\n\n- Considering Algorithm 1's sampling loop, looks like the sampling process goes through all denoising steps without skips (sampling takes T steps)? In that case, perhaps inference time comparisons should also be demonstrated comparatively with other methods.\n\n- Visual results (e.g., in Figures 5, 6, 7) should be blended in the compiled PDF not as a PNG/JPEG, but as a high resolution image as a e.g., PDF (more professional looking figure quality needed). Fine differences/details that one should see when zoomed-in can disappear if figures are blended in the paper using a PNG/JPEG type of format, which loses its meaning.", "limitations": "Sufficiently addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Paper presents a very interesting and novel approach to learn image priors with patch-based diffusion models, and empirically demonstrates rigorous results.", "weaknesses": "- Some experimental clarifications are necessary, and writing & quality of figures can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. \n\n**Comment: Details regarding calculations of PSNR and SSIM are missing**\n\nThe PSNR and SSIM of RGB images are calculated in the RGB domain. Data preprocessing consisting of dividing all the RGB values by 255 was done first, so all the reconstructed images have values between 0 and 1. The PSNR and SSIM values were then computed from these images. \n\n**Comment: Justification for using non-overlapping patches is unclear**\n\nUsing overlapping patches during training would significantly increase the computational cost. Section A.6 details the theory behind which the original score matching process (that must be done on the entire image), used to train the network, can be reduced to score matching on individual patches. In particular, in going from equation (A.6) to equations (A.7) and (A.8), the assumption that the patches do not overlap is made to bring the sum out of the norm. If overlapping patches were to be used, it would be necessary to backpropagate through all the terms of the sum during the training. On the other hand, when nonoverlapping patches are used, we can perform score matching on individual patches and only the loss of these individual patches needs to be backpropagated through the network.\n\n**Comment: Results obtained by [16] are surprisingly bad**\n\nIn Figure 4, for [16], we used the code shared by the paper’s authors. However, there is a key difference between the figures we generated and the figures generated in [16]. In [16], for the best results, some portion of the training time must be spent on learning the distribution of the entire image (without patches). Then during generation, the entire image is used as an input to the network. However, the goal for our paper was to avoid needing to input the entire image into the network both at training time and generation. Therefore, when running [16] in Figure 4, we trained the network only on patches of images and we generated full size images by first generating patches of the images (with positional encoding information) and then simply stitching them together. \n\n**Comment: Did the authors try using [23] in an unsupervised manner**\n\nWhen using [23] in an unsupervised manner, it is necessary to first train an unsupervised diffusion model (the patch-based networks from our work suffices for this) and then apply the unsupervised network to solve the inverse problem. The original paper is able to use the conditional network to enforce data consistency with the measurement, but with an unsupervised network, it is necessary to add in an additional step in the inference loop that enforces data consistency. We ran experiments using DPS as the data consistency strategy that is consistent with PaDIS. Further note that [23] has an additional tunable parameter which is the amount of overlap between patches. \n\nTable 6 shows the results of this method when this parameter has been tuned under the name Patch Averaging. The table shows the method can obtain reasonable results but is outperformed by our proposed method. Additionally, the optimal overlap parameter value requires a significant amount of overlap between patches (approximately 1/4 of the patch dimension in both x and y) which increases the number of patches per iteration whose score function must be evaluated using the network. Finally, while the empirical results are reasonable, there is a lack of mathematical justification in [23] for the procedure of averaging the predicted noises of overlapping patches and future work is required to theoretically justify (from a probability distribution perspective) this method. In the revision, we will include more visual examples of this method compared with the others.\n\n**Comment: Inference time comparisons should be demonstrated with other methods**\n\nAlgorithm 1 indeed indicates that none of the sampling steps are skipped. For a fair comparison with other diffusion model based methods in Table 5, we used the same number of steps (1000) for all the methods; an increase in image quality was present with an increased number of steps for all the methods. DPS requires more time per image due to the computation of the gradient of the norm term which involves backpropagating through the network, and predictor-corrector sampling involves two network evaluations per iterations. The average reconstruction time per image in seconds for the methods in Tables 1 and 5 are shown below for 20 view CT. \n\nBaseline: 0.1\n\nADMM-TV: 0.7\n\nWhole image diffusion: 172\n\nPaDIS (VE-DPS): 195\n\nLangevin dynamics: 98\n\nPredictor-corrector: 189\n\nVE-DDNM: 105\n\n**Comment: Visual results should be presented as a high resolution image**\n\nThe global rebuttal PDF page has some examples of higher resolution images for different CT reconstruction experiments including 60 view reconstruction and fan beam CT. These images are displayed with higher contrast and should be more helpful in performing clinical diagnosis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to sincerely thank all the reviewers for the valuable comments and constructive feedback on our paper. We provide point-by-point responses to address each reviewer’s comments and highlight our response to some key questions and additional experiments and results as below: 

**More baselines for comparison**: We provided new results to compare with more baseline methods in Table 6 as shown in the attached pdf, including three diffusion-based methods [1,7,19], two plug and play (PnP) methods [42, 46], and two patch-based methods [23, 69] as suggested by Reviewers 2 and 3. This comprehensive comparison shows that our proposed method can outperform these relevant methods to a large extent by learning a better image prior and applying the optimal inverse solving algorithm.

**Different forward operators**: We conduct more experiments with different forward models: namely 60 view parallel beam CT, 180 view fan beam CT, and deblurring with a larger kernel of size 19x19. The results are shown in Table 7 and further demonstrate that our proposed method outperforms various SOTA methods for a large variety of forward models.

**Window size**: We displayed CT images corresponding to our new CT experiments in Figure B.1 using a narrower window size of 800 to 1200 HU. These images show better contrast between different organs and are more useful for obtaining a clinical diagnosis.

**Hallucinations and artifacts in the reconstructed images**: The presence of artifacts in some of the reconstructed CT images using generative methods is a natural consequence of using extreme compressed sensing with ultra-sparse views. Due to this lack of information, it is very hard for any model to perform a diagnostic-quality reconstruction, though our proposed method performs best in terms of the quantitative metrics. We added experiments with 60 view CT, where our proposed method is able to obtain excellent quality images as shown in Figure B.1: essentially no artifacts are visible.

**Innovation**: The main innovation of our method is a method to formulate a diffusion based image prior from solely the patches of the image. This work illustrates how image priors for very large images can be learned by learning priors of patches, a much less memory-intensive and data-hungry task. Unlike previous patch-based diffusion model papers such as [16] and [17] that can only be used for generating images, we show how our method learns a full image prior from only image patch training that can be coupled with most diffusion inverse problem solving algorithms, which is not a trivial extension based on [16] and [17]. 

[69]: Rumberger, Josef Lorenz, et al. "How shift equivariance impacts metric learning for instance segmentation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

Meta-review: Proposes a novel approach to scaling diffusion-based generative models to high-resolution images, by modeling randomly-tiled patches of images, which is motivated in part by reducing memory and training-data demands.  An acknowledged limitation is that the resulting generative model cannot capture very-distant correlations, but it is still effective for solving inverse-problems as focused on in experiments on both natural and medical images.  Reviews voiced several concerns about experimental details and baselines, which were mostly addressed by a thorough author rebuttal.  Please ensure that these additional results and clarifications are included in your final manuscript.
Decision: Accept (poster)