Autoregressive Image Diffusion: Generation of Image Sequence and Application in MRI
Authors: Guanxiong Luo
Shoujin Huang
Martin Uecker
Venue: NeurIPS 2024 poster (2024)
Keywords: Autoregressive models
Diffusion models
Inverse problems
Medical imaging
MRI
Primary area: machine_learning_for_healthcare

Abstract: Magnetic resonance imaging (MRI) is a widely used non-invasive imaging modality. However, a persistent challenge  lies in balancing image quality with imaging speed. This trade-off is primarily constrained by k-space measurements, which traverse specific trajectories in the spatial Fourier domain (k-space). These measurements are often undersampled to shorten acquisition times, resulting in image artifacts and compromised quality. Generative models learn image distributions and can be used to reconstruct high-quality images from undersampled k-space data. In this work, we present the autoregressive image diffusion (AID) model for image sequences and use it to sample the posterior for accelerated MRI reconstruction. The algorithm incorporates both undersampled k-space and pre-existing information. Models trained with fastMRI dataset are evaluated comprehensively. The results show that the AID model can robustly generate sequentially coherent image sequences. In MRI applications, the AID can outperform the standard diffusion model and reduce hallucinations, due to the learned inter-image dependencies. The project code is available at https://github.com/mrirecon/aid.

Autoregressive Image Diffusion: Generation of Image Sequence and Application in MRI
Guanxiong Luo
Shoujin Huang
University Medical Center Göttingen guanxiong. @ gee..uni-goettingen
Shenzhen Technology University
Martin Uecker
Graz University of Technology
ttkker@ttgraz.at
Abstract
Magnetic resonance imaging (MRI) a widely used non-invasive imaging modality. However, persistent challenge lies in balancing image quality with imaging speed. This trade-off is primarily constrained by k-space measurements, which traverse specific trajectories in the spatial Fourier domain (k-space). These measurements are often undersampled to shorten acquisition times, resulting in image artifacts and compromised quality. Generative models learn image distributions and can be used to reconstruct high-quality images from undersampled k-space data. In this work, we present the autoregressive image diffusion (AID) model for image sequences and use it to sample the posterior for accelerated MRI reconstruction. The algorithm incorporates both undersampled k-space and pre-existing information. Models trained with fastMRI dataset are evaluated comprehensively. The results show that the AID model can robustly generate sequentially coherent image sequences. MRI applications, the AID can outperform the standard diffusion model and reduce hallucinations, due to the learned inter-image dependencies. The project code is available https: / gighhb..c.. com/m cww/mroireeeeeeooooe wwwmmrco/aaaiddeooooooo aid.
Introduction
Magnetic resonance imaging (MRT) is non-invasive imaging modality widely used in clinical practice to visualize soft tissue. Despite its utility, a persistent challenge in MRI is the trade-off between image quality and imaging speed. The trade-off is influenced by the k-space (spatial Fourier domain) measurements, which traverse spatial frequency data points along given sampling trajectories. To reduce acquisition time, the k-space measurements are often undersampled, resulting in image artifacts and reduced image quality.
In recent years, deep soaming-bbsedd methods have emerged to improve image reconstruction in MRI. These methods formulated
inverse problem building upon compressed sensing techniques [1, 2] and benefit from the learned prior information instead of hand-crafted priors [3-5]. Another successful approach involves learning image prior parameterized by generative neural network [6, 7J, which is then used as the learned and decoupled regularization on the image. Generative priors offer flexibility in handling changes in the forward model and perform well in reconstructing high-quality images from undersampled data.
Diffusion models [8-10], class of generative models, have gained attention in recent years and are making an impact in many fields, including MRI reconstruction [11, 12). These models learn to reverse diffusion process that transforms random noise into structured images, producing high-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

quality, detailed images. Various approaches, including denoising diffusion probabilistic models (DDPMs) [10], denoising score matching [9], and continuous formulations based on stochastic differential equations (SDEs) [13], have been proposed for deriving diffusion models.
Recent studies demonstrate the effectiveness of diffusion models in accelerated MRI and their flexibility in handling various sampling patterns [11, 14-17]. For example, training score-based generative models using Langevin dynamics yields competitive reconstruction results for both in- distribution and out-of-distribution data [1]]. Additionally, score-based diffusion models trained solely on magnitude images can reconstruct complex-valued data [15]. Comprehensive approaches using data- fata-driven Markov chains facilitate efficient MRI reconstruction across variable sampling schemes and enable the generation of uncertainty maps [16].
Autoregressive models are statistical models that predict the current value of a variable based on its past values, capturing temporal dependencies and patterns within the data. They widely used in various fields such as time series analysis, signal processing, and sequence modeling. In natural language processing, autoregressive models like generative pre-trained transform (GPT) [18. 19] predict each token in a sequence based on previously generated tokens, enabling the generation of coherent and contextually relevant text. Similarly, image modeling, autoregressive models like PixeICNN [20] and ImageGPT [21] generate images by predicting each pixel value based on previously generated pixel values, often in a left-to-right, top-to-botton order. Instead of directly modeling pixels, which can be computationally expensive for high-resolution images, the study [22] proposes to first compress the image into smaller representation using vector quantized variational autoencoder (VQVAE). This VQVAE learns a codebook of visually meaningful image components. Then, transformer is applied effectively capturing the global structure of the image. By predicting each image component based on previous ones, the model generates high-resolution images in a sequential manner, maintaining consistency and coherence across the entire image.
model the autoregressive relationship between these components,
The clinical practice of MRI often involves acquiring volumetric image sequences monitor disease progression and treatment response; modeling and generating these image sequences is challenging. Autoregressive models can be employed model the joint distribution of image sequences and extract the dependencies between images. The diffusion process is effective in modelling images by treating each image independently. Therefore, we aim to combine these two models and propose autoregressive image diffusion (AID) model to generate sequences of images.
The contributions of this work the following aspects. We present how to derive the autoregressive image diffusion training loss starting from a common diffusion loss and how to optimize loss in parallel for efficient training. We present the algorithm to sample the posterior for accelerated MRI reconstruction when using AID to facilitate the incorporation of pre-existing information. We performed experiments evaluate its ability generating images when different the amount of initial information is given and to validate its effectiveness in MRI reconstruction. The results show that the AID model can stably generate highly coherent image sequences even without any pre-existing information. When used as generative prior in MRI reconstruction, the AID outperforms the standard diffusion model and reduces the hallucinations the reconstructed images, benefiting from the learned prior knowledge about the relationship between images and pre-existing information.
2 Methods
2.1 Autoregressive image diffusion
Given a dataset X consisting of multiple sequences images, each sequence represented as x {ø1, .2,..., .NN }. J. our goal is to model the joint distribution of these images. This joint distribution autoregressively factorized into the product of conditional probabilities:
N
p(x) q( (2o) II (c)),
(I)
t=2

---

Autoregressive image diffusion
X
X1
X2
X3
X
X5
Target sequence
Conditioning
sequence
X1 X2
X3 X4 X5 X6
X
X3 x4 Xg
Xt
N.C.A, w)
Encoder Block
TSC Block
Encoder Block
TSC Block
bodkeewwn.oo
Encoder Block
TSC Block
AXXi c,
Encoder Block
TSC Block
TSC Block
R.C.A, W)
Mid. Block
Decider Block
Attantion
mask
Decoder Block
Decoder Block
Decoder Block
Causal Attention
noise
The interaction between the images in conditioning sequence occurs in the DiTBlock, which has a causal attention module to ensure Iy is conditioned on previous images Ven- During training, the net predicts the noise for each noisy image that is sampled from the target sequence given the conditioning sequence parallel. During generation, the net iteratively refines the noisy input to produce clean image, which is then appended to the conditioning sequence.
Figure 1:
where Ten = (I1, T2, re...,,,,,,nu 3 and the image 2o is known. The model parameterized by is trained by minimizing the negative lig-likelihood of the data:
CAID = Ex -- (x)] = Ex log po (x1
Σ t=2
log po (xn|
(2)
Sohl-Dickstein al. (2015) and Ho et al. (2020) introduced the denoising diffusion probabilistic model (DDPM). This model gradually introduces fixed Gaussian noise to an observed data point x0 using known scales Be, generating series of progressively noisier values x² _x2 ,..., x²" The final noisy output x" follows a Gaussian distribution with zero and identity covariance matrix I, containing no information about the original data point. The series of positive noise scales B,,.... 3r must be increasing, ensuring that the first noisy output x' closely resembles the original data x while the final value x" represents pure noise. We apply this process to the conditional probability [(=n|=<< in Equation (2) by adding the noise to the image independent of the position in the sequence, i.c., 2 and znn are conditionally independent given x 1-1 Then the transition from xt-1 to s defined as:
q(<%|=/ x.2)) q( |- ( NN /1 B ,311)
Here, 2", represents the image In at time I, - is the image at the previous time step, and x2 denotes all images preceding In at the initial time step. The parameter 3r controls the drift and diffusion this process. The objective to learn to reverse this process. The reverse process is defined as:
(4) where ni and En are parameterized by neural network 0, taking 2// =2n- and l inputs. Using the variational lower bound, the reverse process can be learned by minimizing the negative logelikel of the data:
= N= : (z x2 22.. t), Ee(x " 20 t))
po(z | 20 ) log
log Po (2 1222 log (
= L
(5)
g 1 1-1
xzn)

---

Given the initial image o, and that æú, and z are conditionally independent given xrA. 2 at an arbitrary time step is sampled from a Gaussian distribution:
(( I x2n)) = M Vora ā.)
(6)
using 0g = - Br and ār = II Css. The posterior distribution x given xn and oa is then calculated as:
qx 1 l I =n))) = [o ;e where Pe ..)) = vor 1 1-a, on + var(1-au-1) on and 3r 1-az
=.) 3.1).
1-ãe-1 Be. I-a.
The training objective Equation (5) is further written minimizing the Kullback-Leible (KL) divergence between the forward and reverse processes in Equation (4) and Equation (7), as proposed by Sohl-Dickstein et al. (2015). (See Appendix A for details.)
In practice, the approach proposed by Ho et al. (2020) involves reparameterizing μ4 and predicting the noise for x The expression for x is given by x (=.. E) Vā,z, + vI- ā,c. with Ne (x x2 22 t) = 3e fixed. We realized this with a neural network cØ(x" t, x² shown in Figure I, which predicts the noise for oy at each time step given z²... the end, the objective function in Equation (2) for training autoregressive image diffusion is written as
N Σ 7= n=1
N
LAID 2
L.D
[VVeeV + VT are -
(8)
where the expectation taken over the noise NNO0, 7) and the time step & U(1, 7). To
generate an image sequence, we begin with the noise oं and update it iteratively using Equation (4) with the given =y. following the sequence (z x- .... + 2f). This process yields a clean sample oी. Subsequently, we can sample z22 in the same manner using the generated images x22. and continue this process iteratively to generate the entire sequence of images.
2.2 Architecture
To optimize the objective function in Equation (8) efficiently, ordered images are loaded as se- quences of certain length N + 1 during the training phase. We take the first N images Xcon (2o.21,..,, xy....,,-iii. - as the conditioning sequence and the last images Xtarget = {I1, (u.,,...... ANN } as the target sequence, shown in Figure 1. We adopt an architecture built an Unet [23] with capabilities temporal-spatial conditioning (TSC), designed to process the conditioning sequence and predict the noise for the target sequence. The term "temporal" refers conditioning previous frames along the N dimensions, while the "spatial" refers to the conditioning in the previous frame among the H W dimensions. Additionally, the TSC block is conditioned the time steps / of the diffusion process.
The only interaction between images in the conditioning sequence occurs during the attention operation. To maintain proper conditioning with autoregressive property, we implemented a standard upper triangular mask the
matrix of attention logits. This causal attention module is used in DiTBlock [18, 24]. The modified DiTBlock followed by a ResNet block [25], which a standard building block in the Unet architecture. The features output by the TSC block are then passed the corresponding encoder block in the Unet, which process the target sequence. The change in tensor dimensions inside TSC Block is handled by the einops library' and illustrated in Figure 1.
During training, the net predicts the noise in parallel for each noisy image that is sampled from the target sequence, given the conditioning sequence. During generation of sequence, the net iteratively refines the noisy input to produce a clean image, which is then appended to the conditioning sequence.
2.3 Application in MRI inverse problem
Image reconstruction is formulated as a Bayesian problem where the posterior of image p(zly) is expressed as
p(xly) = p(y(ar) p(x) p(v)
(9)

---

Here, represents the measured k-space data, denotes the image, and p(r) is a generative prior. The minimum mean square error (MMSE) estimator for the posterior minimizes the mean square emor, given by:
MMSE arg min / || x =112p(oly)dr B[a
(10)
2.4 Likelihood function for k-space
The image E C is represented as a complex matrix, where n x n the image size, and y Cmxmc is a vector of complex-value vvllued k-space samples from me receive coils. Assuming cireularl--ymmeericc normal noise with zero mean and covariance matrix I the likelihood p(ylæ) of observing y given is formulated complex normal distribution:
p(ylz) = C.V Ar, o = (T No ||, (x)Az)
(1I)
where I is the identity matrix, on is the standard deviation the noise, Ar represents the mean, and Np is the length of the k-space data vector. The operator A C"xa m maps the image to k-space and is composed of the coil sensitivity maps S, the two-dimensional Fourier transform F, and the k-space sampling mask P, defined as A = P.F.S. For more details and visual understanding on the forward operator, please refer to Appendix C.
2.5 Sampling the posterior
Given sequence of k-space y fyi,...,, UN each posterior in {po(xn Un 21 N} is expressed as
(a )n) plun Tni x2 (xnle2)))
p(yal on (nn)oo(znla p(vn)
P(yn
x p(
(12)
when the acquisition of yn is independent of the image x0 Un and x2,, are conditionally independent given Z, Following the Reference [8], we have
| po( - Un, =22)) or plyn 1at (xch (xt 1p x2 =2n)
(13)
The details for Equation (13) is in Appendix B. To sample the above posterior, the learned reverse process in Equation (4) is used, and the algorithm is constructed with two gradient updates using the log the prior and k-space likelihood: the DDIM (Denoising Diffusion Implicit Model) reverse step proposed by Song et al. (2020), and a data fidelity step derived from the likelihood function Equation (1)), which described follows:
VI a1co xD bn + - -1--a-1100(1 -I-
z s--1 + Va-i
(14)
Va:
2 + 5 + logg(un|=="
(15)
where is the step size, and V a, log (1/-1) is the gradient of the liglliieldood of Equa- tion (11). Then, the reconstruction of sequence images from the undersampled k-space data is achieved by sequentially sampling the posterior in {p(#,lyn Wn, =2n)]1 < n < N) using autoregressive diffusion model as prior. The algorithm is summarized in Algorithm
Experiments and Results
3.1 Model training
Two autoregressive diffusion models were trained on separate datasets: one in image space and the other in latent space. The image space model was trained on brain images that are from the fastMRI training dataset, which includes Te-weighted (some with post-contrast), T2-weighted, and FLAIR images [27]. These complex images were reconstructed from fully sampled multi-chann k-space
5

---

Algorithm Sample the posterior in {p(zn Ba, =2 < n N) using autoregressive diffusion model as prior.
1: Initial image sequence: -²²n = Zio; Time steps: T; Step size: X; Iterations for data fidelity step:
K; Number of samples: S;
2: for Yn in y {11: 1/22.-.... y do
3: Initialize z with Gaussian noise.
4: Construct the forward operator A with sampling pattern P and coil sensitivities S.
5: for & in {T' 1, I,....0) 0} do
Run the DDIM reverse step in Equation (14) to get = given zn and x2n- Run the data fidelity step in Equation (15) to update 4/11 for K step. 8: Add Gaussian noise scaled by /T 0t-1 to =!_
6:
7:
9:
end for
10: Update z²2, f- ( ........ =0]
11: end for
volumes, with coil sensitivity maps computed using the BART toolbox [28]. The images were then normalized to maximum magnitude of 1, and the real and imaginary parts were treated separate channels when input into the neural network. The number of images in each volume ranged from 12 to 16. Images were loaded without reordering and resized to x20x320 pixels if they were not already that size.
The latent space model is trained with the cardiac dataset that contains cine images reconstructed by the SSA-FARY method [29]. Firstly, a VQVAE was trained on the cine images that were preprocessed similarly to images in fastMRI dataset. The cine images have size of 2 256 pixels. Then, generates latent space for the training AID. (See the details for configuration of VQVAE in Appendix J). All the training was performed on 4 NVIDIA A100 GPUs with 80GB memory. The models were trained using the Adam optimizer with a learning rate of 10-1 and a batch size of 1 for image space model and 4 for latent space model. Two models were trained for 440,000 iterations. It took around 2 hours to train brain model for 10k steps and 1.2 hours for cardiac model. The length of conditioning sequence N for brain and cardiac models are 10 and 42. The network as illustrated in Figure I was implemented based on OpenAl's guided diffusion codebase2. We also trained a standard diffusion model, Guide, on the brain dataset for comparison. The Guide model was trained using the same hyperparameters as the AID model, except the batch size is 10. The Guide model uses the same Unet blocks as AID.
3.2 Generating sequence of images
To test different aspects of the autoregressive diffusion models, we generate the sequence of images using the following two approaches.
Retrospective sampling: This method generates a new sequence of images { ( [ based on the given sequence ..o....,NN-1] H. is sampled from Equation (4) given ( (0o.11.,
Prospective sampling: A fixed-length sliding window is initialized with the given sequence T>N (zo...., IN-1] ZN is generated from Equation (4) with the current window as conditioning. Subsequently, the window is updated by appending the newly generated v and removing the earliest image Zg. This autoregressive sampling process is repeated until the stop condition is met. We refer to this process as warm start. In a cold start, the window initialized with zeros, and each element x, in it is updated with newly generated images from the beginning to the end, after which the generation warmed up.
In the retrospective sampling, the model generates a sequence of images that are sequentially coherent and visually similar the conditioning sequence, as shown in Figure 2 (a). The prospective sampling generates a sequence of images that extends the initial images in the sliding window and constitutes multiple volumes, as shown in Figure 2 (b). As for a cold start, Figure demonstrates the model's ability to generate a sequence of images using black background as initial status. This shows the model's generative capabilities from a minimal initial condition, thereby proving its robustness and flexibility. Due to the limit of space, the samples with similar quality from the model trained on

---

akg
2a/ Big
2s/2s E2
/g/
#n/ãg
(a)
Generated
Ssa Jaras
iiva /irea
Jirzs
mia Jaeas
/
iis Jares
Sta /irgn
(b)
Figure 2: (a): A sequence of images from dataset is shown in the first row and is used as conditioning generate retrospective samples that shown in the second row. (b): With the given sequence in (a) as a warm start, prospective samples extending it are shown.
Jgn
2n11
Weg
94n
Figure 3: Prospective samples with cold start. The initial images generated in the cold start are not sequentially coherent, but as the sampling process continues, the model progressively generates more sequentially coherent and realistic images.
the cardiac dataset are shown in Appendix D. We also implemented boosted sampling technique which use previous slice with added noise as the initial image for the current slice. This requires less iterations generate the sequence of images. Further details can be found in our codebase.
3.3 MRI reconstruction
The MMSE estimator in Equation (10) cannot be computed in a closed form, and numerical approxi- mations are typically required. Once the samples from the posterior is obtained with Algorithm 1, consistent estimate of 2rMSSE can be computed by averaging those samples, i.e. the empirical mean of samples converges in probability to I MMME due weak law of large numbers. The variance of those samples provides solution to the error assessment in the reconstruction assuming the trained model trusted. To highlight the regions with large uncertainty, we compute the pseudo-confidence intervals based on the assumption that each pixel's 's intensity is normally distributed. This involves determining the standard error from the variance, then multiplying it by t-score corresponding to 95% confidence level.

---

Unfolding of aliased single-coil image: To investigate how the trained model, AID, reduces the folding artifacts in the reconstruction, we designed the single coil unfolding experiment. The single- channel k- k-space is simulated out of multichannel k-space data. The odd lines in k-space are retained, y. Ten samples were drawn from the posterior p( (y: 2o) using Algorithm with parameters: T = 1000, À 1, K = 5. The experiment was repeated using a standard diffusion model, Guide. The results are shown in Figure The AID model significantly reduces the errors in the region of folding artifacts compared to the Guide model. The mean over samples, TMMSE. is highlighted with a confidence interval computed from the variance of samples. The highlighted mean image shows the reconstruction by AID is more trustworthy in the folding region. In general, the highlighted region lies in the folding region, where large errors remains, as we expected.
(b)
Error
Sta
Mean
Reference
Zero Filled
A1D
Mask
Guide
Figure 4: (a): The folded single-coil image caused by two-times undersampling mask. (b): The comparison of unfolding ability by the autoregressive and the standard diffusion model, i.e., AID (top) and Guide (bottom). Reference image is reconstructed from k-space data without undersampling. The error is the difference between the mean, UMMSE, and the reference image. The "Mean+sid" is the mean highlighted with confidence interval, which indicates the reconstruction by AID is more trustworthy the region of folding artifacts.
Reconstruction from undersampled data: reconstruction, we conducted experiments on 20 volumes from the fastMRI validation dataset where k-space data was retrospectively undersampled using various sampling masks. We created four types of sampling masks: random with autocalibration signal (ACS), random without ACS, equispaced with ACS, and equispaced without ACS. The undersampling factor is 12. Setting parameters: T= 1000, 1, K = 4 for Algorithm 1, the images were reconstructed from the undersampled k-space data using the AID and Guide as prior, respectively. Another method proposed in Ref. [11] is used the baseline (CSGM), which uses scored-basee model (NCSNv2) from Ref. [30] trained on the fastMRI dataset. All the reconstruction tasks are performed by sampling the posterior. The likelihood p(y|z) is determined by forward model and the image prior is determined by the trained models, such as NCSNv2, Guide, and AID. This means that when the sampling method remains consistent, the performance of the reconstruction task is determined by the quality of the image prior. Our algorithm treats p(y(z) in the same manner, and the key difference is the image prior.
To further investigate the model's performance in
(PSNR in dB) and normalized root-mean square error (NRMSE) to evaluate the reconstruction quality against the reference image that is reconstructed from full k-space. The comparison of metrics across experimental conditions is illustrated in Figure 5. The proposed AID model outperforms the Guide and NCSNv2 in terms of PSNR and NRMSE especially in the absence of ACS, demonstrating its superior performance in image reconstruction from undersampled k-space data. The results are consistent across different undersampling factors and sampling masks, indicating the model's robustness and flexibility in handling various types of undersampled k-space data.
We used moan--ignal-noise-eeee
For the visual impression of the improvement by the AID model in reconstruction, we show the reconstructed images in Figure 6 and more of them Appendix E. The images reconstructed using AID are more visually similar to the reference images than using Guide, even which also provides aliased-free images. Furthermore, it is worth noting that more visually notable hallucinations were introduced by the Guide model than the AID model, which means AID is more trustworthy.
8

---

a Comparison Experimental
(b)
Comparison NAMSE Experimental
30 32
AD
30
0.200
Guide CRIM
0.150
26, 50
0.175
0.170
23.
23.5
0.150
15
0.075
10
0.056
0.007
0.023
ACS
8. ACS
E, ACS
mb ACS
ACS
m/o ACS
E
E. w/o AC5
Figure 5: E: equispaced, R: random. (a): PSNR and (b): NRMSE of the images reconstructed from the twelve-times undersampled k-space data using the autoregressive diffusion model (AID), the standard diffusion model (Guide), and the baseline method CSGM. PSNR higher is better, and NRMSE lower better.
Zero_filled (E)
Zero.filled (R)
Guide (E)
Guide (R)
AID (E)
AID (R)
Ref./R Mask
ACS-
ACS-D
Figure 6: E: equispaced, R: random. The last column shows the reference and the random sampling mask in k-space. The red lines are autocalibration signal (ACS) and equispaced mask is not shown. Zero-filles images are computed by inverse Fourier transform of the zero-fill eero-filled k-space data. The hallucinations are pointed with red arrows.
3.4 Other models for image sequence generation
To further evaluate the model's ability generate image sequences, we further trained two AID models two datasets: one using brain images from autism studies called ABIDE [31, 32], and the other using images from Unmanned Aerial Vehicle (UAV) view dataset [33]. We reported the computation and model complexity in Appendix F for all AID models trained in this work. We also implemented a two-stage training to improve the efficiency for training models on ABIDE and presented correspondingly generated samples in Appendix G. We demonstrated the natural image sequence generation in Appendix H and showed the sample consistency along the temporal axis in Appendix I.
4 Discussion
In this work, we propose an autoregressive image diffusion model for generating image sequences, with specific applications to accelerated MRI reconstruction. conducted comprehensive evaluation of its performance an image prior in reconstruction algorithms, comparing it to standard diffusion model. Due to the learned prior information on inter-image dependencies, the proposed model outperforms the standard diffusion model across various scenarios. Our model is particularly well-

---

suited for medical applications where image sequences are often acquired (e.g., in volumetric format) from patients in elinical practice. For instance, when different contrast images are acquired during an examination session [34], our model is designed to capture the relationships between these images. This enables more accurate and coherent reconstructions from undersampled k-space data using the proposed Algorithm 1, Additionally, other medical imaging tasks like dynamic MRT, multi-contrast, super-resolution, and denoising could benefit from our model's ability by leveraging inter-image dependencies [35]. Furthermore, the proposed algorithm holds great promise for facilitating the incorporation pre-existing information from other imaging modalities into MRI image reconstruction. This opens up wide range of potential medical applications, with the potential to improve patient care and reduce healtheare costs by enabling faster and more accurate image acquisition and diagnosis.
As this model has the capability to generate coherent image sequences, it is crucial to consider the privacy implications associated with its use, particularly in clinical settings. The generation of such images may inadvertently expose sensitive patient information, including identifi- able features such as facial characteristics. Safeguarding patient privacy must be a top priority when deploying it. We recommend that the model be used in controlled environment where access to the generated images is restricted to authorized personnel only. Additionally, is essential to ensure that the model is trained on anonymized data and that the generated images are not stored or shared without proper consent.
Privacy Issue:
Limitation and future work: We did not evaluate the model on common image dataset such as ImageNet or Cifar- 10, nor did we compute metrics such as FID and Inception Score, which could be limitation of our work. We plan to address these limitations in future work by running the model on a large dataset and comparing it with other sate-of-theear models. Additionally, given the model's suitability for modeling image sequences, it is worth exploring its potential for optimizing MRI k-space acquisition strategies, as the acquisition process constitutes a sequence of operations.
5 Conclusion
The proposed autoregressive image diffusion model offers an approach to generating image sequences, with significant potential as a trustworthy prior in accelerated MRI reconstruction. In various experiments, it outperforms the standard diffusion model in terms of both image quality and robustness by taking the advantage of the prior information on inter-image dependencies.
Acknowledgements
This work supported by DZHK (German Centre for Cardiovascular Research) funding code: 8100300155. We acknowledge funding by the "Niedersächsisches, Vorab" funding line of the Volkswagen Foundation. This work supported by the Federal Ministry of Education and Research (BMBF), Germany under the Al service center KISSKI (grant no. 01IS22093A).
References
[1] M. Lustig, D. Donoho, and M. Pauly. Sparse MRI: The application of compressed sensing for rapid MR imaging, Magn. Reson. Med., 58()) 2007.
[2] K. T. Block, M. Uecker, and J. Frahm. Undersampled radial MRI with multiple coils. Iterative image reconstruction using a total variation constraint. Magn. Reson. Med., 176),108 2007. ISSN 1522-2594. doi: 10. 10.10022mm..21255
[3] Yan Yang, Jian Sun, Huibin Li, and Zongben Xu. Deep ADMM-Net for Compressive Sensing MRI. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[4] Kerstin Hammernik, Teresa Klatzer, Erich Kobler, Michael Recht, Daniel K. Sodickson, Thomas Pock, and Florian Knoll. Learning variational network for reconstruction of ac- celerated MRI data. Magn. Reson. Med., 3966)3355--071, 2017. ISSN 1522-2594. doi: m0.1002/mm.26977.
10

---

[5] Morteza Mardani, Enhao Gong, Joseph Y Cheng, Shreyas S Vasanawala, Greg Zaharchuk, Lei Xing, and John M Pauly. Deep generative adversarial neural networks for compressive sensing mRI. IEEE transactions on medical imaging, 38(1):167-179, 2018.
[6] Kerem C TeTzaan, Christian F Baumgartner, Roger Luechinger, Klaas P Pruessmann, and Ender Konukoglu. MR image reconstruction using deep density priors. IEEE transactions on medical imaging, 38(7):1633 0011163316444 2019. doi: 10.1 101.9TMM..2182888777777
[7] Guanxiong Luo, Na Zhao, Wenhao Jiang, Edward S. Hui, and Peng Cao. MRI reconstruction using deep bayesian estimation. Magn. Reson. Med., 84(4): 2221122,, apr 2020. doi: 10.1002/ mrm.28274.
[8] Jaseha Dicksiicsttein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference machine learning, pages 2256-2265. PMLR, 2015.
[9] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché- Buc, Emily Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 11895-1 1907, 2019.
[10] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems,
2020.
[11] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 34: 14988-14494, 2021.
[12] Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong. Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and ding-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
[13] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representutions, 2021.
[14] Alper Güngör, Salman UH Dar, Şaban Öztürk, Yilmaz Korkmaz, Hasan Bedel, Gokberk Elmas, Muzaffer Ozbey, and Tolga Çukur. Adaptive diffusion priors for accelerated mri reconstruction. Medical Image Analysis, 88:102872, 2023.
[15] Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical image analysis, 80: 80249,, 2022.
[16] Guanxiong Luo, Moritz Blumenthal, Martin Heide, and Martin Uecker. Bayesian mri recon- struction with joint uncertainty estimation using diffusion models. Magnetic Resonance in Medicine, 90(11)225-33 -3-1111, 1, 2023.
[17] Martin Zach, Florian Knoll, and Thomas Pock. Stable deep mri reconstruction using generative priors. IEEE Transactions Medical Imaging, 2023.
[18] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[19] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[20] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espcholt, Oriol Vinyals, Alex Graves, al. Con- ditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
[21] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020.

---

[22] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[23] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAA 2015: 18th international conference, Munich, Germany, October 5-9, 2015. proceedings. part III 18. pages
Springer, 2015.
[24] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4200 2023.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
[26] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[27] Florian Knoll, Jure Zbontar, Anuroop Sriram, Matthew J Muckley, Mary Bruno, Aaron Defazio, Mare Parente, Krzysztof J Geras, Joe Katsnelson, Hersh Chandarana, et al. fastmri: A publicly available raw k-space and dicom dataset of knee images for accelerated mr image reconstruction using machine learning. Radiology: Artificial Intelligence, 2(1):e190007, 2020.
[28] Moritz Blumenthal, Martin Heide, Christian Holme, Martin Juschitz, Bernhard Rapp, Philip Schaten, Nick Scholand, Jon Tamir, Christian Tönnes, and Martin Uecker. mrirecon/bart: version 0.9.00, December 2023. URL attps:://oo org/ http:/...811///// 0081522200eee httt///eeodoo.02700000. 102779...999999
[29] Sebastian Rosenzweig, Nick Scholand, H Christian M Holme, and Martin Uecker. Cardiac and respiratory self-gating in radial mri using an adapted singular spectrum analysis (ssa-fary). IEEE transactions medical imaging, 39(10):0229-300
2020.
[30] Yang Song and Stefano Ermon. Improved techniques for training score-basee based generative models. Advances in neural information processing systems, 33:12438 13444488112448 2020.
[31] A. Di Martino, C-G Yan, Q. Li, E. Denio, F. X. Castellanos, K. Alaerts, J. S. Anderson, M. Assaf, S. Y. Bookheimer, M. Dapretto, B. Deen, S. Delmonte, I. Dinstein, B. Fitt-Waaner D. A. Fair,
L. Gallagher, D. P. Kennedy, C. L. Kcown, C. Keysers, J. Lainhart, C. Lord, B. Luna, V. Menon, N. J. Minshew, C. S. Monk, S. Mueller, R. A. Mueller, M. B. Nebel, J. T. Nigg,
K. O'Hearn, K. A. Pelphrey, S. Peltier, J. D. Rudie, S. Sunaert, Marc Thioux, M. Tyszka,
Q. Uddin, J. S. Verhoeven, N. Wenderoth, J. L. Wiggins, S. H. Mostofsky, and M. P. Milham. The autism brain imaging data exchange: towards large-scale evaluation of the intrinsic brain architecture in autism. Molecular Psychiatry, 19(6):659-66) 9669996666666 June 2014. ISSN 1359-4184. doi: 10. 10.1000m...01... 2013..77.
[32] Adriana Di Martino, David O' d'connor, Bosi Chen, Kaat Alaerts, Jeffrey S Anderson, Michal Assaf, Joshua H Balsters, Leslie Baxter, Anita Beggiato, Sylvie Bernaerts, Laura M E Blanken, Susan y Bookheimer, B. Blair Braden, Lisa Byrge, F. Xavier Castellanos, Mirella Dapretto, Richard Delorme, Damien A Fair, Inna Fishman, Jacqueline Fitzgerald, Louise Gallagher, R. Joanne Jao Keehn, Daniel P Kennedy, Janet E Lainhart, Beatriz Luna, Stewart H Mostofsky, Ralph-Axel Müller, Mary Beth Nebel, Joel T Nigg, Kirsten O' O'hearn, Marjorie Solomon, Roberto Toro. Chandan J Vaidya, Nicole Wenderoth, Tonya White, R. Cameron Craddock. Catherine Lord, Bennett I.. I Leventhal, and Michael Milham. Enhancing studies of the connectome in autism using the autism brain imaging data exchange II. Scientific Data, 4:1700 March 2017.
[33] ibrahim Delibaşoğlu. Pesmod:
Small moving object detection benchmark dataset for moving cameras. In 2022 7th International Conference on Frontiers of Signal Processing (ICFSP), pages 23-29. IEEE, 2022.
[34] Brett Levac, Ajil Jalal, Kannan Ramchandran, and Jonathan I Tamir. Conditional score-based reconstructions for multi-contrast mri. arXiv preprint arXiv:2303. 2023.
[35] Guangyuan Li, Chen Rao, Juncheng Mo, Zhanjie Zhang, Wei Xing, and Lei Zhao. Rethinking diffusion model for multi-contrast mri super-resolution. arXiv preprint 2xXX::2404.047855 2024.
12

---

A Loss function derivation
Below is derivation of Equation (5), the reduced variance variational bound for diffusion models. This adapted from Sohl-Dickstein et al. (2015) and Ho et al. (2020). We include it here only for completeness. In the forward process, =² and z are conditionally independent given x
log pa(a 1 x22u) g(0t lxnn x22n)
pe pa 1 = g( L x0
L = E
logp(æ, (=2n))
log
(16)
po (a( | x xb
( [m
po |= x2n))
B
logg(<z (=))n)
log
log
gl 1-1 | 1 x=)))
(
q s x-1 x0
p T
Pa (oth x2 <n
po po 1og x
<n
En
log
log
log
(18)
( p x xnn
qivn J/h vn x v²2n)
[-1
g( 2i xm
Egg
DKI( 1o xn)
p x xn))
Σ
DRL. (1(10 1 xn) po (2/1 1o xm)m)
(19)
log x22))
(20)
B Posterior derivation
When samples drawn from the posterior started from the standard Gaussian noise, with Equation (12)
we have
p(a" Uny x2n) p(unlllle (1
(21)
for all the reverse time steps. Because
p(a = / p( x
(22)
and
I p(ah lottt 3n x 22 pn)p(xt (xxx p(a l MBa x22n)=
(23)
plynl ) [22
(24)
p(yn)
then we have
/ p(atlktt Mr. x2 )pxz 2 ydo a1 p(ynlot).
/ placilatt I bpltttl ddz
(25)
p(yn)
Therefore, we have
a Mns x2n))= plun lxt )((4 wott x0
(26)
p(yn)
p(y.) is a constant for evidence. Then with gradient based method, the posterior p(an lottt Mns x2 is sampled from the likelihood p(ynl=?) and the reverse process p(a,, |1,411 .²2)),
C Likelihood function for k-space
The autocalibration signal (ACS) region are lines through the center of k-space, however, are fully sampled. The sensitivity of a coil is a spatial profile that describes the receiving field that induces signals the coil. The simultaneous data acquisition, with each coil's sensitivity corresponding to different subregion, leads to a complete image without aliasing artifacts.
13

---

k-space, E y
Aky
A
image, X
A
Akk
A
Frequency encoding
Trans.
a
k a
4 mane
Phase encodiag
FOV,
Figure 7: The relationship between k-space and image. The Nyquist theorem states that the sampling rate must be at least twice the highest frequency component the signal.
kspace, y
fullsampling pattern, grid
coil-img, s O x sensitivity, s
ground truth,
Fourler Trans.
Coll Comb.
Figure The signal detected by coil is weighted by its local coil profile, which called sensitivities and imposes weights the signal intensity. Consequently, it causes dark and bright regions in coil images. The ground truth image is the combination all coil images.
14

---

D
Cardiac samples
220
2:30
Figure 9: Prospective samples from the model trained on cardiac dataset.
#1 ~ 1110
31 ~
2111 ~e 1200
31 ~ P20
1330
Fag ~ Son
33 ~ #400
F31 ~o 30
Figure 10: Retrospective samples from the model trained on cardiac dataset.
15

---

E Reconstruction from undersampled data
Reconstructions for 12x-equispaced undersampled k-space without ACS lines
Reference
AID
Guide
Zero. filled
Reconstructions for 12x-equispaced undersampled k-space with ACS lines
Reference
AID
Guide
Zero. filled
Reconstructions for 12x-randomly undersampled k-space without ACS lines
Reference
AID
Guide
Zero filled
16

---

Reconstructions for 12x-randomly undersampled k-space with ACS lines
Reference
AID
Guide
Zero. filled
Computation and model complexity
Table presents the computation needed train the AID models four different datasets with different model complexities.
Dataset: the name of the dataset (fastMRI, cardiac, ABIDE, UAV).
Length: the length of the image sequence.
Image size: the dimensions of the images in the dataset.
Latent: the latent space representation used for the model, with options like VQVAE, Autoencoder- KL. or None.
Two-stage: boolean indicating whether two-stage training process used. Two-stage training is explained the following section.
Parameters: the number of parameters the model.
Train steps/s: training speed in steps per second.
Inference (it/s): inference speed in iterations per second.
Table 1: Datasets and computational resources used to train the four different AID models.
Dataset
Length
Image size
Latent
Two-stage
Parameters
Train steps/s
Inference (it/s)
fastMRI
320
None
False
-139M
-1.31
-10.07
cardiac
42
256
VQVAF, 4x
False -26M
-1.27
-20.20
ABIDE
46
128
None
True -36M
-0.89
-4.10
UAV
256
Autoencoder--F KL, 8x
True
-83M
-1.05
-39.60
G Two-stage training
We implemented a two-stage training process to improve training efficiency. In the first stage, we trained the U-net model. the second stage, we trained the temporal-spatial conditioning block with the pre-trained U-net model frozen. By doing so, we are able to train an AID model on ABIDE dataset, where the image sequence has a dimension of 46x128xx22 after preprocessing. The generated image squeence is shown in Figure 11.
H Natural image sequence generation
We trained an AID model on an UAV dataset in the latent space and generated images using the trained model. The generated images are displayed in Figure 12. The generated images demonstrate the effectiveness of the proposed method in learning sequentially coherent natural images generation. Each frame in Figure 2 shows acrial view of rural landscape with roads and/or water pond.
17

---

Figure 11: The generated ABIDE image sequence captures the changes in the brain structure.
Figure 12: The light changes in the water surface captured in the generated UAV image sequence.
I Sample consistency along the temporal axis
Figure 13 shows the sample consistency along the temporal (or axis. Columns and 2: Show sagittal and coronal views brain image sequence. These images appear to be medical scans with clearly stretched anatomical structures. Column 3: Displays the plane of cardiac image sequence. This displays the heart's s activity over time and shows the diastolic and systolic phases from left to right. Columns and 5: Show the x-t plane of UAV image sequence, both generated and real. These images show the change in aerial views of a landscape over time. The generated x-t plane are generally consistent with the real x-t plane images but suffer from the striped artifacts.
sagittal, y-z coronal, X-Z cardiac, x-t
UAV gen., x-t UAV real, x-t
Figure Temporal consistency of the images generated by AID models trained on different datasets. The first two columns show the sagittal and coronal view of brain image sequence. The x-t plane of cardiac image and UAV sequence are shown the last three columns.
18

---

VQVAE configuration for cardiac dataset
The VQVAE is trained the cardiac dataset generate the latent space for the training of the autoregressive diffusion model, using the official implementation' The VQVAE is trained with the following configuration:
base_lea mase_learning_eat 4.5
params:
embed_dim: dim: 3
n_embed: 8192
ddconfig:
double_z: false
z_chnnels: 3
resolution: 256
channells 3
out_ch:
ch: 128
ch_mult: [1, 2,
mom_res_blocke 2
attn_resolutions:
dropout: 0.0
lossconfig:
target: losses. basessesw.gggroaaaaaaa//SStththa WWW.ePPPSWiiDiiirriiiitttta.aa..
params:
disc_conditional: false
3
disc_start: 30001
disc_ disc_weight: 0.8
bodebook_wiight. 1.0
K
3D volume generation
To further improve the model's ability to generate 3D volumes, the position embedding is added to the third dimension z of the volume. This allows the model trained on the ABIDE dataset to have better consistency along the z-axis.
Figure 14: The generated 3D volumes from the model trained on the ABIDE dataset. The diffusion process is applied the transverse plane (x-y) (c.f. the image the left) and the autoregressive process is applied on the z-axis.
hhunss/ggtttbbcoommmmmmmittttinn hrw.:///ceeccottotooionnannannne transformers..pp.....
19

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See the last paragraph of the introduction in Page 2.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See the last paragraph of discussion in Page 10.
20

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, authors should discuss possible limitations of their approach to address problems of privacy and faireess
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
See Appendix and B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides all the information needed to reproduce the main experi- mental results, include data, training details, algorithm details, and evaluation metrics in the section of experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is available http:://ggittub..c///rrrrroooo//
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ http://oooSomiisiiooooo...o
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. //ipps.ccppuliiii http::cccppppppccccceeeeooeppece httpc//gggiiee//eeuuuuiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
22

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides all the necessary details to understand the results in the section of experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper reports error bar in Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 3.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
23

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ httpp::///eurip.... wwwpuurpssccppppppeeee Spppblic/Bthcsssuddeee
Answer: [Yes]
Justification: See Privacy Issue in the section of discussion.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Privacy Issue in the section of discussion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not release data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators original owners of assets are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does existing
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com candaatsees has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not submit new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: This work does not involve crowdsourcing nor research with human subjects.
25

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: This paper paper use publicly released fastMRI dataset and have been approved by the local IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "2VCQzv5SBh", "reviewer": "Reviewer_NC3G", "summary": "This paper introduces an autoregressive image diffusion (AID) model for generating image sequences and accelerating MRI reconstruction. The model combines autoregressive and diffusion approaches to leverage inter-image dependencies, aiming to improve reconstruction from undersampled k-space data in MRI. It was trained on the fastMRI dataset using 4 NVIDIA A100 GPUs for 440,000 iterations with the Adam optimizer. The model architecture incorporates several key components, including DiTBlock, DDIM, and VQVAE.\n\nExperiments demonstrate that AID outperforms standard diffusion models in terms of PSNR and NRMSE metrics, particularly for twelve-times undersampled data. The model shows a reduction in hallucinations in reconstructed images compared to standard models. The paper provides a detailed explanation of the model's theoretical foundations, describing the autoregressive factorization of the joint distribution of image sequences and how the diffusion process is applied to each conditional probability in the factorization.\n\nThe authors derive the training loss for the AID model from a common diffusion loss and present an algorithm for sampling the posterior for accelerated MRI reconstruction using AID. The model is evaluated on its ability to generate images with varying amounts of initial information, including both retrospective and prospective sampling approaches. The paper demonstrates the model's effectiveness in unfolding aliased single-coil images and shows improved reconstruction quality across various sampling masks and undersampling factors. The authors discuss the potential applications of the model in other medical imaging tasks and acknowledge limitations, proposing future work to address them.", "questions": "1. What is the computational cost of the proposed method compared to standard diffusion models, both in training and inference?\n2. How sensitive is the model to the choice of hyperparameters, particularly the sequence length?\n3. Can the model be extended to handle multi-modal or multi-contrast imaging data?\n4. What are the privacy implications of using this model in clinical settings? How does the model's performance scale with increasing dataset size or diversity? Can the model be adapted for other medical imaging modalities beyond MRI?\n5. What is the potential for integrating this model into existing clinical workflows or systems? How does the model handle cases where there are significant anatomical variations between sequential images?\n6. Can the model be extended to generate 3D volumetric data or time-series data? How do different k-space sampling patterns impact the model's performance? How does the model perform when there are motion artifacts or other types of image degradation?\n7. Can the model be used for other tasks, such as image segmentation or anomaly detection in medical imaging?", "limitations": "The authors acknowledge the limitation of not evaluating the model on common image datasets like ImageNet or CIFAR-10 and note the lack of standard generative model metrics like FID and Inception Score in their evaluation. The paper does not explicitly discuss potential negative societal impacts of the technology. The evaluation is primarily focused on MRI reconstruction, limiting insights into the model's generalizability, and the authors do not provide a comprehensive comparison with other state-of-the-art methods in image generation.\n\nThe paper lacks a detailed analysis of the model's computational efficiency and resource requirements. There is limited exploration of the model's performance on diverse pathological cases or rare conditions, and the authors do not discuss the potential limitations of the autoregressive approach in certain scenarios. The paper does not address the interpretability of the model's decision-making process or its robustness to adversarial attacks or input perturbations.\n\nThe authors do not explore the privacy implications of using the model in clinical settings or analyze the model's performance in low-resource or edge computing environments. There is no discussion on the potential for transfer learning or domain adaptation of the trained model, and the authors do not address the scalability of the model to larger or more diverse datasets. The paper does not explore the model's ability to handle multi-modal or multi-contrast imaging data or discuss the integration of the model with existing clinical workflows or systems.\n\nThe authors do not provide an analysis of the model's failure cases or edge scenarios, and the paper lacks exploration of the model's performance on other medical imaging modalities beyond MRI. There is no discussion on the potential ethical considerations of using AI-generated medical images, and the authors do not address the model's ability to handle out-of-distribution or anomalous cases in medical imaging.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel combination of autoregressive and diffusion models for image sequence generation, which is well-motivated, particularly for medical imaging applications like MRI reconstruction. The authors provide a comprehensive set of experiments demonstrating improved performance over standard diffusion models, effectively leveraging inter-image dependencies to enhance reconstruction quality. There is a clear demonstration of reduced hallucinations in reconstructed images using the AID model.\n\nThe paper offers a detailed theoretical foundation for the proposed model, deriving the training loss and sampling algorithm in a clear and reproducible manner. The experimental methodology is robust, using appropriate datasets and metrics for evaluation across various sampling patterns and undersampling factors. The authors include both qualitative and quantitative assessments of the model's performance, providing visual examples that effectively illustrate the improvements in image quality.\n\nThe model's ability to generate coherent image sequences is demonstrated through retrospective and prospective sampling. The paper discusses the potential broader impact of the model on medical imaging applications and acknowledges limitations, proposing future work and showing scientific integrity. The model architecture is clearly explained and illustrated with helpful diagrams, demonstrating flexibility in handling different types of undersampled k-space data. The authors provide a thorough comparison with a standard diffusion model baseline and discuss the model's potential for incorporating pre-existing information from other imaging modalities. The paper explores the model's performance in both image space and latent space and provides insights into the model's uncertainty estimation capabilities.", "weaknesses": "The evaluation is primarily limited to medical imaging datasets, lacking comparison on standard image datasets. While the theoretical justification for the model is provided, it could be expanded further. The paper does not thoroughly discuss potential negative societal impacts or ethical considerations of the technology, and computational requirements and efficiency compared to standard methods are not extensively discussed.\n\nThe paper lacks comparison with other state-of-the-art approaches beyond standard diffusion models and does not provide standard generative model metrics like FID or Inception Score. The model's sensitivity to hyperparameter choices, particularly sequence length, is not thoroughly explored, and there is limited discussion on the model's scalability to larger or more diverse datasets.\n\nThe paper does not explore the model's performance on other medical imaging modalities beyond MRI and lacks a detailed analysis of the model's failure cases or limitations. There is no discussion on the interpretability of the model's decisions or outputs and no exploration of the model's robustness to adversarial attacks or noise in the input data. The authors do not discuss the potential privacy implications of using the model in medical settings or provide a comparison of training times or computational resources required versus other methods.\n\nThe paper lacks discussion on the model's ability to handle multi-modal or multi-contrast imaging data. It does not explore the potential for transfer learning or fine-tuning the model on different datasets. There is limited discussion on the model's performance in low-resource or edge-computing scenarios and no exploration of the model's ability to handle out-of-distribution or rare pathological cases. The authors do not discuss the potential integration of their model with existing clinical workflows or systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Answers to Questions\n\nWe thank the reviewer for the detailed review and insightful questions. We address the reviewer's comments and questions below:\n\n1. **Computational Cost Comparison**: We have included a section in the general rebuttal that provides a detailed comparison of the computational cost of the proposed method compared to standard diffusion models in terms of training and inference. We have also included Table 1 in the supporting PDF file that presents the computation needed to train the AID models on four different datasets with different model complexities.\n\n2. **Sensitivity to Hyperparameters**: We choose the sequence length based on the trade-off between computational efficiency and dataset characteristics. For example, we use a sequence length of 10 in the fastMRI dataset, as a typical volume in the dataset contains around 16 slices. And we increased the sequence length to 46 for ABIDE dataset, as the typical volume in this 3D dataset contains over 100 slices.\n\n3. **Multi-Modal Imaging Data**: The fastMRI dataset contains volumes that are weighted with different contrasts, such as T1, T2, and PD. And in the generated sequence in Figure 2 and 3 of the manuscript, we show the model's ability to generate coherent sequences across different contrasts.\n\n4. **Privacy Implications and Scale**: We acknowledge the importance of privacy implications when using AI models in clinical settings, as it may reveal sensitive patient information like facial features. Usually, the model would perform better with larger and more diverse datasets, as it can learn more representative features. This model is applicable to other medical imaging modalities beyond MRI, as long as the data is in the form of image sequences.\n\n5. **Potential for Clinical Workflows and Out of Distribution**: Increase the throughput of MRI scanner because of AID allows higher acceleration factors, which can be beneficial in clinical settings. If the model has never seen a certain pathology, it may not be able to reconstruct it accurately. However, the model can be trained on a diverse dataset to handle out-of-distribution cases.\n\n6. **3D Volumetric Data and K-space Sampling Patterns**: The model can be extended to generate 3D volumetric data, as shown in the section *Two-stage training* of the general rebuttal. Figure 4 in the supporting PDF file displays the generated 3D volumetric data. We might be able to sample less and less k-space data as the sequence progresses, as the model has already seen the previous images and can predict the next one more accurately. When there are motion artifacts or other types of image degradation, the model might not perform well without taking them into account when constructing the likelihood model $p(y|x)$ as investigated in Reference [1].\n\n7. **Other Tasks**: It might be possible when segmenting a full volume.\n\n## Weakness and Limitations\n\n1. **Systematic Evaluation** We initially trained the model on natural image datasets such as aerial image sequences and obtained similar results to those on the medical image dataset. We have included these results in Figure 3 of the supporting PDF file. However, curating a large natural image sequence dataset to systematically evaluate the model's performance on a wide range of tasks would be time-consuming. That is why we chose to focus on medical image datasets and model's application in the manuscript. We acknowledge that worthwhile future work would be to evaluate the model on a broader range of tasks and datasets using standard generative model metrics (FID and Inception Score).\n\n2. **Comparison with State-of-the-Art Methods**: We have included a section, *Comparison with the existing method*, in the general rebuttal that discusses the comparison with the other method. We provide new statistics in Figure 1 of the supporting PDF file that show the comparison with the state-of-the-art method on the fastMRI dataset.\n\n3. **Computational Efficiency**: We have included a section, *Computation and model complexity*, in the general rebuttal and Table 1 of the supporting PDF file that discusses and presents the computational efficiency of the proposed method. Comparing to the standard diffusion models like Guide Diffusion, the AID model trained on fastMRI dataset achieve ~10 times evaluation per second, and Guide model achieves ~13 times evaluation.\n\n4. **Multi-modal/Fine-tuning**: We have included a section, *Two-stage training*, in the general rebuttal that discusses a more efficient training. That can be used to fine-tune the model on different datasets.\n\n5. **Societal Impacts**: We will include a new section in the manuscript that discusses the potential negative societal impacts and ethical considerations of the technology.\n\n6. **Interpretability and Robustness**: We will include a new section in the manuscript that discusses the interpretability of the model's decisions and outputs and its robustness to adversarial attacks or noise in the input data.\n\n\n## References\n[1] Levac, Brett, Ajil Jalal, and Jonathan I. Tamir. \"Accelerated motion correction for MRI using score-based generative models.\" 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI). IEEE, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r2TD8FTjjn", "reviewer": "Reviewer_oZEE", "summary": "The paper proposes an autoregressive diffusion model, where each image in an MRI sequence is generated by a diffusion model, but the noise predictions of the diffusion model are autoregressively conditioned on previous MRI images in the sequence. Essentially, previously introduced single-image diffusion based MRI reconstruction [16] is upgraded to use sequence information with an autoregressive formulation. Improvements over the single-image model are demonstrated both qualitatively and quantitatively in MRI images.", "questions": "I've addressed my questions related to evaluation in the \"Weaknesses\" section. If the evaluation concerns were addressed, I'd certainly have an even more positive opinion of the paper. I have one more:\n* One of the contributions is listed as the technique for efficiently optimizing the autoregressive loss in parallel. To me, this seems rather straightforward and directly based on autoregressive/diffusion model training processes. Would you be willing to share the particular challenges you have encountered in this process? I might be missing something here and I'd be glad to learn.", "limitations": "The authors address the limitations of their work in the paper sufficiently.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The autoregressive formulation makes use of image sequence information and clearly outperforms single-image diffusion. \n* The problem formulation incorporating sequential information into single-image diffusion model predictions with the use of VQVAE encodings appears to be novel and powerful. \n* Various domain-specific experiments prove the effectiveness of the model in the MRI setting.", "weaknesses": "Evaluation could be much more detailed:\n* While results are compared to the single image diffusion based approach \"Guide\", this model is internal. Direct comparisons to previous MRI reconstruction approaches would be much more convincing. Is there a particular reason why this could not be done?\n* The VQVAE and transformer-based autoregressive model appears fairly complex. What would be the result of applying **only** this model to predict the image sequence? i.e. having the autoregressive model directly predict $x_n$ given $[x_0, \\dots, x_{n-1}]$, with no diffusion model? It is clear that the sequence information makes the autoregressive-diffusion combination superior to single-image diffusion. But we cannot infer the effect of diffusion here: How important is the existence of the diffusion part of the model? How superior is the autoregressive-diffusion combination compared to pure autoregressive?\n* Pure generative performance on standard natural image datasets is not evaluated. While authors do state this limitation in their paper, it is still a shame to be missing these results. \n\nAlso, a few typos to be fixed:\n1. Line 57: \"As in the clinical practice of MRI, we often involves acquiring...\" This start is grammatically incorrect. Maybe: \"As the clinical practice of MRI often involves acquiring...\"?\n2. Figure 1: \"... noisy image that sampled from ...\" -> \"... noisy image that **is** sampled from ...\"\n3. Line 186: \"... OpenAI's guide diffusion codebase ...\" -> \"... OpenAI's guide**d** diffusion codebase ...\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and insightful questions. We address the reviewer's questions and concerns below:\n\n1. **Comparison to other MRI reconstruction approaches**: We have compared our method to CSGM method from Reference [1] in the general rebuttal, using an external model trained in Reference [1]. We further provided new statistics in Table 1 of the supporting PDF file, which shows that our method outperforms CSGM in terms of PSNR and NRMSE.\n\n2. **Design choice for the image sequence**: Using a transformer-based autoregressive model on latent codes from the encoder (e.g., VQ-VAE) is indeed a good choice for modeling temporal-alike 3D data. It has shown remarkable performance by OpenAI's Sora. A transform-based model alone can surely predict the long latent code for an image sequence without using the diffusion model. But when the latent code is long, the computation cost will increase significantly. What's more, that type of model may not be applicable to MRI reconstruction tasks, where the image sequence is acquired sequentially in the k-space. The application to MRI reconstruction tasks is the main motivation of our work. Compared to the single-image diffusion model, the autoregressive-diffusion model can better capture the inter-dependencies in the image sequence. This is beneficial for MRI reconstruction tasks, as demonstrated in our experiments.\n \n3. *But we cannot infer the effect of diffusion here: How important is the existence of the diffusion part of the model?*  \n   Using a purely autoregressive model to predict an image sequence can be highly computationally expensive. This is because a pure autoregressive model for 3D-like data involves processing long sequences, which significantly increases the computation cost. When the diffusion model is used as the primary component with a temporal-spatial conditioning (TSC) block, the length of the autoregressive mechanism within TSC would be much shorter, thereby decreasing the overall computational cost. Figure 1 in the manuscript shows how AID handles the image sequence.\n\n4. *How superior is the autoregressive-diffusion combination compared to pure autoregressive?*  \n\n   We haven't evaluated this in our work. The comparison between the autoregressive-diffusion combined model and the pure autoregressive model is hard to make, as they are designed for different tasks and could be specialized in different aspects. Autoregressive model like transformer is designed for token sequence generation and achieves remarkable results when there are enough data and computation resources. However, this paper tends to focus on an efficient way to handle the image sequence in MRI reconstruction tasks, where the image sequence is acquired sequentially in the k-space.\n\n3. **Evaluation of the autoregressive model**: We initially trained the model on natural image datasets such as aerial image sequences and obtained similar results to those on the medical image dataset. We have included these results in Figure 3 of the supporting PDF file. However, curating a large natural image sequence dataset to systematically evaluate the model's performance on a wide range of tasks would be time-consuming. That is why we chose to focus on medical image datasets and model's application in the manuscript. We acknowledge that worthwhile future work would be to evaluate the model on a broader range of tasks and datasets using standard generative model metrics (FID and Inception Score).\n\n4. **Typos**: We will correct the typos in the final version of the paper.\n\n5. **Efficient optimization of the autoregressive loss**: In the training stage, the efficient computation of the autoregressive loss within a sequence requires evaluating $\\epsilon_\\theta(x_2|x_1), \\epsilon_\\theta(x_3|x_2, x_1), \\cdots, \\epsilon_\\theta(x_n|x_{n-1}, \\cdots, x_1)$ in parallel. In the inference stage, the generation of the image sequence requires solely evaluating $\\epsilon_\\theta(x_n|x_{n-1}, \\cdots, x_1)$ for the last image.\n\n## Reference\n\n[1]  Jalal, A., Arvinte, M., Daras, et al, 2021. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 34, pp.14938-14954.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nJ1jcsU2b8", "reviewer": "Reviewer_YPrH", "summary": "This paper presents a diffusion model for MRI acceleration.  In which, an autoregressive image diffusion (AID) model is proposed to sequentially generate MRI image conditions on a given prior image sequences. This method is evaluated on the accelerated MRI reconstruction task using the public available dataset, fastMRI dataset.", "questions": "The method is conditioned on previous images, how about the computational cost, as the method will use all previous slices?\n\nThe efficiency of the proposed method should be horrible as it is a drawback of the autoregressive model, is this correct?\n\nAs the cost is high, can the AIM model be constructed in the latent space?", "limitations": "1. The method lacks novelty as diffusion models are commonly used in MRI acceleration tasks\n2. The method is more like for normal images but not for medical images like MRI; For medical studies, we care more about he anatomy plausibility while the image metrics, like MSE, may not be a good measurement for this purpose;\n3. 3D medical images are not videos; we care more about the structure consistency from any of the three views.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The figures are very helpful for the understanding of the paper.\n\nThe paper is well-organized and easy to follow.", "weaknesses": "1. Lack of quantitive comparison with other methods for MRI reconstruction.\n2. The paper should highlight the technical contribution, as most of the context of the methods is from the existing works.\n3. The paper claims the contribution on 3D and dynamic MRI, but expects more such results, not just for 2D images.\n4. The paper deals with 3D reconstruction but the results only show 2D slices. It is not clear the consistency from other views like Coronal and Sagittal views.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Author Response\n\nWe thank the reviewer for the insightful comments. We address the reviewer's questions and concerns below:\n\n### Weaknesses:\n\n1. **Comparison with other methods**: We have compared our method with the CSGM method from Reference [1] in the general rebuttal, using an external model trained in Reference [1]. We further provided new statistics in Table 1 of the supporting PDF file, which shows that our method outperforms CSGM in terms of PSNR and NRMSE.\n\n2. **Technical contribution**: The main technical contribution of our work is the autoregressive-diffusion model for the generation of high-dimensional data, such as volumetric MRI images of size 46x128x128, using affordable computation resources such as 4xA100 (see general rebuttal).\n\n3. **3D and dynamic MRI**: We have included the results for dynamic MRI image (cardiac) in the appendix of the initial submission. We further included 3D volumetric MRI image in Figure 4 of the supporting PDF file.\n\n4. **Sample consistency**: We have included other views like Coronal and Sagittal views in Figure 3 of the supporting PDF file (see general rebuttal). In the application to MRI reconstruction tasks, the consistency of image slices in different views is guaranteed by k-space.\n\n### Answers to Questions:\n\n1. *The method is conditioned on previous images, how about the computational cost, as the method will use all previous slices?*  \n   The computational cost of the proposed model (AID) is not high, as it is a diffusion model with an efficient temporal-spatial conditioning (TSC) block. Comparing to the standard diffusion models like Guide Diffusion, the AID model trained on fastMRI dataset achieve ~10 times evaluation per second, and Guide model achieves ~13 times evaluation.\n\n2. *The efficiency of the proposed method should be horrible as it is a drawback of the autoregressive model, is this correct?*  \n   If the length of the autoregressive mechanism within TSC is long, the computation cost of the model will increase significantly. However, we found that the length of AID is affordable for MRI reconstruction tasks, as the speed difference between AID and Guide is not significant.\n\n3. *As the cost is high, can the AID model be constructed in the latent space?*  \n    Yes. We constructed two AID models in the latent space by using the VQ-VAE or Autoencoder-KL encoder to encode the image sequence into a sequence of latent code. You can find all our models in the supporting PDF file from the general rebuttal.\n\n### Limitations:\n\n1. *The method lacks novelty as diffusion models are commonly used in MRI acceleration tasks.*  \n   Diffusion models are indeed commonly used in MRI acceleration tasks. However, the novelty of our work lies in the application of the autoregressive-diffusion model to MRI reconstruction tasks, which allows us to incorporate temporal information for better image reconstruction quality and has not been explored before. And our results show that the proposed AID model outperforms the single-image diffusion model in MRI reconstruction tasks.\n\n2. *The method is more like for normal images but not for medical images like MRI; For medical studies, we care more about the anatomy plausibility, while the image metrics, like MSE, may not be a good measurement for this purpose;*  \n   NRMSE and PSNR are commonly used metrics for evaluating the quality of MRI images when references are available. Further medical studies, like anatomy plausibility, would be worthwhile future work performed by radiologists.\n\n3. *3D medical images are not videos; we care more about the structure consistency from any of the three views.*  \n   We have included the results for different views of the generated image sequence in Figure 3 of the supporting PDF file. The consistency of image slices in different views is ensured by k-space in MRI reconstruction tasks. In practical MRI applications, 3D volumetric MRI images are acquired sequentially in k-space. It is important to note that AID is not designed for video generation tasks, which is a different research direction and requires a much higher level of temporal consistency in the generated image sequence.\n\n## Reference\n[1] Jalal, A., Arvinte, M., Daras, et al, 2021. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 34, pp.14938-14954.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

The authors would like to thank the reviewers for their valuable feedback and insightful comments. We have carefully considered all the comments and suggestions and have addressed them in this rebuttal. One PDF file is provided that contains a table and four figures that present the results of the experiments conducted in response to the reviewers' comments. They will be included in the revised manuscript. The following five sections provide detailed explanations for those results, respectively.

## 1. Comparison with existing methods

The method proposed in [Reference 1](#references) is used as the new baseline (CSGM), which uses a scored-based model (NCSNv2) from [Reference 2](#references) trained on the fastMRI dataset. In our initial submission, we trained a DDPM model and autoregressive diffusion model, named Guide and AID, which are built on the implementation from [Reference 3](#references), and applied them to various reconstruction tasks.  
All the reconstruction tasks are performed by sampling the posterior $p_\theta(x|y) \propto p(y|x)p_\theta(x)$. The likelihood $p(y|x)$ is determined by forward model and the prior $p_\theta(x)$ is determined by the trained models, such as NCSNv2, Guide, and AID. This means that when the sampling method remains consistent, the performance of the reconstruction task is determined by the image prior. Our algorithm treats $p(y|x)$ in the same manner as CSGM (see Eq. (11) and Eq. (15) in the manuscript), and the key difference is the image prior.  
The results indicate that AID outperforms both CSGM and Guide in terms of PSNR and NRMSE, particularly in the absence of an autocalibration signal (ACS). What's more, AID use the same U-net as Guide, with an added temporal-spatial block that significantly boosts performance. Figure 1 in the supporting PDF presents detailed results.

## 2. Computation and model complexity

Table 1 in the supporting PDF file presents the computation needed to train the AID models on four different datasets with different model complexities. All the training was performed on 4 NVIDIA A100 GPUs. When using latent space, the encoder is integrated into the dataloder.

- Dataset: (fastMRI, cardiac, ABIDE, UAV)
- Length: the length of  image sequence
- Image size
- Latent: latent space representation used for the model, with options like VQVAE, Autoencoder-KL, or None
- Two-stage: a boolean indicating whether a two-stage training process was used. [Two-stage training](#5-two-stage-training) is explained in the following section
- Parameters: number of trainable parameters
- Train steps/s: training speed in steps per second
- Inference (it/s): inference speed in iterations per second

## 3. Natural image sequence generation

We trained an AID model on an Unmanned Aerial Vehicle (UAV) view dataset [[4]](#references) in the latent space and generated images using it. The generated images are displayed in Figure 2 of the supporting PDF file, and each frame in it shows an aerial view of a rural landscape with roads and/or a water pond. This demonstrates the effectiveness of the proposed method in learning sequentially coherent natural image generation. 

## 4. Sample consistency along the temporal axis

Figure 3 in the supporting PDF file shows the sample consistency along the temporal (or z) axis.

Columns 1 and 2: Show sagittal and coronal views of a brain image sequence. These images appear to be medical scans with clearly stretched anatomical structures.

Column 3: Displays the x-t plane of a cardiac image sequence. This displays the heart's activity over time and shows the diastolic and systolic phases from left to right.

Columns 4 and 5: Show the x-t plane of a UAV image sequence, both generated and real. These images show the change in aerial views of a landscape over time. The generated x-t plane are generally consistent with the real x-t plane images but suffer from the striped artifacts.

## 5. Two-stage training

We implemented a two-stage training process to improve training efficiency. In the first stage, we trained the U-net model. In the second stage, we trained the temporal-spatial conditioning block with the pre-trained U-net model frozen. By doing so, we are able to train an AID model on ABIDE dataset [[5,6]](#references) where the image sequence has a dimension of 46x128x128 after preprocessing. The generated image sequence is shown in Figure 4 of the supporting PDF file.

## 6. Conclusion

In conclusion, we compared the proposed AID model with the CSGM method, reported the computation and model complexity, demonstrated the natural image sequence generation, and showed the sample consistency along the temporal axis. We also implemented a two-stage training process to improve the training efficiency. The more specific answers to the reviewers' comments are provided seperately.


## 7. References

[1] Jalal, A., Arvinte, M., Daras, et al, 2021. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 34, pp.14938-14954.  
[2] Song, Y. and Ermon, S., 2019. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32.  
[3] Dhariwal, P. and Nichol, A., 2021. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34, pp.8780-8794.  
[4] Delibaşoğlu, İ., 2022, September. PESMOD: Small moving object detection benchmark dataset for moving cameras. In 2022 7th International Conference on Frontiers of Signal Processing (ICFSP) (pp. 23-29). IEEE.  
[5] Di Martino A., Yan C-G, Li Q., et al. The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Molecular Psychiatry. 2014;19(6):659-667.  
[6] Di Martino Adriana, O’connor David, Chen Bosi, et al. Enhancing studies of the connectome in autism using the autism brain imaging data exchange II. Scientific Data. 2017;4:170010.

Meta-review: A reviewer recommends Accept, another reviewer recommends Borderline Accept and a third Reviewer recommends Borderline Reject. The reviewer who recommends Borderline Reject argues a lack of experimental comparisons against baselines and the limitation to 2D data. After reading the reviews and the rebuttal, the authors have provided an even stronger baseline and have further discussed the results in 3D. The paper makes a sufficient contribution and therefore I recommend Accept.
Decision: Accept (poster)