ResoNet: Noise-Trained Physics-Informed MRI Off-Resonance Correction
Authors: Alfredo De Goyeneche
Shreya Ramachandran
Ke Wang
Ekin Karasan
Joseph Yitan Cheng
Stella X. Yu
Michael Lustig
Venue: NeurIPS 2023 poster (2023)
Keywords: Inverse problem
MRI
Medical Imaging
Computational Imaging
Deep Learning
Off-Resonance
TL;DR: Our deep learning framework effectively corrects off-resonance blurring in MRI scans, leveraging a physics-informed model trained solely on synthetic data to improve image quality, perform fat/water separation, and enable faster scans.

Abstract: Magnetic Resonance Imaging (MRI) is a powerful medical imaging modality that offers diagnostic information without harmful ionizing radiation. Unlike optical imaging, MRI sequentially samples the spatial Fourier domain (k-space) of the image. 
Measurements are collected in multiple shots, or readouts, and in each shot, data along a smooth trajectory is sampled.
Conventional MRI data acquisition relies on sampling k-space row-by-row in short intervals, which is slow and inefficient. More efficient, non-Cartesian sampling trajectories (e.g., Spirals) use longer data readout intervals, but are more susceptible to magnetic field inhomogeneities, leading to off-resonance artifacts. Spiral trajectories cause off-resonance blurring in the image, and the mathematics of this blurring resembles that of optical blurring, where magnetic field variation corresponds to depth and readout duration to aperture size. Off-resonance blurring is a system issue with a physics-based, accurate forward model. We present a physics-informed deep learning framework for off-resonance correction in MRI, which is trained exclusively on synthetic, noise-like data with representative marginal statistics. Our approach allows for fat/water separation and is compatible with parallel imaging acceleration. Through end-to-end training using synthetic randomized data (i.e., noise-like images, coil sensitivities, field maps), we train the network to reverse off-resonance effects across diverse anatomies and contrasts without retraining. We demonstrate the effectiveness of our approach through results on phantom and in-vivo data. This work has the potential to facilitate the clinical adoption of non-Cartesian sampling trajectories, enabling efficient, rapid, and motion-robust MRI scans. Code is publicly available at: https://github.com/mikgroup/ResoNet.

ResoNet: Traseed Physics-Informed MRI Off-Resonance Correction
Alfredo De Goyeneche', Shreya Ramachandran' Ke Wang', Ekin Karasan',
Joseph Cheng", Stella X. Y Michael Lustig'
1 Electrical Engineering and Computer Sciences, University of California, Berkeley Radiology, Stanford University
Computer Science and Engineering, University of Michigan
fadg, shreyar, kewang, ekinkarasan, stellayu, @berkeey jychengdstanford. edu, stellayubumich. wt..mmmicc....
Abstract
Magnetic Resonance Imaging (MRT) is a powerful medical imaging modality that offers diagnostic information without harmful ionizing radiation. Unlike optical imaging, MRI sequentially samples the spatial Fourier domain (k-space) of the image. Measurements are collected multiple shots, or readouts, and each shot, data along smooth trajectory is sampled. Conventional MRI data acquisition relies on sampling k-space row-by-row in short intervals, which is slow and inefficient. More efficient, Carttraniiin sampling trajectories (e.g., Spirals) use longer data readout intervals, but are more susceptible to magnetic field inhomogeneities, leading to off-resonance artifacts. Spiral trajectories cause off- resonance blurring in the image, and the mathematics of this blurring resembles that of optical blurring, where magnetic field variation corresponds depth and readout duration to aperture size. Off-resonance blurring is a system issue with a physics- based, accurate forward model. We present physics-informed deep learning framework for off-resonance correction in MRI, which is trained exclusively on synthetic, noise-like data with representative marginal statistics. Our approach allows for fat/water separation and is compatible with parallel imaging accelcration. Through end-to-end training using synthetic randomized data (i.e., noise-like images, coil sensitivities, field maps), we train the network to reverse off-resonance effects across diverse anatomies and contrasts without retraining. We demonstrate the effectiveness of our approach through results on phantom and in-vivo data. This work has the potential to facilitate the clinical adoption of con-Cartesian sampling trajectories, enabling efficient, rapid, and motion-robust MRI scans. Code is publicly available at:
Introduction
Magnetic Resonance Imaging (MRI) is a powerful medical imaging modality that enables the acquisition of diagnostic information of soft tissue without harmful ionizing radiation. MRI visualizes hydrogen spins, mostly in water and lipids that are subjected to a very strong yet homogeneous magnetic field. When spins are excited, they emit signal at frequency proportional to the magnetic field present. Spatial encoding is possible by applying additional so-called gradient fields, which vary linearly in space and result in a Fourier relationship between the position of the spins and the frequency spectrum of the acquired signal. Since position is mapped to frequency, any additional magnetic field inhomogeneities will result in incorrect mapping and lead to image distortions and undesired artifacts. Magnetic field inhomogeneity can occur due to magnetic susceptibility differences between air and tissue, such as near the sinuses and ear canals in the head near the lungs and
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

bowel in the body. Also, due to different molecular environments, spins in water and fat see different magnetic fields. which results the so-called chemical shift phenomenon the signal frequency. The severity of these off- off-resonance effects depends on different imaging parameters, with general trade-off between sean time or signal-to-noise ratio and the level of artifacts.
While there are numerous intricacies to MRI acquisition, the process can be succinctly understood the collection of data in the spatial frequency domain, k-space, of the imaged object magnetization m(F). The relationship between the image and k-space data M M(E is defined by the Fourier relationship.
1.1 MRI Acquisition
(k) = F(m((F) = f vol
m(F)e--2a(ke) dV.
(1)
Here, is the position vector in the object space, and k are the coordinates in k-space. The MRI system employs a k-space traversal strategy where data is acquired sequentially over time along lines or smooth trajectories k(t). The time-dependent signal y(t) corresponds to samples of M ((K))), described by the equation [[]
yt, k(t)) = ((((t)) J. m(r) (277)-)2
(2)
Figure 1 illustrates different two-dimensional k(1) trajectories. The k-space measurements are collected in multiple shots, called readouts, with the scan time increasing linearly with the number of readouts. most clinical seans, k-space is sampled consists of many short readouts. However, these brief readout durations are relatively inefficient in covering k-space, resulting in prolonged scan times and susceptibility to motion artifacts. Researchers have developed more efficient non- Ca--Caressinn sampling trajectories, such as Spiral, which utilize longer but fewer readouts, allowing traversal of k-space in a significantly shorter time. However, these longer readouts make the trajectory
a row-by-row Cartesian manner, which
susceptible to off-resonance effects.
1.2 Off-Resonance in MRI Off-resonance arises mainly from two sources: 1) undesired spatial variations in the main magnetic field, which cause spins to resonate at frequency offset A A((r)
Cartesian Trajectory Short readout, slow scan Long readout, fast scan
EPI Trajectory
Spiral Trajectory
((k
Long readout, fast scan
F
Spatially varying distortion
Spatially varying blur
Figure I: Trade-offs between MRI acquisition strategies and off-resonance effects: The MRI
system directly samples the spatial Fourier domain (k-space) M (kk of the object m(F) in many shots, called readouts, each represented a by yellow arrowed path. The illustrated acquisition strategies assume fully sampled k-space, where off- off-resonance effects depend Cartesian trajectories collect k-space samples in many short readouts in a row-by-row manner, with minimal off-resonance artifacts. Echo-planar imaging (EPI) and Spiral trajectories collect k-space samples via longer but fewer readouts in a significantly shorter total time. However, these are more susceptible off-resonance effects, manifesting as image distortions in EPI and image blur Spiral.
readout duration and geometry.

---

proportional to the field inhomogeneities, and 2) the inherent chemical properties of tissue types in the object, such as fat and water, where spins in fat tissue resonate a fixed frequency offset fat from those in water. These offsets are relative to a carrier frequency So the resonant frequency of hydrogen protons in water - which we omit in our equations for simplicity, presenting instead the baseband signal. Equation 2 assumed a homogeneous magnetic field and single tissue type, specifically water; however, since images contain both water and fat components and both components experience the spatially varying off- off-resonance (F),,, Equation 2 be extended to
= (1rwater (7) + max((F) e
y(t, v(t,R(t))
av.
(3)
In this equation, the additional exponential terms introduce space-varying linear phase modulation to image samples, where the range of the phase modulation increases with the readout duration, t, thus breaking the Fourier model. Therefore, reconstruction using an inverse Fourier transform will result in image artifacts that depend on the trajectory used.
In Cartesian trajectories, all the readouts are in the same direction, accruing linear phase across k-space. This results in image domain shift according to the Fourier shift property and thus to geometric distortion for space-varying inhomogencities. Due to the short duration of Cartesian readouts, these off off-resonance artifacts are negligible compared to those from trajectories with longer readouts, like Spirals (Figure IT. Please refer to [2 for a detailed derivation of the exact shift. Since Spiral trajectories go outwards from the center of k-space, the phase accrual is center-out. This results in spatial frequency components being displaced in different directions, leading to blurring and loss of sharpness in the image. The Point Spread Function (PSF) of this effect corresponds to spatially varying blurring kernel proportional to the amount of local off-resonance. Please refer to the supplementary material for visual representations of these PSFs.
The mathematics underlying off-resonance blurring resembles that of optical blurring due to depth, as illustrated in Figure 2 The spatially varying magnetic field is mathematically similar to spatial variation in depth, while the trajectory readout length is analogous to the size of the optical aperture. Just like larger apertures increase both SNR and blurring from depth variations, longer readouts increase both SNR and blurring from variations in resonant frequency. However, unlike blurring in optical systems, off-resonance blurring is additive, layering out-of-focus regions on top of those in focus. One could bring off-resonant regions into focus by demodulating the signal equation with specific phase offset (Figure 2). For instance, extracting the fat phase term -22 from the integral in Equation 3 and demodulating by it makes the fat image components sharp, leaving water components blurred. Also, MRI introduces an additional complexity known as partial volume effects 3 where a single voxel can contain both resonant (water) and off-resonant (fat) spins.
1.3 Our Approach Off-resonance blurring is system problem and has known forward model, but solving for a direct solution is challenging due to its ioo-posed nature. Recently, Deep Learning (DL) has enabled significant improvements over classical methods for solving inverse problems (548) However, there are concerns over the black-box nature of end-to-end networks, overfitting, potential hallucinations (9) and the necessity of large representative training datasets. Physics-inspired DL methods, such as basel unrolled networks (6 10]. mitigate these concerns by enforcing consistency to the acquired data with the known forward model of the system. Utilizing a forward model also enables the simulation of a training dataset. cireumventing the need to collect large amounts of training data on an MRI scanner representative of all anatomies and contrasts.
In this work, we propose a physics- MRI. The key contributions of this work are follows:
unrolled-DL. framework for off-resonance correction in
End-to-end training using only synthetic data: A distinguishing feature of our approach is the exclusive utilization of synthetic data with representative marginal statistics for network training (inspired by [I].. By leveraging synthetic random field maps, partial volume effects, coil sensitiv- ities, and noise-like images, we climinate the requirement for real MRI data during the training process. This approach allows us to overcome the limitations of acquiring large training datasets on an MRI scanner, which can be costly, time-consuming, and unfeasible diverse scenarios.
Estimation of a resonance frequency spectrum per voxel using a physics-informed DL framework: This effectively handles partial volume effects, i.e., separates fat and water contributions within each voxel. While we demonstrate results for a Spiral trajectory, we propose a generic framework that could be applied to other non-Cartesian trajectories.
3

---

Reference &ir) map liza
Reference
bemodulate oote at
Demodulate at 120Hz
100
100
500
Simulated Chemical Shift (Hz)
Reference
Demodulate at Mar
Demodulate a
fat
4000
Figure 2: Illustration of off-resonance blurring effects in Spiral MRI and analogy with Depth of Focus. Top row: Water-based phantom data exhibiting off-resonance effects. The reference short- readout image required 4x the time. Middle row: simulated example with non-overlapping water and fat circles. Demodulating at the water resonance frequency (/)) brings the water circle into focus but causes blurring of fat, and vice versa. Bottom in optical imaging, where focal point is sharp, while other depths are blurred. In MRI, different demodulation frequencies correspond different focal lengths. Unlike optical blurring, off-resonance blurring is additive, with out-of-focus regions blurring on top of in-focus areas, making it challenging create a seamless all-in-focus image by stitching together multiple demodulated images.
(4) Analogy with depth of focus
Generalizability to multiple anatomies and contrasts: The proposed framework demonstrates strong generalizability, effectively correcting off refonacenance artifacts across different anatomies and contrasts without the need for retraining.
Recent developments in MRI highlight the potential of physics-driven DL methods in reconstruction tasks, spanning from physics-informed loss functions unrolled network architectures 66. [0] In off-resonance correction, traditional approaches include direct deblurring methods like autofocusing [2]. which iteratively determines the sharpest image resonance frequency per voxel; however, these often rely on simplified sharpness criteria from optics [13,, limiting their ability to generalize more complex off-resonance scenarios encountered in MRI, such as partial volume effects. Model-based analytical methods, such as conjugate phase reconstruction [4] require a magnetic field map measured from additional scans. The field map model does not represent partial volume effects, introduces additional scan time, and is susceptible motion artifacts; hence, this method often cannot completely correct for off off-resonance [15]
1.4 Related Work
More recently, data-driven DL-based approaches have been proposed to convolutional neural networks (CNNs) to directly deblur the image 16 or estimate the field map from image [19]. However, these approaches often neglect the physics of off-resonance blurring, do not handle partial volume effects, are limited in handling multi-coil accelcration, and rely on specific training data tailored to certain anatomies and contrasts.
Methods
Our proposed approach addresses the limitations previous off-resonance correction methods by developing forward model that handles partial volume effects and is compatible with multi-coil acceleration. By slicing the object into multiple frequency bins, we represent the tissue resonant at each frequency, leading to comprehensive model that captures the full spectrum of frequency offsets. Our approach recognizes that off-resonance a system problem intrinsic the MRI acquisition process rather than a direct image feature problem. As a result, leverage training synthetic

---

random data, providing an effective strategy to address this challenge and improve generalization across different anatomies and contrasts.
2.1 Physics-Informen Forward Model The key in our approach is that we do not model the spatially varying field and fat shift directly. Instead, we extend the dimensionality of our image to ar,, Af), which includes a frequency dimension A such that:
/ xfF d&f مم
We approximate this representation as a sum of multiple frequency bins = Aft). such that m(r) x= E; (F), with each bin resonating at uniform frequency Af; due field inhomogeneities and water/fat frequency offsets. Relating back to the optical imaging analogy, this is equivalent to representing 3D scene instead of a 2D scene with depth map. Using this modeling and the linearity of the signal equation
3 we can express the contribution to the signal from each frequency bin Af; as
vi ((t))(t)) = J. 2 -aneiinniftee -2akk))
(5)
where y(t, k(t))(()) = (i,, k(()) Since the phase modulation in bin applies uniformly to the image bin content, it can be taken outside of the integral, i.e.
(t k(t) JJ S.
(6)
Discretizing the Fourier Transform, we obtain
yi Maj, Exi,
(7)
where E an encoding matrix implementing a Non- Non-Uniform Fourier Transform (NUFFT), and MaJ, a diagonal matrix with elements e -2mm in the diagonal. Maj; phase modulates the k-space of the bin by the sample time and frequency offset.
Putting all together, we can construct complete forward model
y/ Σ y/ Ax,
(8)
where æ is the multi- multi-freqqency bin image, and A is the complete encoding matrix which combines the Fourier and modulation operators. Figure 3 iillsstaates our forward model.
is important to note that for the sake of brevity and simplicity, we ignore in our description the use of multiple receiver coils in MRI, which readers refer to in the following references [20] 21). However, by using general linear encoding operator E it is trivial to extend our approach from a single receiver in which E represents Fourier operator multiple receiver case where E. also includes multi-channel receiver sensitivities.
A
Σ
Maj E
Mata
Af-l
y
E
Mafi
Σ
Af
A
Mafen
Afei
k-space
Frequency images
Figure 3: Physics-informed forward model A: The image is modeled as a stack of sharp images at different frequency bins. Each bin image is encoded using encoding matrix E (e.g., Non-Uniform Fast Fourier Transform (NUFFT)), and then phase modulated by Maj. corresponding to its bin frequency. Finally, the k-spaces across the bins are summed (Σ) reconstruct the acquired k-space.
5

---

2.2 Synthetic Training Data Generation Inspired by [ all the training data consists of simulated synthetic random field maps, fat/water partial-volume effects, coil sensitivities, and noise- like images. The statistics and smoothness of these synthetic images are designed to model those of real data. Specifically, we generate random images and field maps AF(r) by applying an FFT to exponentially weighted random complex data, where the weighting radius determines the level of smoothness. Additionally, to create image examples I, we combine multiple random images to obtain edges with various sharpness levels. Similarly, random sensitivity maps S are generated from weighted random SPIRIT kernels [22]]
To model the fat-water partial volume effects, we generate random weight map W that determines the percentage of fat vs. water at each voxel (Figure 4.. From this weight map and image I, we obtain a fat image Lin and water image water. Since fat has a resonant frequency offset from water, in addition to the random magnetic field map frequency offset (F()), the fat image also includes constant offset ffat Subsequently, the fat and water images are assigned to different frequency bins based on the random field map A (r), fat offset A Afats and the corresponding bin frequency values. This results in multi-bin image representation fed into the forward model A to obtain the simulated k-space data y.
To address the ill-posed nature of the problem, we employ neural network for regularization. Inspired by the MoDL approach [6] we adopt an unrolled iterative model that is trained end-to-end. The input to the model is A stack of images demodulated at multiple frequencies obtained by applying the adjoint operator A on the raw k-space y. Regions of the object are sharp at the frequency bin corresponding to their local resonant frequency, while other off-resonant regions blur on top (Figure(5p) The objective of our model is to obtain clean image at each frequency bin, as well combined output image, water image, and fat image, all without off-resonance blurring.
2.3 Model Architecture
The unrolled model consists of Data Consistency (DC) modules and CNN-based proximal steps, as illustrated Figure 5b. The objective function for the DC module defined as follows:
arg min ||Ax v1 A||z DoL 1112
Here, x represents the reconstructed image at iteration k, A is the forward model, y is the acquired data, Das denotes the CNN with learnable parameters Φk. and À controls the trade-off between data consistency and regularization. The objective aims to minimize the discrepancy between the forward model's 's prediction and the acquired data, as well the difference between the reconstructed image and the output of the preceding CNN. From Equation
9 we can obtain:
(A A + AT)x A" + AD (x*-1).
(10)
which can be solved using the Conjugate Gradient (CG) method 23 to update x² at each iteration.
The CNN takes as input complex bin images with the real and imaginary components represented as separate channels. It includes residual blocks 24 and incorporates an attention module over
White
Decaving exponemiaa
Random Image
Waiter Image
Assign bins acund f
Feedmap
Simulated
K-ppace
Proomssirg
3/ww 1AU
1/-
4/u. 1432
Assign bins around Afr
Image frar
Figure 4: Synthetic training data generation. Random noise-like image J, fat/water percentage map W, off-resonance field map (F) are generated by applying Inverse Fourier Transform to exponentially weighted white noise images, where the weighting radius controls smoothness. From and W, fat and water images, Itst and water are obtained. Each voxel in these images is assigned a frequency bin based on AF(F) and the fat frequency offset A ftat. obtaining our multi-frequenccy image æ. Finally, x is processed via the forward model A to generate the k-space data y.

---

Supervise
Random
Simulated k-space
Afti
De.
DC
DC
H
M.
Ayer
luveer
y/
Awater
Supervise
(a) Training
A
Acquired
k-space
Afi-
D
DC Module
Dax
DC Module
H
Af
for Conjugate Gradient Descent:
y
(1" A 2Dx* = A" AD (x--1)
Afe
byater
Inference
Figure 5: Physics-informed deep learning framework for MRI off-resonance correction. (a) Training process: The model is trained using randomly generated data, including images (I, [water Itat) and frequency bins (x)) Simulated k-space data y obtained by applying the forward model A to x. The network, fed with images demodulated from A A" reconstructs frequency bins å and images f, Trats Twater: and is directly supervised by the original data. (b) Inference process: Acquired k-space y processed through A and fed to the model reconstruct bins and images. The unrolled network comprises N unrolls of CNNs and Data Consistency (DC) modules. Each DC module takes A, y, and previous CNN estimate Day (1) to solve for x using the Conjugate Gradient method.
frequency bins. This module allows the network to selectively focus on features from specific frequency ranges at each spatial location. Following the attention module, series of residual blocks are employed. Finally, convolutional layer produces the cleaner complex images at each frequency, which can then be fed into the DC module. Additionally, the network of the last unroll outputs final combined image i, fat image Lats and water image [water
2.4 Model Training and Inference During training, we utilize our synthetic data strategy to generate images 1, Lwaters fat, and the bin representation æ, from which we obtain simulated k-space y data using the forward model A. The unrolled model is fed with A y input, alongside A and for the DC modules. We task the model with reconstructing the bin representation å and images î, iwater, and Frat. with direct supervision provided by the synthetic data (Figure 5p).
The forward model A incorporates the acquisition characteristics of the readout trajectory in the NUFFT step, such the type of non-Cartesian k-space trajectory, readout duration, number of interleaves, resolution, and field of view (FOV). The model learns to correct for a specific PSF associated with the trajectory, and a single model becomes versatile and adaptable to various anatomies and MRI contrasts, such as proton density (PD), Ti- weigheet and Te-weithted images [
During inference, real acquired k-space data y, the forward model A, and A are provided to the network to reconstruct the bin representation and images (Figure 5b).
Experiments & Results
3.1 Phantom and In-vivo Acquisitions To evaluate the performance of our model, we acquired phantom data, as well as brain, knee, and abdominal in-vive data using a GE 3T MR750W System. MR Pulse Sequences were designed using SpinBench Software, and data was acquired using RTHawk

---

v2.5.2 Software (Vista.ai Inc., Palo Alto, CA.). The in-vivo data was acquired from three volunteers who provided informed consent, and the study was conducted with IRB approval.
For the phantom, knee, and brain scans, we used a Spiral trajectory with 36 interleaves, a 5.2 ms readout duration, resolution of 1 1mm², and 23 cm FOV. We acquired Proton Density (PD) weighted images for the knee and TI-weighted images for the brain. Alongside each scan, we acquired a reference scan with the same resolution and FOV but with a shorter readout (148 interleaves, 1.48 ms readout) to reduce artifacts. It is worth noting that for the knee scan, where we used a 2-second repetition time (TR), the acquisition time using the shorter readout trajectory is 5 minutes for single slice, compared to 1.2 minutes for the longer readout sequence, resulting in a 4x speedup.
TI ---weighted abdominal scans employed 38- 58-interleave Spiral trajectory with 10.4 ms readout duration, a Imm² resolution, and a 35 cm FOV, with data acquired during breath holds. A short readout reference sean was also acquired (320 repetitions, 1.52ms readout duration); however, as breathing was permitted between scans, slices will not exactly align. With a 50ms TR, the long readout trajectory takes less than 2 seconds to acquire versus 16 seconds for the short readout one.
More details on Pulse Sequences, coils, and other parameters used are in the supplementary material.
3.2 Model and Training Details In a 3T MRI, the fat frequency offset, Aftat, is reported to be approximately -400Hz [1]. our models, we simulated an off-resonance field map covering a range of ( = =80Hzz. To approximate the off-resonance spectrum, we utilized 21 frequency bins within the range of 600HHz to 200Hz, with a step size of 40Hz between bins. To better capture partial volume effects and avoid subtle data artifacts (26]. data generation was performed at a high resolution, three times better than the reconstructed spatial resolution and ten times the reconstructed frequency bin resolution. Subsequently, the generated data was decimated in the Fourier domain to match the target resolution and dimensions.
The network architecture featured four unrolls. Each CNN module in these unrolls included three residual blocks, each containing two convolutional layers with 128 filters and kernel size of employing ReLU activation. The DC module used 12 conjugate gradient iterations with trainable regularization parameter X. empirically initialized ensure that a ground truth input remained uncorrupted (A 1). We trained our model using PyTorch [2 in a Nvidia RTX GPU, employing & losses and the Adam optimizer (28. We used the torchkbnufft library 29 for NUFFT operations. We pre-generated training dataset to optimize memory usage and speed during training.
Two models were trained: one for the 23cm FOV trajectory, and another for the 35cm FOV scans.
In addition to our proposed approach, we aimed to replicate the methods in [6 and 17 to provide a relevant baseline for comparison, referred to as the "DL Baseline". We used the fastMRI brain dataset [30 with simulated Spiral trajectories and employed the network architecture and augmentation strategy proposed by [16] for training. It is important to note that this method only outputs combined corrected image, without separate fat and water images or a spectrum across frequencies.
3.3 Results on Phantom and In-vivo The phantom and in-vivo results were obtained using models for specific trajectories and trained solely on noise data, Sensitivity maps for the acquired k-space y were estimated using ESPIRIT (3I] with the BART toolbox (32). Though these maps may not always be perfectly accurate, they are fed to the model alongside the k-space, and our model remained robust to their potential inaccuracies.
Figure O showcases a phantom scan, a Tl Tl-weighted brain scan, a Proton Density (PD) weighted knee scan, and TI-weighted abdominal scan, illustrating the capability of our approach to generalize across multiple anatomies and contrasts. The figures depict 1) the uncorrected long readout input data demodulated at the water frequency fo, corresponding to the image a clinician would use for diagnosis, 2) the predicted water image output, 3) fat image output, and 4) combined clean output. Additionally, we compare our results against 5) the DL Baseline described in section 3.2 and 6) Autofocus [12] technique, which both aim to correct without the need for a separately acquired field map. Finally, we present 7) the uncorrected short readout reference image and 8) the reference field map obtained with separate seans. Please refer to the supplementary material for more detailed results showing all input and network output bins.
8

---

Water based Phantom
T1 Brain
PD Knee
T1 Abdominal
Uncorrected
Long Readout Image (fast)
[Ours] Output Water Image
wuter
[Ours] Output
Fat Image Jaaz
[Ours] Output Final Image
f
DL Baseline Output Image
Autofocus Corrected Image
Reference Uncorrected
Short Readout Image (slow)
Reference (f)) Field map (Hz) (extra scans)
Figure 6: Results of the proposed method phantom and in-vivo data. Results are presented for Water-based ACR phantom, as well as in-vivo T1-weighted brain, PD-weighted knee, and T weighted abdominal scans. Top-to-bottom rows: The uncorrected long readout input image. 2) output water image 1water 3) output fat image I 4) combined output image f. Reference baselines include 5) DL Baseline output image, 6) Autofocus [12] corrected image, 7) reference uncorrected short readout image (4x scan time for phantom, brain, and knee; 8x for abdominal scans). Lastly, 8)
reference field map to visualize resonance frequency variations.

---

3,4 Results on Simulated Off-Resonance Validation Set While obtaining reference scans with shorter data readouts possible, these seans will alter image contrast and introduce different artifacts. As a result, obtaining directly comparable ground truth data to compute meaningful quantitative metrics is infeasible. Therefore, as a performance proxy, we conducted an evaluation on a validation set created using the fastMRI brain dataset [30]. This set consists of brain anatomy images that were corrupted with simulated random off-resonance field variations and fat/water partial volume effects. The validation set comprised 1000 examples.
We compared the performance of our proposed approach and the DL Baseline by measuring the NRMSE (Normalized Root Mean Squared Error) and PSNR (Peak Signal to Noise Ratio) on the magnitude of the combined output images within the validation set. We report average metrics and their standard deviation over the 1000 examples in Table Despite our approach being trained solely on synthetic non-anatomy data, outperforms the DI. Baseline regarding NRMSE and PSNR.
Method \ Metric
NRMSE (L) 0.016 + 0.011 0.049 + 0.028
PSNR (t) 37.1 - 4.7
Our approach
Deep Learning (DL) Baseline
27.7 = 5.7
Table I: Performance comparison on simulated off-resonance validation set. NRMSE (lower is better) and PSNR (higher is better) for our method and the DL Baseline on a synthetic validation set of fastMRI [30] brain images with simulated random off-resonance and partial volume effects. Metrics show the average and standard deviation over 1000 examples.
4 Discussion
In this work, we propose a ihysice-inspired unrolled DL framework for off-resonance correction in MRI, showcasing its effectiveness through training solely on synthetic data. This approach addresses the challenges of collecting real MRI datasets for off-resonance correction, which can be unfeasible. Off-resonance blurring varies greatly across different anatomies due to differences in magnetic field variations, as seen in Figure 6 Our in-vivo evaluation showcases the generalization capabilities of our model across various contrasts and anatomies. Correcting off-resonance MRI particularly valuable for enabling non-Cartesian trajectories, such as Spirals, which significantly reduce scan time, leading to fewer motion-induced artifacts, shorter MRI exams, and improved clinical throughput.
The proposed framework and training strategy could bring further value to fields outside of MRI. The mathematics that arises from off off-resonance correction in MRI a generalization of lens aberration and depth from focus problems in microscopy and astronomy [33 34). The proposed technique could be adapted to these cases where there is a system error, like lens imperfection or atmospheric blurring.
4.1 Limitations and Future Work Despite promising results, our framework has limitations and areas for future improvement. Currently, networks are trained for specific trajectories, and future work involves handling multiple trajectories within a single model. This will include diverse trajectory patterns (such Rosette (35), Looping Star (36), resolutions, and FOVs. Another aspect for future improvement is simulating more extreme magnetic field map variations during training and increasing forward model resolution and network capacity, allowing for correction in settings such as regions close to air in the ear canals and nose. Currently, performance degrades when moving beyond the distribution of the training data simulation range.
Conclusion
In conclusion, have presented a physics-ifformed unrolled DL framework for off-resonance correction in MRI. Our approach incorporates data consistency using a multi-frequency forward model that represents resonance firqueeccy spectrum for each voxel. We have bypassed the challenges of collecting large training datasets by training our network solely on synthetic data.
We have demonstrated results in generalization and performance across diverse anatomies and contrasts. By addressing off-resonance issues, our framework has implications for accelerating scanning, increasing clinical throughput, and making MRI more accessible. Continued research and refinement in this area will further contribute to the advancement of MRI technology and its application in healthcare.
10

---

Acknowledgments
We acknowledge support from GE Healthcare and NIH grants RO1EB009690 and U01EB029427.
The authors would like to thank Suma Anand, Justine Taylor, and Ana Cismaru for their help with the paper editing.
References
[1] D. G. Nishimura, Principles of Magnetic Resonance Imaging. Lulu.com, 2010.
[2] M. W. Haskell, J.-F. Nielsen, and D. C. Noll. "Off-resonance artifact correction for mri:
A review." NMR in Biomedicine. vol. 36, no.
e4867. 2023. [Online]. Available: 10. 1002/00..44440
[3] M. Ángel González Ballester, A. Zisserman, and M. Brady, "Estimation of the partial volume effect in mri." Medical Image Analysis, vol. 6, no. 4, pp. 389-405, 2002. [Online]. Available:
[4] Wikipedia. (2023) Light field camera Wikipedia, the free encyclopedia. File:
Lytro_Illum Illum light field camera wwweraaaammeeeeeaaaaaaaaaaaaeeee demoostraation.ppps [Online]. Available: http://cnn.wikiiddia.oo
index. index.php'tille phtpppplpe--iiihhheeeie 20w.eed
11543798651
[5] F. Knoll, K. Hammernik, Zhang, Moeller, T. Pock, Sodickson, and Akçakaya, "Deep-learning methods for parallel magnetic resonance imaging reconstruction: A survey of the current approaches, trends, and issues," IEEE Signal Processing Magazine, vol. 37, pp. 128-140, 01 2020.
[6] H. K. Aggarwal, M. P. Mani, and Jacob, "Modl: Model-based deep learning architecture for inverse problems." " IEEE Transactions on Medical Imaging, vol. 38, no. 2, p. 394-405, Feb 2019. [Online]. Available: http://dx.doi.org/ httpb//doo..rg/.0..11///TM 10tp/10..111MMMM001 20T.....1..222322
[7] K. Wang. J. Tamir, A. De Goyeneche, U. Wollner, R. Brada, S. X. Yu, and M. Lustig. "High fidelity deep learning- based mri reconstruction with instance- wise discriminative feature matching loss," Magnetic Resonance in Medicine, vol. 88, no. 1, pp. 476-491, 2022.
[8] K. Hammernik, T. Klatzer, E. Kobler, M. P. Recht, K. Sodickson, T. Pock, and Knoll, "Learning a variational network for reconstruction of accelerated mri data," Magnetic resonance in medicine, vol. 79, no. 6, pp. 3055-3071, 2018.
[9] M. J. Muckley, B. Riemenschneider, A. Radmanesh, S. Kim, G. Jcong, Ko, Jun, H. Shin, D. Hwang, M. Mostapha, S. Arberct, Nickel, Z. Ramzi, P. Ciuciu, J.-L. Starck, J. Teuwen, D. Karkalousos, C, Zhang, A. Sriram, Z. Huang, N. Yakubova, Y. W. Lui, and F. Knoll, "Results of the 2020 fastmri challenge for machine learning image reconstruction," IEEE Transactions on Medical Imaging, vol. 40, 9, pp. 2306-2317, 2021.
[10] K. Hammernik, T. Küstner, B. Yaman, Z. Huang, D. Rueckert, F. Knoll, and M. Akçakaya, "Physics-driven deep learning for computational magnetic resonance imaging: Combining physics and machine learning improved medical imaging," IEEE Signal Processing Magazine, vol. 40, no. 1. pp. 98-114 14,14,220 2023.
[11] S. Hu and J. Y. Cheng, "System and method for noise-hased training of prediction model," Mar 2020.
[12] D. C. Noll, J. M. Pauly, C. H. Meyer, D. G. Nishimura, and A. Macovskj, "Deblurring for non-2d fourier transform magnetic resonance imaging," Magnetic Resonance in Medicine, vol. 25, no. 2, pp. 319-333, 1992. [Online]. Available:
[13] R. A. Muller and A. Buffington, "Real-time correction of atmospherically degraded telescope images through image sharpening." J. Opt. Soc. Am., vol. 64, no. 9. pp. 1200-1210, Sep 1974. [Online). Available: http://www.oapubbishinggg
64-9 1200
[14] D. Noll, J. Fessler, and B. Sutton, "Conjugate phase mri reconstruction with spatially variant sample density correction," IEEE Transactions on Medical Imaging, vol. 24, no. 3. pp. 325-336, 2005.
[15] A. K. Funai, J. A. Fessler, D. T. B. Yeo, T. Olafsson, and D. C. Noll, "Regularized field map estimation in mri," JEEE Transactions on Medical Imaging, vol. 27, no. 10, pp. 1484-1494, 2008
[16] D. Zeng, Shaikh, Holmes, R. L. Brunsing. J. M. Pauly, G. Nishimura, Cheng, "Deep residual network for off iiffrresoance artifact correction with application to pediatric body mra with 3d cones." Magnetic Resonance in Medicine, vol. 82, no. 4, pp. 1398 1411, 2019. [Online]. Available:
Vasanawala, and

---

[17] Y. Lim, Y. Bliesener, S. Narayanan, and K. S. Nayak, "Deblurring for spiral real-time mri using convolutional neural networks," Magnetic Rexonance in Medicine, vol. 84, 6, pp. 3438-3452, 2020. [Online]. Available:
[18] K. Wang, M. Doneva, J. Meineke, T. Amthor, E. Karasan, F. Tan, J. Tamir, S. X. Yu, and M. Lustig, "High- fidelity direct contrast synthesis from magnetic resonance fingerprinting." arXiv preprint ar 2r11v:2222.00811 2022.
[19] M. W. Haskell, A. A. Cao, D. C. Noll, and J. A. Fessler, "Deep learning field map estimation with model-based image reconstruction for off resonance correction of brain images using a spiral acquisition."
[20] A. Deshmane, Gulani, M. A. Griswold, and N. Seiberlich, "Parallel mr imaging." Journal of Magnetic Resonance Imaging, vol. 36, no. I, pp. 55-72, 2012.
[21] K. P. Pruessmann, M. Weiger, M. B. Scheidegger, and P. Boesiger, "Sense: Sensitivity encoding for fast mri," Magnetic Resonance Medicine, vol. 42, no. 5, pp. 952-962, Nov 1999.
[22] M. Lustig and J. M. Pauly, "Spirit: Iterative self-consistent parallel imaging reconstruction from arbitrary k-space," Magnetic Resonance in Medicine, vol. 64, no. 2, pp. 457-471 2010. [Online]. Available: https://onlinelibrary.yo www.ey.com/do//abee.e... http:///wnnnibbbaaiiilioooo/o//a http:////oaab//1...00////..mmmmm 224288
[23] R. Shewchuk et al., "An introduction to the conjugate gradient method without the agonizing pain." 1994.
[24] K. He X. Zhang, S. Ren, and J. Sun. "Deep residual learning for image recognition." 2015.
[25] T. A. Bley, O. Wichen, C. J. François, J. H. Brittain, and Reeder, "Fat and water magnetic resonance imaging," Journal of Magnetic Resonance Imaging, vol. 31, no. 1, pp. 4-18, 2010. [Online]. Available:
https///mninelirary..llcc.cccco wwtp://wwwwww..//////////....... 10000....0002mm..1......
[26] E. Shimron, J. Tamir, Wang, and M. Lustig. "Subile Inverse Crimes: Naively using Publicly Available Images Could Make Reconstruction Results Seem Misleadingly Better!" in Proceedings of the 29th Annual Meeting of ISMRM, 2021.
[27] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, I.. Antiga, A. Desmaison, A. Kopf, E. Yang. 7. Vitoto, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, I.. Fang, J. Bai, and S. Chintala, "Pytorch: An imperative style, high-performance deep learning library." in Advances in Neural Information Processing Systems 32. Н. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garneu, Eds. Curran Associates. Inc.. 2019, pp. 8024-8035. [Online]. Available: http://paperr..euurps.ccuuuppp learnin----nn------- library.pdf
9015-pytorch- an imperative imppraiive-style-iigeee
[28] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization." in 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, Y. Bengio and Y. LeCun, Eds., 2015. [Online]. Available: http://arxiv.oor///s//111.6600
[29] M. J. Muckley, R. Stern, T. Murrell, and F. Knoll. "TorchKbNufft: A high-level, hardware-agnoste non-uniform fast Fourier transform." in ISMRM Workshop on Data Sampling & Image Reconstruction. 2020, source code available at https://gtthh..com/// com//mmucccttyyorcceuuuu..
[30] J. Zbontar, F. Knoll, A. Sriram, T. Murrell, Z. Huang, M. J. Muckley, A. Defazio, R. Stern, P. Johnson, M. Bruno, M. Parente, K. J. Geras, J. Katsnelson, H. Chandarana, Z. Zhang, M. Drozdzal, A. Romero, M. Rabbat, P. Vincent, N. Yakubova, Pinkerton, D. Wang. E. Owens, C. L. Zitnick, M. P. Recht. D. K. Sodickson, and W. Lui, "fastMRI: An open dataset and benchmarks for accelerated MRI," 2018.
[31] M. Uecker, P. Lai, M. J. Murphy, P. Virtue, M. Elad, J. M. Pauly, S. S. Vasanawala, and M. Lustig, "Espirit-an eigenvalue approach to autocalibrating parallel mri: Where sense meets grappa," Magnetic Resonance in Medicine, vol. 71, 2014.
[32] M. Uecker, F. Ong. J. I. Tamir, D. Bahri, P. Virtue, J. Y. Cheng, T. Zhang, and M. Lustig, "Berkeley advanced reconstruction toolbox," in Proc. Intl. Soc. Mag. Reson. Med, vol. 23, no. 2486, 2015.
[33] W. Welford, Aberrations of Optical Systems, 1st ed. Routledge, 1986.
[34] D. Schroeder, Astronomical Optics, 2nd ed. Elsevier Science, 1999.
[35] D. C. Noll, "Multishot rosette trajectories for spectrally selective mr imaging." IEEE Transactions on Medical Imaging, vol. 16. pp. 372-377, 1997.
[36] F. Wiesinger, A. Menini, and A. B. Solana, "Looping star," Magnetic Resonance in Medicine, vol. 81. no. 1. pp. 57 68, 2019. [Online]. Available:
12


Reviews:
[{"review_id": "qfmPQTTUwk", "reviewer": "Reviewer_h1kS", "summary": "This work introduces a physics-informed unrolled-deep-learning framework for off-resonance correction in MRI. The proposed approach maintains data consistency by incorporating a forward model with coil sensitivities, multi-frequency bins, and non-uniform Fourier transforms. \n\nThe proposed model is first trained on simulated data, and then transferred to real MRI data. The experiments have shown the effectiveness of the pre-training stage on synthetic data, as well as the generality of the pre-trained model to real-world MRI data. ", "questions": "N/A", "limitations": "Yes, the authors have discussed about their method's limitations and potential future directions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The presentation is straightforward and easy to follow.\n\n2. The proposed approach is well-motivated and reasonable. The lack of large training dataset size is a well-known challenge for medical imaging field, and synthetic data for pre-training is helpful for model training, especially when the simulation is sufficiently realistic. The authors also show their generation process for the synthetic data and illustrate the effective of it in experiments. \n\n3. Comparison results on real-world MRI data from both brain and knee demonstrate the effectiveness of the proposed model.", "weaknesses": "In experiments, only Autofocus, a traditional non-deep-learning method proposed in 1992, has been compared against the proposed method, making it less convincing in terms of the proposed model's effectiveness, given deep-learning-based models are generally more resource-consuming. It is necessary to at least provide comparisons with other present deep-learning-based SOTAs in order to demonstrate the superiority of the proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Our short answer is that recent deep learning methods on off-resonance correction are not directly comparable.  See details below.\n\nWe chose Autofocus as a fair baseline comparison since it is an analytical method for direct off-resonance correction. Compared to other methods and recent deep-learning approaches for MRI off-resonance correction [2,3,4], our approach does not rely on actual MRI data for training. Acquiring sufficient training data can be expensive and time-consuming, and acquiring a reference image can even be unfeasible. Instead, we train our model only on simulated random noise-like data and leverage data consistency on our physics-informed framework. This approach allows us to simulate diverse sampling trajectories for each use case without requiring specific acquisition datasets for training. The only data we acquire consists of the test data in the real in-vivo scenario; however, our method does not train on this data.\n\nA traditional method is the Conjugate Phase method [1], which requires the acquisition of a fieldmap for correcting, which can be time-consuming, does not handle partial volume effects, and may lead to additional artifacts. [2] proposes an approach to directly estimate the fieldmap from the acquired image with a CNN, which is then used to correct the image with the conjugate phase method. Their approach requires acquiring a dataset of multiple scans to create the reference fieldmap used for training.\n\nOther recent deep-learning-based approaches, such as [3] and [4], rely on training datasets specific to particular acquisition trajectories, making them less applicable to our case, where we aim for a versatile model that can handle diverse trajectories and scenarios. Their method also aims to deblur the image directly and does not handle partial volume effects. In many cases, obtaining a reference image is unfeasible due to scan length constraints (e.g., breath hold limits, and involuntary motion such as bowel and cardiac, which limit scan resolution and quality).\n\nNevertheless, to directly address the reviewer’s concern, we have conducted additional experiments to compare our proposed approach. We aimed to replicate work similar to [3] and [4]; however, it is not possible to exactly replicate results, since their datasets are specific to a sampling trajectory and wouldn’t be able to actually compare to our test set without collecting a training dataset. Therefore, we instead used the fastMRI brain dataset [5] and simulated the spiral trajectory we used for our test set, and trained with their augmentation strategy and network architecture proposed by [3]. Please refer to Figure 1, attached to the rebuttal. The deep learning baseline performs decently in regions of water-only content but struggles in regions with fat content (e.g., knee scan (d) and brain outer region (c)).\n\nLastly, our proposed model extends beyond image deblurring; it also enables the reconstruction of fat and water images and provides a low-frequency-resolution B0 spectrum at each voxel, effectively handling partial volume effects and offering comprehensive insights into tissue properties. To the best of our knowledge, no comparable deep learning method achieves these same outcomes in one framework. \n\nWe hope that these clarifications demonstrate the unique contributions of our physics-informed unrolled deep-learning framework and highlight its practical relevance and potential impact in the field of MRI off-resonance correction.\n\n\n\n--\n\n[1] Noll DC, Fessler JA, Sutton BP. Conjugate phase MRI reconstruction with spatially variant sample density correction. IEEE Trans Med Imaging. 2005 Mar;24(3):325-36. doi: 10.1109/tmi.2004.842452. PMID: 15754983.\n\n[2] M. W. Haskell, A. A. Cao, D. C. Noll, and J. A. Fessler, (2020). Deep learning field map estimation with model-based image reconstruction for off-resonance correction of brain images using a spiral acquisition.\n\n[3] Zeng, D. Y., Shaikh, J., Holmes, S., Brunsing, R. L., Pauly, J. M., Nishimura, D. G., Vasanawala, S. S., & Cheng, J. Y. (2019). Deep residual network for off-resonance artifact correction with application to pediatric body MRA with 3D cones. Magnetic Resonance in Medicine, 82(4), 1398-1411. https://doi.org/10.1002/mrm.27825 \n\n[4] Lim, Y., Bliesener, Y., Narayanan, S., & Nayak, K. S. (2020). Deblurring for spiral real-time MRI using convolutional neural networks. Magnetic Resonance in Medicine, 84(6), 3438-3452. https://doi.org/10.1002/mrm.28393\n\n[5] Zbontar, J., Knoll, F., Sriram, A., Murrell, T., Huang, Z., Muckley, M. J., Defazio, A., Stern, R., Johnson, P., Bruno, M., Parente, M., Geras, K. J., Katsnelson, J., Chandarana, H., Zhang, Z., Drozdzal, M., Romero, A., Rabbat, M., Vincent, P., Yakubova, N., Pinkerton, J., Wang, D., Owens, E., Zitnick, C. L., Recht, M. P., Sodickson, D. K., & Lui, Y. W. (2018). fastMRI: An Open Dataset and Benchmarks for Accelerated MRI. ArXiv e-prints, arXiv:1811.08839."}, {"review_id": "qzisYcfgeK", "reviewer": "Reviewer_tQmb", "summary": "The paper presents an innovative approach to off-resonance correction in Magnetic Resonance Imaging (MRI) using a physics-inspired unrolled-deep-learning framework. This framework is designed to address the challenges of off-resonance effects that significantly degrade MRI quality, especially in non-Cartesian trajectories, which are faster and more susceptible to motion artifacts.", "questions": "- To make the writing more clear\n- Recheck the notations\n- Experiment result in whole dataset\n- Clinical study", "limitations": "See the 'weakness' and 'question' part.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The framework has successfully applied to different anatomies and contrasts, such as T1-weighted, T2-weighted, and Proton Density (PD) images, and for different scan trajectories. This wide range of applicability is particularly beneficial, making the model versatile for various clinical settings.\n- The proposed method has the potential to significantly reduce scan times and minimize motion-induced artifacts, leading to improved image quality and patient comfort during the scanning process.\n- Figure 1 is good, which help reader to understand the Off-resonance blurring effects.", "weaknesses": "- The writing is not clear and the authors do not consistently use notations, leading to confusion. For instance, the received signal is represented by s(t) in line 110, while it is denoted as y in line 133. It is unclear what the 't' denotes in s(t) and whether there is a time dimension associated with it.\n- The distinction between the non-uniform fast Fourier transform and the regular fast Fourier transform is not well-explained. If both transforms share the same formula, the choice of the non-uniform fast Fourier transform over the standard version needs further justification. Use of uncommon terminology may create unnecessary confusion.\n- The experimental results presented in the paper are mostly limited to visual examples. Comprehensive results demonstrating the overall performance of the model across the whole dataset are not reported, which makes it hard to assess the model's general performance.\n- The paper has not conducted a full clinical study. This limitation prevents a clear evaluation of the model's robustness, efficacy, and impact on clinical workflows and patient care in real-world scenarios. It is crucial for any imaging technique to undergo extensive clinical testing before it can be deemed reliable and safe for regular use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Yes, we understand the importance of clear and consistent notation to ensure the ease of understanding for all readers, and will clarify raised confusion and improve the writing throughout the paper.\n\nRegarding the notational inconsistency, we acknowledge that this is an oversight on our part. We previously assumed that the equations mentioned in the paper are standard textbook examples; however, due to the reviewer’s comments, we now realize that readers outside the field of MRI might not be familiar with these common notations. Our goal is to provide a comprehensive and clear presentation of our work to benefit all readers, regardless of their background. We apologize for any confusion caused and will update the paper to address this issue and ensure that all notations are explained clearly and consistently throughout the manuscript.\n\nFor the particular example in lines 110 and 133, we would like to clarify that “t” indeed denotes time. The equation for s(t) represents the actual received signal over time, which corresponds to samples at k-space coordinates  (kx(t), ky(t)). From here, we can obtain the corresponding complete sampled k-space associated with the acquired image “y(kx, ky)”. We will make sure to explain this better in the manuscript.\n\nRegarding the distinction between the Non-Uniform Fast Fourier Transform (NUFFT) and the regular Fast Fourier Transform (FFT), both are efficient implementations of the same discrete Fourier Transform formula. FFT assumes that both the spatial image and the spatial frequency domains are discretized uniformly, so NUFFT is commonly used when this is not the case. In our work, we are using a non-uniformly sampled spatial frequency data (non-cartesian trajectories such as Spiral, which are useful for very fast imaging). Hence, we use NUFFT to compute the Fourier Transform. We will make sure to address this in the paper. \n\nRegarding the limited scope of experimental results, the decision to present visual examples was not an oversight but was carefully considered. Off-resonance effects arise due to phase accrued during lengthy data readouts, but reference scans with shorter data readouts will alter image contrast and introduce different artifacts. This makes it problematic to obtain directly comparable ground truth data to compute meaningful quantitative metrics.\n\nWe want to clarify that we have thought about the limitations and have considered other means of providing validation. Given the challenges of acquiring reliable ground truth data, we have considered including additional simulation-based experiments to provide a more comprehensive assessment of our model's performance. However, we want to emphasize that these results would be overly-optimistic compared to an ideal ground truth comparison since our model was fitted to the same simulation framework, so we opted not to include this in the manuscript. Conducting a comprehensive clinical study is not a trivial endeavor. It involves coordinating with hospitals, radiologists, and other resources, which presents challenges beyond the scope of this initial work. We are evaluating the feasibility of conducting a small subject study by the final submission to gather qualitative feedback on image quality. Although not a comprehensive clinical study, this would provide initial insights into the model's diagnostic quality and potential impact on clinical workflows. \n\nTo address the reviewer's concern about the lack of a full clinical study, we believe it is essential to clarify the nature of this technical paper. Our primary contribution lies in the technical innovations for off-resonance correction, and our focus has been on demonstrating the effectiveness and versatility of our proposed framework across various anatomies, contrasts, and scan trajectories. A full clinical study, while valuable, requires significant resources and time beyond the scope of this initial work on technical innovation. However, we intend to undertake such a study in the future as part of the progression of this research.\n"}, {"review_id": "ClttiCKe0o", "reviewer": "Reviewer_XeBV", "summary": "This work proposes a physics-informed unrolled DL framework for off-resonance correction in MRI trained on synthetic data and they demonstrate the effectiveness of their approach on phantom and in-vivo data. Overall, the approach is sound and has a potential big clinical impact. Empirical evaluations are convincing, hence I recommend the paper for acceptance.", "questions": "Would it be possible to open-source the used datasets to make the evaluations more reproducible and potentially seek more attention to this task?", "limitations": "Limitations are sufficiently addressed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Very clear presentation.\n- Potential big clinical impact with the approach, given it can alleviate the current challenges with off-resonance artifacts for non-Cartesian sampling patterns.\n- Generalizability to other non-Cartesian trajectories and contrasts.\n- Alleviation of the need to collect large training datasets for the off-resonance correction task.", "weaknesses": "Given off-resonance correction is a fairly niche task, I am not sure how much the general NeurIPS community would benefit from the paper.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive evaluation of our work and the reviewer's confidence in its impact and soundness. \n\nRegarding open-sourcing our datasets, we are committed to making our work reproducible and accessible to the community. We will release our code (which includes the code for generating the synthetic training data), models, and test data. \n\nWe understand the concern raised about the niche nature of the off-resonance correction task in MRI and its potential relevance to the broader NeurIPS community. The mathematics that arises from off-resonance correction in MRI is a generalization of lens aberration and depth from focus problems in microscopy and astronomy. The technique we propose could be adapted to these cases where there's a system error, like lens imperfection or atmospheric blurring. We can incorporate this into the manuscript as well.\n\nDespite its niche topic, approximately five papers related to MRI have been accepted at NeurIPS in recent years. MRI has also demonstrated a significant presence in workshops such as \"Medical Imaging meets Neurips,\" where on average around 13% of the accepted abstracts are related to the field of MRI, reflecting the acknowledgement and enthusiasm of the NeurIPS community for the field of MRI :) \n"}, {"review_id": "iF4gbwrLC8", "reviewer": "Reviewer_9HEj", "summary": "The Authors describe a physics informed neural network based method to improve the quality and or reduction of MR image acquisition time. The raw signal from magnetic resonance imaging is in Fourier domain, hence normally an inverse Fourier transformation is performed to obtain an image in spatial domain that is much more interpretable for human eye. The manuscript explains how the inhomogeneities in the magnetic field that aligns the nuclear spins.  The nuclear spin resonances of hydrogen depend on the immediate surroundings - so hydrogen in fat has a slightly different nuclear magnetic resonance frequency than hydrogen in water. These off-resonance effects create artefacts in the spatial image.  The effects to the spatial domain image are analogous to motion blurring in normal images with some differences.\n\nThe paper introduces a multi-frequency  forward model for simulations of the distortions with non-uniform Fourier transformations. The model is trained on purely synthetic data and demonstrates clear image quality improvement for in-vivo example cases.\n\n\n", "questions": "See the weakness part.", "limitations": "NA", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "-Excellent presentation with figures that support the narrative.\n-The use of good physically understandable models with synthetic data leads  to  a high image quality and potentially shorter detector times. This is beneficial for clear images as moving of the subject has less time to move.\n-Excellent supplementary material.\n", "weaknesses": "It would be nice to have a discussion on how unique the provided solutions are. As the blurring operation looses information, one could assume that multiple different pixel configurations could produce the same blurred solution. As the question is about fine detail features of medical images, one would like to know if the more detailed features are generative hallucinations or actually more accurate MRI images.\n\nThis is important as the relevant findings are often rare and out of data of normal image distribution - so there is a possibility that the de-blurring operation replaces unusual anomalies with normal looking content that gives the practitioners false sense of security of not finding anything even from a more accurate image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive evaluation and valuable feedback on our work. We are pleased that the presentation, figures, and supplementary material have been well-received.\n\nRegarding the discussion on the uniqueness and potential generative hallucinations in the de-blurring operation, we find this aspect to be of great interest as well. The blurring operation in MRI could indeed result in multiple pixel configurations that produce the same blurred solution. It is essential to ensure that the de-blurring process accurately retains fine detail features of medical images rather than introducing artifacts or hallucinations. We consider two aspects of this issue. \n\nThe condition of the problem depends on the level of off-resonance or the length of the readout, as well as on the structure of the image. In our experiments, we limit our training to trajectories in which off-resonance exhibits up to ~1 water cycle of phase. This is a tunable parameter that can be used in a clinical application to design a trajectory that will not exceed a certain level of blurring. Much like autofocusing, the condition of the problem will depend on the structure of the image. In flat contrast areas, the solution is more under-determined but, at the same time, deblurring is not so essential. Autofocusing is effective when there is structure to focus on.\n\nAdditionally, we emphasize the importance of data consistency in our framework, as these blocks use knowledge of the acquisition physics to ensure reconstructions that agree with the acquired kspace data. Our empirical results on real data show promising outcomes, but we acknowledge that further rigorous clinical validation will be relevant.\n"}]

Author rebuttal:
A figure including a comparison to a deep learning baseline is attached. The deep learning baseline performs decently in regions of water-only content but struggles in regions with fat content (e.g., knee scan (d) and brain outer region (c)).

Meta-review: Resonance properties of hydrogen in fat and water are different, so when calculating MRI, these effects create artifacts in an MRI spatial image that are analogous to motion blurring. The paper introduces a forward model to simulate artifacts in the case of non-uniform Fourier transformation. Based on the forward model, the authors present a physically informed solution to the inverse problem with a data-adaptive regularisation based on a learnable CNN prior. The authors demonstrate that it is possible to train the model on synthetic data and improve the quality of real images.


The paper's topic is quite specific; it is not standard for a general ML audience. The authors use some terminology and discuss approaches/challenges not apparent to non-experts in this domain. The description of the algorithm should be provided in a more explicit and easy-to-follow way. Each step is better to explain based on some simple example. Some basic comments on the data acquisition process should also be provided. Overall, it is important to re-write this part of the paper.


The paper lacks a detailed quantitative experimental study. The authors provided some results in their rebuttal. However, there are already many schemes based on DL and on physics-informed DL to solve problems similar to the considered problem.


To make the final version suitable for publication, we expect the authors to discuss past schemes based on DL and on physics-informed DL (including, for example, K. Hammernik et al., "Physics-Driven Deep Learning for Computational Magnetic Resonance Imaging: Combining physics and machine learning for improved medical imaging," in IEEE Signal Processing Magazine, vol. 40, no. 1, pp. 98-114, Jan. 2023, doi: 10.1109/MSP.2022.3215288. https://ieeexplore.ieee.org/document/10004819) and to compare their approach with any applicable prior work from that literature.  


In addition, we recommend an ablation study of the different components of the proposed scheme. For example, what if we remove a neural network D in (4)? Operator A in (3) depends on frequency characteristics. What if some of them are not precisely known? How do these differences influence the final result? Is the proposed scheme robust w.r.t. these parameters? Is the proposed method robust w.r.t. some disturbance in the operator A?

How to select lambda in (5)? To what extent can we estimate the coil sensitivity maps S? How many iterations do we need until convergence of the proposed scheme (5)? In ML it is very important to understand to what extent the obtained results are stable.


It would be good to do this kind of experimental study on different samples of MRI data.


In conclusion, the paper has promising results; the proposed approach is based on essential and state-of-the-art ideas and solves an important practical problem. I vote for accepting the paper; however, the paper should be further improved before publication.


Comments:
- notations are not always consistent, e.g., x and y are used to denote coordinates, but in line 133 they are used to denote a clean image and k-space measurements,
- I proposed introducing a section with some background information describing a physical model behind MRI, the main steps of the data acquisition process, etc. Otherwise, the paper is not easy to follow for non-experts.
Decision: Accept (poster)