SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alfredo De Goyeneche<sup>1</sup>, Shreya Ramachandran<sup>1</sup>, Ke Wang<sup>1</sup>, Ekin Karasan<sup>1</sup>, Joseph Cheng<sup>2</sup>, Stella X. Yu<sup>1,3</sup>, Michael Lustig<sup>1</sup>
<sup>1</sup> Electrical Engineering and Computer Sciences, University of California, Berkeley
$^{2}$  Radiology, Stanford University
<sup>3</sup> Computer Science and Engineering, University of Michigan
{adg, shreyar, kewang, ekinkarasan, stellayu, lustig}@berkeley.edu
jycheng@stanford.edu, stellayu@umich.edu
Magnetic Resonance Imaging (MRI) is a powerful medical imaging modality that offers diagnostic information without harmful ionizing radiation. Unlike optical imaging, MRI sequentially samples the spatial Fourier domain ( $k$ -space) of the image. Measurements are collected in multiple shots, or readouts, and in each shot, data along a smooth trajectory is sampled. Conventional MRI data acquisition relies on sampling  $k$ -space row-by-row in short intervals, which is slow and inefficient. More efficient, non-Cartesian sampling trajectories (e.g., Spirals) use longer data readout intervals, but are more susceptible to magnetic field inhomogeneities, leading to off-resonance artifacts. Spiral trajectories cause off-resonance blurring in the image, and the mathematics of this blurring resembles that of optical blurring, where magnetic field variation corresponds to depth and readout duration to aperture size. Off-resonance blurring is a system issue with a physics-based, accurate forward model. We present a physics-informed deep learning framework for off-resonance correction in MRI, which is trained exclusively on synthetic, noise-like data with representative marginal statistics. Our approach allows for fat/water separation and is compatible with parallel imaging acceleration. Through end-to-end training using synthetic randomized data (i.e., noise-like images, coil sensitivities, field maps), we train the network to reverse off-resonance effects across diverse anatomies and contrasts without retraining. We demonstrate the effectiveness of our approach through results on phantom and in-vivo data. This work has the potential to facilitate the clinical adoption of non-Cartesian sampling trajectories, enabling efficient, rapid, and motion-robust MRI scans. Code is publicly available at: https://github.com/mikgroup/ResoNet
Magnetic Resonance Imaging (MRI) is a powerful medical imaging modality that enables the acquisition of diagnostic information of soft tissue without harmful ionizing radiation. MRI visualizes hydrogen spins, mostly in water and lipids that are subjected to a very strong yet homogeneous magnetic field. When spins are excited, they emit a signal at a frequency proportional to the magnetic field present. Spatial encoding is possible by applying additional so-called gradient fields, which vary linearly in space and result in a Fourier relationship between the position of the spins and the frequency spectrum of the acquired signal. Since position is mapped to frequency, any additional magnetic field inhomogeneities will result in incorrect mapping and lead to image distortions and undesired artifacts. Magnetic field inhomogeneity can occur due to magnetic susceptibility differences between air and tissue, such as near the sinuses and ear canals in the head or near the lungs and
bowel in the body. Also, due to different molecular environments, spins in water and fat see different magnetic fields, which results in the so-called chemical shift phenomenon in the signal frequency. The severity of these off-resonance effects depends on different imaging parameters, with a general trade-off between scan time or signal-to-noise ratio and the level of artifacts.
1.1 MRI Acquisition While there are numerous intricacies to MRI acquisition, the process can be succinctly understood as the collection of data in the spatial frequency domain, or  $k$ -space, of the imaged object magnetization  $m(\vec{r})$ . The relationship between the image and  $k$ -space data  $M(\vec{k})$  is defined by the Fourier relationship,
$$
M (\vec {k}) = \mathcal {F} \{m (\vec {r}) \} = \int_ {\text {v o l}} m (\vec {r}) e ^ {- j 2 \pi (\vec {k} \cdot \vec {r})} d V. \tag {1}
$$
Here,  $\vec{r}$  is the position vector in the object space, and  $\vec{k}$  are the coordinates in  $k$ -space.
The MRI system employs a  $k$ -space traversal strategy where data is acquired sequentially over time along lines or smooth trajectories  $\vec{k}(t)$ . The time-dependent signal  $y(t)$  corresponds to samples of  $M(\vec{k}(t))$ , described by the equation [1]:
$$
y (t, \vec {k} (t)) = M (\vec {k} (t)) = \int_ {\text {v o l}} m (\vec {r}) e ^ {- j 2 \pi (\vec {k} (t) \cdot \vec {r})} d V \tag {2}
$$
Figure 1 illustrates different two-dimensional  $\widetilde{k}(t)$  trajectories. The  $k$ -space measurements are collected in multiple shots, called readouts, with the scan time increasing linearly with the number of readouts. In most clinical scans,  $k$ -space is sampled in a row-by-row Cartesian manner, which consists of many short readouts. However, these brief readout durations are relatively inefficient in covering  $k$ -space, resulting in prolonged scan times and susceptibility to motion artifacts. Researchers have developed more efficient non-Cartesian sampling trajectories, such as Spiral, which utilize longer but fewer readouts, allowing traversal of  $k$ -space in a significantly shorter time. However, these longer readouts make the trajectory more susceptible to off-resonance effects.
1.2 Off-Resonance in MRI Off-resonance arises mainly from two sources: 1) undesired spatial variations in the main magnetic field, which cause spins to resonate at a frequency offset  $\Delta f(\vec{r})$
Figure 1: Trade-offs between MRI acquisition strategies and off-resonance effects: The MRI system directly samples the spatial Fourier domain ( $k$ -space)  $M(\vec{k})$  of the object  $m(\vec{r})$  in many shots, called readouts, each represented a by yellow arrowed path. The illustrated acquisition strategies assume a fully sampled  $k$ -space, where off-resonance effects depend on readout duration and geometry. Cartesian trajectories collect  $k$ -space samples in many short readouts in a row-by-row manner, with minimal off-resonance artifacts. Echo-planar imaging (EPI) and Spiral trajectories collect  $k$ -space samples via longer but fewer readouts in a significantly shorter total time. However, these are more susceptible to off-resonance effects, manifesting as image distortions in EPI and image blur in Spiral.
proportional to the field inhomogeneities, and 2) the inherent chemical properties of tissue types in the object, such as fat and water, where spins in fat tissue resonate at a fixed frequency offset  $\Delta f_{\mathrm{fat}}$  from those in water. These offsets are relative to a carrier frequency  $f_0$  - the resonant frequency of hydrogen protons in water - which we omit in our equations for simplicity, presenting instead the baseband signal. Equation 2 assumed a homogeneous magnetic field and single tissue type, specifically water; however, since images contain both water and fat components and both components experience the spatially varying off-resonance  $\Delta f(\vec{r})$ , Equation 2 can be extended to
$$
y (t, \vec {k} (t)) = \int_ {\text {v o l}} \left(m _ {\text {w a t e r}} (\vec {r}) + m _ {\text {f a t}} (\vec {r}) e ^ {- j 2 \pi \Delta f _ {\text {f a t}} t}\right) e ^ {- j 2 \pi \Delta f (\vec {r}) t} e ^ {- j 2 \pi \vec {k} (t) \cdot \vec {r}} d V. \tag {3}
$$
In this equation, the additional exponential terms introduce space-varying linear phase modulation to image samples, where the range of the phase modulation increases with the readout duration,  $t$ , thus breaking the Fourier model. Therefore, reconstruction using an inverse Fourier transform will result in image artifacts that depend on the trajectory used.
In Cartesian trajectories, all the readouts are in the same direction, accruing linear phase across  $k$ -space. This results in image domain shift according to the Fourier shift property and thus to geometric distortion for space-varying inhomogeneities. Due to the short duration of Cartesian readouts, these off-resonance artifacts are negligible compared to those from trajectories with longer readouts, like Spirals (Figure 1). Please refer to [2] for a detailed derivation of the exact shift. Since Spiral trajectories go outwards from the center of  $k$ -space, the phase accrual is center-out. This results in spatial frequency components being displaced in different directions, leading to blurring and loss of sharpness in the image. The Point Spread Function (PSF) of this effect corresponds to a spatially varying blurring kernel proportional to the amount of local off-resonance. Please refer to the supplementary material for visual representations of these PSFs.
The mathematics underlying off-resonance blurring resembles that of optical blurring due to depth, as illustrated in Figure 2. The spatially varying magnetic field is mathematically similar to a spatial variation in depth, while the trajectory readout length is analogous to the size of the optical aperture. Just like larger apertures increase both SNR and blurring from depth variations, longer readouts increase both SNR and blurring from variations in resonant frequency. However, unlike blurring in optical systems, off-resonance blurring is additive, layering out-of-focus regions on top of those in focus. One could bring off-resonant regions into focus by demodulating the signal equation with a specific phase offset (Figure 2). For instance, extracting the fat phase term  $e^{-j2\pi \Delta f_{\mathrm{fat}}t}$  from the integral in Equation 3 and demodulating by it makes the fat image components sharp, leaving water components blurred. Also, MRI introduces an additional complexity known as partial volume effects [3], where a single voxel can contain both resonant (water) and off-resonant (fat) spins.
1.3 Our Approach Off-resonance blurring is a system problem and has a known forward model, but solving for a direct solution is challenging due to its ill-posed nature. Recently, Deep Learning (DL) has enabled significant improvements over classical methods for solving inverse problems [5-8]. However, there are concerns over the black-box nature of end-to-end networks, overfitting, potential hallucinations [9], and the necessity of large representative training datasets. Physics-inspired DL methods, such as model-based unrolled networks [6, 10], mitigate these concerns by enforcing consistency to the acquired data with the known forward model of the system. Utilizing a forward model also enables the simulation of a training dataset, circumventing the need to collect large amounts of training data on an MRI scanner representative of all anatomies and contrasts.
In this work, we propose a physics-inspired unrolled-DL framework for off-resonance correction in MRI. The key contributions of this work are as follows:
- End-to-end training using only synthetic data: A distinguishing feature of our approach is the exclusive utilization of synthetic data with representative marginal statistics for network training (inspired by [11]). By leveraging synthetic random field maps, partial volume effects, coil sensitivities, and noise-like images, we eliminate the requirement for real MRI data during the training process. This approach allows us to overcome the limitations of acquiring large training datasets on an MRI scanner, which can be costly, time-consuming, and unfeasible in diverse scenarios.
- Estimation of a resonance frequency spectrum per voxel using a physics-informed DL framework: This effectively handles partial volume effects, i.e., separates fat and water contributions within each voxel. While we demonstrate results for a Spiral trajectory, we propose a generic framework that could be applied to other non-Cartesian trajectories.
Figure 2: Illustration of off-resonance blurring effects in Spiral MRI and analogy with Depth of Focus. Top row: Water-based phantom data exhibiting off-resonance effects. The reference short-readout image required  $4\mathrm{x}$  the scan time. Middle row: a simulated example with non-overlapping water and fat circles. Demodulating at the water resonance frequency  $(f_0)$  brings the water circle into focus but causes blurring of fat, and vice versa. Bottom row: [4] Analogy with depth of focus in optical imaging, where focal point is sharp, while other depths are blurred. In MRI, different demodulation frequencies correspond to different focal lengths. Unlike optical blurring, off-resonance blurring is additive, with out-of-focus regions blurring on top of in-focus areas, making it challenging to create a seamless all-in-focus image by stitching together multiple demodulated images.
- Generalizability to multiple anatomies and contrasts: The proposed framework demonstrates strong generalizability, effectively correcting off-resonance artifacts across different anatomies and contrasts without the need for retraining.
1.4 Related Work Recent developments in MRI highlight the potential of physics-driven DL methods in reconstruction tasks, spanning from physics-informed loss functions to unrolled network architectures [6, 10]. In off-resonance correction, traditional approaches include direct deblurring methods like autofocusing [12], which iteratively determines the sharpest image resonance frequency per voxel; however, these often rely on simplified sharpness criteria from optics [13], limiting their ability to generalize to more complex off-resonance scenarios encountered in MRI, such as partial volume effects. Model-based analytical methods, such as conjugate phase reconstruction [14], require a magnetic field map measured from additional scans. The field map model does not represent partial volume effects, introduces additional scan time, and is susceptible to motion artifacts; hence, this method often cannot completely correct for off-resonance [15].
More recently, data-driven DL-based approaches have been proposed to use convolutional neural networks (CNNs) to directly deblur the image [16-18] or estimate the field map from the image [19]. However, these approaches often neglect the physics of off-resonance blurring, do not handle partial volume effects, are limited in handling multi-coil acceleration, and rely on specific training data tailored to certain anatomies and contrasts.
Our proposed approach addresses the limitations of previous off-resonance correction methods by developing a forward model that handles partial volume effects and is compatible with multi-coil acceleration. By slicing the object into multiple frequency bins, we represent the tissue resonant at each frequency, leading to a comprehensive model that captures the full spectrum of frequency offsets. Our approach recognizes that off-resonance is a system problem intrinsic to the MRI acquisition process rather than a direct image feature problem. As a result, we leverage training on synthetic
random data, providing an effective strategy to address this challenge and improve generalization across different anatomies and contrasts.
2.1 Physics-Informed Forward Model The key in our approach is that we do not model the spatially varying field and fat shift directly. Instead, we extend the dimensionality of our image to  $x(\vec{r},\Delta f)$ , which includes a frequency dimension  $\Delta f$  such that:
$$
m (\vec {r}) = \int_ {\Delta f} x (\vec {r}, \Delta f) d \Delta f \tag {4}
$$
We approximate this representation as a sum of multiple frequency bins  $x_{i}(\vec{r}) = x(\vec{r},\Delta f_{i})$ , such that  $m(\vec{r})\approx \sum_{i}x_{i}(\vec{r})$ , with each bin resonating at a uniform frequency  $\Delta f_{i}$  due to field inhomogeneities and water/fat frequency offsets. Relating back to the optical imaging analogy, this is equivalent to representing a 3D scene instead of a 2D scene with a depth map. Using this modeling and the linearity of the signal equation 3, we can express the contribution to the signal from each frequency bin  $\Delta f_{i}$  as
$$
y _ {i} (t, \vec {k} (t)) = \int_ {\text {v o l}} x _ {i} (\vec {r}) e ^ {- j 2 \pi \Delta f _ {i} t} e ^ {- j 2 \pi \vec {k} (t) \cdot \vec {r}} d V, \tag {5}
$$
where  $y(t,\vec{k}(t)) = \sum_{i}y_{i}(t,\vec{k}(t))$ . Since the phase modulation in a bin applies uniformly to the image bin content, it can be taken outside of the integral, i.e.,
$$
y _ {i} (t, \vec {k} (t)) = e ^ {- j 2 \pi \Delta f _ {i} t} \int_ {\text {v o l}} x _ {i} (\vec {r}) e ^ {- j 2 \pi \vec {k} (t) \cdot \vec {r}} d V. \tag {6}
$$
Discretizing the Fourier Transform, we obtain
$$
y _ {i} = M _ {\Delta f _ {i}} E x _ {i}, \tag {7}
$$
where  $\mathrm{E}$  is an encoding matrix implementing a Non-Uniform Fourier Transform (NUFFT), and  $M_{\Delta f_i}$  is a diagonal matrix with elements  $e^{-j2\pi \Delta f_it}$  in the diagonal.  $M_{\Delta f_i}$  phase modulates the  $k$ -space of the bin by the sample time and frequency offset.
Putting it all together, we can construct a complete forward model
$$
y = \sum_ {i} y _ {i} = A x, \tag {8}
$$
where  $x$  is the multi-frequency bin image, and  $A$  is the complete encoding matrix which combines the Fourier and modulation operators. Figure 3 illustrates our forward model.
It is important to note that for the sake of brevity and simplicity, we ignore in our description the use of multiple receiver coils in MRI, which readers can refer to in the following references [20, 21]. However, by using a general linear encoding operator  $E$  it is trivial to extend our approach from a single receiver in which  $E$  represents a Fourier operator to a multiple receiver case where  $E$  also includes multi-channel receiver sensitivities.
Figure 3: Physics-informed forward model  $A$ : The image is modeled as a stack of sharp images at different frequency bins. Each bin image is encoded using an encoding matrix  $E$  (e.g., Non-Uniform Fast Fourier Transform (NUFFT)), and then phase modulated by  $M_{\Delta f_i}$  corresponding to its bin frequency. Finally, the  $k$ -spaces across the bins are summed  $(\Sigma)$  to reconstruct the acquired  $k$ -space.
2.2 Synthetic Training Data Generation Inspired by [11], all the training data consists of simulated synthetic random field maps, fat/water partial-volume effects, coil sensitivities, and noise-like images. The statistics and smoothness of these synthetic images are designed to model those of real data. Specifically, we generate random images and field maps  $\Delta f(\vec{r})$  by applying an FFT to exponentially weighted random complex data, where the weighting radius determines the level of smoothness. Additionally, to create image examples  $I$ , we combine multiple random images to obtain edges with various sharpness levels. Similarly, random sensitivity maps  $S$  are generated from weighted random SPIiT kernels [22].
To model the fat-water partial volume effects, we generate a random weight map  $W$  that determines the percentage of fat vs. water at each voxel (Figure 4). From this weight map and image  $I$ , we obtain a fat image  $I_{\mathrm{fat}}$  and a water image  $I_{\mathrm{water}}$ . Since fat has a resonant frequency offset from water, in addition to the random magnetic field map frequency offset  $\Delta f(\vec{r})$ , the fat image also includes a constant offset  $\Delta f_{\mathrm{fat}}$ . Subsequently, the fat and water images are assigned to different frequency bins based on the random field map  $\Delta f(\vec{r})$ , fat offset  $\Delta f_{\mathrm{fat}}$ , and the corresponding bin frequency values. This results in a multi-bin image representation  $x$ , fed into the forward model  $A$  to obtain the simulated  $k$ -space data  $y$ .
2.3 Model Architecture To address the ill-posed nature of the problem, we employ a neural network for regularization. Inspired by the MoDL approach [6], we adopt an unrolled iterative model that is trained end-to-end. The input to the model is  $A^H y$ , a stack of images demodulated at multiple frequencies obtained by applying the adjoint operator  $A^H$  on the raw  $k$ -space  $y$ . Regions of the object are sharp at the frequency bin corresponding to their local resonant frequency, while other off-resonant regions blur on top (Figure 5b). The objective of our model is to obtain a clean image at each frequency bin, as well as a combined output image, a water image, and a fat image, all without off-resonance blurring.
The unrolled model consists of Data Consistency (DC) modules and CNN-based proximal steps, as illustrated in Figure 5b. The objective function for the DC module is defined as follows:
$$
x ^ {k} = \arg \min  _ {x} | | A x - y | | _ {2} ^ {2} + \lambda | | x - D _ {\phi_ {k}} \left(x ^ {k - 1}\right) | | _ {2} ^ {2} \tag {9}
$$
Here,  $x^{k}$  represents the reconstructed image at iteration  $k$ ,  $A$  is the forward model,  $y$  is the acquired data,  $D_{\phi_k}$  denotes the CNN with learnable parameters  $\phi_k$ , and  $\lambda$  controls the trade-off between data consistency and regularization. The objective aims to minimize the discrepancy between the forward model's prediction and the acquired data, as well as the difference between the reconstructed image and the output of the preceding CNN. From Equation 9, we can obtain:
$$
\left(A ^ {H} A + \lambda I\right) x ^ {k} = A ^ {H} y + \lambda D _ {\phi_ {k}} \left(x ^ {k - 1}\right), \tag {10}
$$
which can be solved using the Conjugate Gradient (CG) method [23] to update  $x^{k}$  at each iteration.
The CNN takes as input complex bin images with the real and imaginary components represented as separate channels. It includes residual blocks [24] and incorporates an attention module over
Figure 4: Synthetic training data generation. Random noise-like image  $I$ , fat/water percentage map  $W$ , off-resonance field map  $\Delta f(\vec{r})$  are generated by applying an Inverse Fourier Transform to exponentially weighted white noise images, where the weighting radius controls smoothness. From  $I$  and  $W$ , fat and water images,  $I_{\mathrm{fat}}$  and  $I_{\mathrm{water}}$ , are obtained. Each voxel in these images is assigned to a frequency bin based on  $\Delta f(\vec{r})$  and the fat frequency offset  $\Delta f_{\mathrm{fat}}$ , obtaining our multi-frequency image  $x$ . Finally,  $x$  is processed via the forward model  $A$  to generate the  $k$ -space data  $y$ .
Figure 5: Physics-informed deep learning framework for MRI off-resonance correction. (a) Training process: The model is trained using randomly generated data, including images  $(I, I_{\mathrm{water}}, I_{\mathrm{fat}})$  and frequency bins  $(x)$ . Simulated k-space data  $y$  is obtained by applying the forward model  $A$  to  $x$ . The network, fed with images demodulated from  $A^H y$ , reconstructs frequency bins  $\hat{x}$  and images  $\hat{I}, \hat{I}_{\mathrm{fat}}, \hat{I}_{\mathrm{water}}$ , and is directly supervised by the original data. (b) Inference process: Acquired k-space  $y$  is processed through  $A^H$  and fed to the model to reconstruct bins and images. The unrolled network comprises  $N$  unrolls of CNNs and Data Consistency (DC) modules. Each DC module takes  $A, y$ , and previous CNN estimate  $D_{\phi_k}(x^{k-1})$  to solve for  $x^k$  using the Conjugate Gradient method.
frequency bins. This module allows the network to selectively focus on features from specific frequency ranges at each spatial location. Following the attention module, a series of residual blocks are employed. Finally, a convolutional layer produces the cleaner complex images at each frequency, which can then be fed into the DC module. Additionally, the network of the last unroll outputs a final combined image  $\hat{I}$ , a fat image  $\hat{I}_{\mathrm{fat}}$ , and a water image  $\hat{I}_{\mathrm{water}}$ .