SlowGuess/NeurIPS_2023_OCR huggingface dataset:
2.4 Model Training and Inference During training, we utilize our synthetic data strategy to generate images  $I$ ,  $I_{\mathrm{water}}$ ,  $I_{\mathrm{fat}}$ , and the bin representation  $x$ , from which we obtain simulated  $k$ -space  $y$  data using the forward model  $A$ . The unrolled model is fed with  $A^H y$  as input, alongside  $A$  and  $y$  for the DC modules. We task the model with reconstructing the bin representation  $\hat{x}$  and images  $\hat{I}$ ,  $\hat{I}_{\mathrm{water}}$ , and  $\hat{I}_{\mathrm{fat}}$ , with direct supervision provided by the synthetic data (Figure 5a).
The forward model  $A$  incorporates the acquisition characteristics of the readout trajectory in the NUFFT step, such as the type of non-Cartesian  $k$ -space trajectory, readout duration, number of interleaves, resolution, and field of view (FOV). The model learns to correct for a specific PSF associated with the trajectory, and a single model becomes versatile and adaptable to various anatomies and MRI contrasts, such as proton density (PD),  $T_{1}$ -weighted, and  $T_{2}$ -weighted images [1].
During inference, real acquired k-space data  $y$ , the forward model  $A$ , and  $A^H y$  are provided to the network to reconstruct the bin representation and images (Figure 5b).
3.1 Phantom and In-vivo Acquisitions To evaluate the performance of our model, we acquired phantom data, as well as brain, knee, and abdominal in-vivo data using a GE 3T MR750W System. MR Pulse Sequences were designed using SpinBench Software, and data was acquired using RTHawk
v2.5.2 Software (Vista.ai Inc., Palo Alto, CA.). The in-vivo data was acquired from three volunteers who provided informed consent, and the study was conducted with IRB approval.
For the phantom, knee, and brain scans, we used a Spiral trajectory with 36 interleaves, a  $5.2\mathrm{ms}$  readout duration, a resolution of  $1\mathrm{mm}^2$ , and a  $23\mathrm{cm}$  FOV. We acquired Proton Density (PD) weighted images for the knee and T1-weighted images for the brain. Alongside each scan, we acquired a reference scan with the same resolution and FOV but with a shorter readout (148 interleaves, 1.48 ms readout) to reduce artifacts. It is worth noting that for the knee scan, where we used a 2-second repetition time (TR), the acquisition time using the shorter readout trajectory is 5 minutes for a single slice, compared to 1.2 minutes for the longer readout sequence, resulting in a 4x speedup.
T1-weighted abdominal scans employed a 38-interleave Spiral trajectory with a  $10.4\mathrm{ms}$  readout duration, a  $1\mathrm{mm}^2$  resolution, and a  $35~\mathrm{cm}$  FOV, with data acquired during breath holds. A short readout reference scan was also acquired (320 repetitions,  $1.52\mathrm{ms}$  readout duration); however, as breathing was permitted between scans, slices will not exactly align. With a  $50\mathrm{ms}$  TR, the long readout trajectory takes less than 2 seconds to acquire versus 16 seconds for the short readout one.
More details on Pulse Sequences, coils, and other parameters used are in the supplementary material.
3.2 Model and Training Details In a 3T MRI, the fat frequency offset,  $\Delta f_{\mathrm{fat}}$ , is reported to be approximately  $-440\mathrm{Hz}$  [1, 25]. In our models, we simulated an off-resonance field map covering a range of  $\Delta f(r) = \pm 180\mathrm{Hz}$ . To approximate the off-resonance spectrum, we utilized 21 frequency bins within the range of  $-600\mathrm{Hz}$  to  $200\mathrm{Hz}$ , with a step size of  $40\mathrm{Hz}$  between bins. To better capture partial volume effects and avoid subtle data artifacts [26], data generation was performed at a high resolution, three times better than the reconstructed spatial resolution and ten times the reconstructed frequency bin resolution. Subsequently, the generated data was decimated in the Fourier domain to match the target resolution and dimensions.
The network architecture featured four unrolls. Each CNN module in these unrolls included three residual blocks, each containing two convolutional layers with 128 filters and a kernel size of 5, employing ReLU activation. The DC module used 12 conjugate gradient iterations with a trainable regularization parameter  $\lambda$ , empirically initialized to ensure that a ground truth input remained uncorrupted ( $\lambda = 1$ ). We trained our model using PyTorch [27] in a Nvidia RTX3090 GPU, employing  $\ell_1$  losses and the Adam optimizer [28]. We used the torchkbnuff library [29] for NUFFT operations. We pre-generated a training dataset to optimize memory usage and speed during training.
Two models were trained: one for the  $23\mathrm{cm}$  FOV trajectory, and another for the  $35\mathrm{cm}$  FOV scans.
In addition to our proposed approach, we aimed to replicate the methods in [16] and [17] to provide a relevant baseline for comparison, referred to as the "DL Baseline". We used the fastMRI brain dataset [30] with simulated Spiral trajectories and employed the network architecture and augmentation strategy proposed by [16] for training. It is important to note that this method only outputs a combined corrected image, without separate fat and water images or a spectrum across frequencies.
3.3 Results on Phantom and In-vivo The phantom and in-vivo results were obtained using models for specific trajectories and trained solely on noise data. Sensitivity maps for the acquired  $k$ -space  $y$  were estimated using ESPIRiT [31] with the BART toolbox [32]. Though these maps may not always be perfectly accurate, they are fed to the model alongside the  $k$ -space, and our model remained robust to their potential inaccuracies.
Figure 6 showcases a phantom scan, a T1-weighted brain scan, a Proton Density (PD) weighted knee scan, and a T1-weighted abdominal scan, illustrating the capability of our approach to generalize across multiple anatomies and contrasts. The figures depict 1) the uncorrected long readout input data demodulated at the water frequency  $f_{0}$ , corresponding to the image a clinician would use for diagnosis, 2) the predicted water image output, 3) fat image output, and 4) combined clean output. Additionally, we compare our results against 5) the DL Baseline described in section 3.2 and 6) Autofocus [12] technique, which both aim to correct without the need for a separately acquired field map. Finally, we present 7) the uncorrected short readout reference image and 8) the reference field map obtained with separate scans. Please refer to the supplementary material for more detailed results showing all input and network output bins.
Figure 6: Results of the proposed method on phantom and in-vivo data. Results are presented for Water-based ACR phantom, as well as in-vivo T1-weighted brain, PD-weighted knee, and  $T_{1}$ -weighted abdominal scans. Top-to-bottom rows: 1) The uncorrected long readout input image, 2) output water image  $\hat{I}_{\mathrm{water}}$ , 3) output fat image  $\hat{I}_{\mathrm{fat}}$ , 4) combined output image  $\hat{I}$ . Reference baselines include 5) DL Baseline output image, 6) Autofocus [12] corrected image, 7) reference uncorrected short readout image (4x scan time for phantom, brain, and knee; 8x for abdominal scans). Lastly, 8) reference field map to visualize resonance frequency variations.
3.4 Results on Simulated Off-Resonance Validation Set While obtaining reference scans with shorter data readouts is possible, these scans will alter image contrast and introduce different artifacts. As a result, obtaining directly comparable ground truth data to compute meaningful quantitative metrics is infeasible. Therefore, as a performance proxy, we conducted an evaluation on a validation set created using the fastMRI brain dataset [30]. This set consists of brain anatomy images that were corrupted with simulated random off-resonance field variations and fat/water partial volume effects. The validation set comprised 1000 examples.
We compared the performance of our proposed approach and the DL Baseline by measuring the NRMSE (Normalized Root Mean Squared Error) and PSNR (Peak Signal to Noise Ratio) on the magnitude of the combined output images within the validation set. We report average metrics and their standard deviation over the 1000 examples in Table 1. Despite our approach being trained solely on synthetic non-anatomy data, it outperforms the DL Baseline regarding NRMSE and PSNR.
Method \ Metric	NRMSE (↓)	PSNR (↑)
Our approach	0.016 ± 0.011	37.1 ± 4.7
Deep Learning (DL) Baseline	0.049 ± 0.028	27.7 ± 5.7
Table 1: Performance comparison on simulated off-resonance validation set. NRMSE (lower is better) and PSNR (higher is better) for our method and the DL Baseline on a synthetic validation set of fastMRI [30] brain images with simulated random off-resonance and partial volume effects. Metrics show the average and standard deviation over 1000 examples.
In this work, we propose a physics-inspired unrolled DL framework for off-resonance correction in MRI, showcasing its effectiveness through training solely on synthetic data. This approach addresses the challenges of collecting real MRI datasets for off-resonance correction, which can be unfeasible. Off-resonance blurring varies greatly across different anatomies due to differences in magnetic field variations, as seen in Figure 6. Our in-vivo evaluation showcases the generalization capabilities of our model across various contrasts and anatomies. Correcting off-resonance in MRI is particularly valuable for enabling non-Cartesian trajectories, such as Spirals, which significantly reduce scan time, leading to fewer motion-induced artifacts, shorter MRI exams, and improved clinical throughput.
The proposed framework and training strategy could bring further value to fields outside of MRI. The mathematics that arises from off-resonance correction in MRI is a generalization of lens aberration and depth from focus problems in microscopy and astronomy [33, 34]. The proposed technique could be adapted to these cases where there is a system error, like lens imperfection or atmospheric blurring.
4.1 Limitations and Future Work Despite promising results, our framework has limitations and areas for future improvement. Currently, networks are trained for specific trajectories, and future work involves handling multiple trajectories within a single model. This will include diverse trajectory patterns (such as Rosette [35], Looping Star [36]), resolutions, and FOVs. Another aspect for future improvement is simulating more extreme magnetic field map variations during training and increasing forward model resolution and network capacity, allowing for correction in settings such as regions close to air in the ear canals and nose. Currently, performance degrades when moving beyond the distribution of the training data simulation range.
In conclusion, we have presented a physics-informed unrolled DL framework for off-resonance correction in MRI. Our approach incorporates data consistency using a multi-frequency forward model that represents a resonance frequency spectrum for each voxel. We have bypassed the challenges of collecting large training datasets by training our network solely on synthetic data.
We have demonstrated results in generalization and performance across diverse anatomies and contrasts. By addressing off-resonance issues, our framework has implications for accelerating scanning, increasing clinical throughput, and making MRI more accessible. Continued research and refinement in this area will further contribute to the advancement of MRI technology and its application in healthcare.
We acknowledge support from GE Healthcare and NIH grants R01EB009690 and U01EB029427. The authors would like to thank Suma Anand, Justine Taylor, and Ana Cismaru for their help with the paper editing.