Data augmentation for deep learning based accelerated MRI reconstruction

1 INTRODUCTION . In magnetic resonance imaging ( MRI ) , an extremely popular medical imaging technique , it is common to reduce the acquisition time by subsampling the measurements , because this reduces cost and increases accessibility of MRI to patients . However , since due to the subsampling , there are fewer equations than unknowns , the signal is not uniquely identifiable from the measurements . To overcome this challenge there has been a flurry of activity over the last decade aimed at utilizing prior knowledge about the signal , in a research area referred to as compressed sensing ( Candes et al. , 2006 ; Donoho , 2006 ) . Classical compressed sensing methods reduce the required number of measurements by utilizing prior knowledge about the images during the reconstruction process , often via a convex regularization that enforces sparsity in an appropriate transformation of the image . More recently , deep learning techniques have been used to enforce much more nuanced forms of prior knowledge ( see Ongie et al . ( 2020 ) and references therein for an overview ) . The most successful of these approaches aim to directly learn the inverse mapping from the measurements to the image by training on a large set of training data consisting of signal/measurement pairs . This approach often enables faster reconstruction of images , but more importantly , deep learning techniques yield significantly higher quality reconstructions . This in turn enables fewer measurements further reducing image acquisition times . For instance , in an accelerated MRI competition known as FastMRI Challenge ( Zbontar et al. , 2018 ) , all the top contenders used deep learning reconstruction techniques . Contrary to classical compressive sensing approaches , however , deep learning techniques typically rely on large sets of training data often consisting of images along with the corresponding measurement . This is also true about the use of deep learning techniques in other areas such as computer vision and Natural Language Processing ( NLP ) were superb empirical success has been observed . While large datasets have been harvested and carefully curated by tech companies in areas such as vision and NLP , this is not feasible in many scientific applications including MRI . It is difficult and expensive to collect the necessary datasets for a variety of reasons , including patient confidentiality requirements , cost and time of data acquisition , lack of medical data compatibility standards , and the rarity of certain diseases . A common strategy to reduce reliance on training data in classification tasks is data augmentation . In a classification setting data augmentation consists of adding additional synthetic data obtained by performing invariant alterations to the data ( e.g . flips , translations , or rotations ) which do not affect the labels . Such data augmentation techniques are commonly used in classification tasks to significantly increase the performance on standard benchmarks such as ImageNet and CIFAR-10 . More specific to medical imaging , data augmentation techniques have been successfully applied to registration , classification and segmentation of medical images . More recently , several studies ( Zhao et al. , 2020b ; Karras et al. , 2020 ; Zhao et al. , 2020a ) have demonstrated that data augmentation can significantly reduce the data needed for GAN training for high quality image generation . In regression tasks such as image reconstruction , however , data augmentation techniques are less common and much more difficult to design in part due to the lack of the aforementioned invariances ( e.g. , measurements of a rotated image are not the same as measurements from the original image ) . The goal of this paper is to explore the benefits of data augmentation techniques for accelerated MRI with limited training data . By carefully taking into account the physics behind the MRI acquisition process we design a data augmentation pipeline , which we call MRAugment , that can successfully reduce the amount of training data required . Specifically , our contributions are as follows : • We propose a data augmentation technique tailored to the physics of the MR reconstruction problem . We note that it is not obvious how to perform data augmentation in the context of inverse problems , because by changing an image to enlarge the training set , we do not automatically get a corresponding measurement , contrary to classification problems , where the label is retained . In fact , we demonstrate that naive forms of data augmentation that do not properly take into account the underlying physics do not work . • We demonstrate the effectiveness of MRAugment on a benchmark accelerated MRI data set , specifically on the fastMRI ( Zbontar et al. , 2018 ) dataset . For 8-fold acceleration and multi-coil measurements ( multi-coil measurements are the standard acquisition mode for clinical practice ) we can achieve performance comparable to the state of the art with only 10 % of the training data . Similarly , again for 8-fold acceleration and single-coil experiments ( an acquisition mode popular for experimentation ) MRAugment can achieve the performance of reconstructions methods trained on the entire data set while using only 33 % of the training data . • We perform experiments showing that in a low-data regime , where we have only limited training data MRAugment is beneficial , whereas in the high-data regime it has diminishing returns . In particular we demonstrate that in a low data regime ( ≈ 1 % of training data ) , data augmentation very significantly boosts reconstruction performance and captures diagnostically significant detail that is missed without data augmentation . In a moderate data regime ( ≈ 10 − 33 % of training data ) , MRAugment still achieves significant improvement in reconstruction performance and may help to avoid hallucinations caused by overfitting without the use of DA . • Finally , while we focus on the top performing neural networks , MRAugment can seamlessly integrate with any deep learning model and therefore it can be useful in a variety of MRI problems . 2 BACKGROUND AND PROBLEM FORMULATION . In this section , we provide a brief background on accelerated MRI and formulate the problem . MRI is a medical imaging technique that exploits strong magnetic fields to form images of the anatomy . MRI is a prominent imaging modality in diagnostic medicine and biomedical research because it does not expose patients to ionizing radiation , contrary to competing technologies such as computed and positron emission tomography ( CT and PET ) . However , performing an MR scan is time intensive , which is problematic for the following reasons . First , patients are exposed to long acquisition times in a confined space with high noise levels . Second , long acquisition times induce reconstruction artifacts caused through patient movement , which sometimes requires patient sedation in particular in pediatric MRI ( Vasanawala et al. , 2010 ) . Reducing the acquisition time can therefore increase both the accuracy of diagnosis and patient comfort . Furthermore , decreasing the acquisition time needed allows more patients to receive a scan using the same machine . This can significantly reduce patient cost , since each MRI machine comes with a high cost to maintain and operate . Since the invention of MR in the 1980s there has been tremendous research focusing on reducing their acquisition time . The two main ideas are to i ) perform multiple acquisitions simultaneously ( Sodickson & Manning , 1997 ; Pruessmann et al. , 1999 ; Griswold et al. , 2002 ) and to ii ) subsample the measurements , known as accelerated acquisition or compressive sensing ( Lustig et al. , 2008 ) . Most modern scanners combine both techniques , and therefore we consider such a setup . 2.1 ACCELERATED MRI ACQUISITION . In magnetic resonance imaging , measurements of a patient ’ s anatomy are acquired in the Fourierdomain , also called k-space , through receiver coils . In the single-coil acquisition mode , the k-space measurement k ∈ Cn of a complex-valued ground truth image x∗ ∈ Cn is given by k = Fx∗ + z , where F is the two-dimensional Fourier-transform , and z ∈ Cn denotes additive noise arising in the measurement process . In parallel MR imaging , multiple receiver coils are used , each of which captures a different region of the image , represented by a complex-valued sensitivity map Si . In this multi-coil setup , coils acquire k-space measurements modulated by their corresponding sensitivity maps : ki = FSix∗ + zi , i = 1 , .. , N , where N is the number of coils . Obtaining fully-sampled k-space data is time-consuming , and therefore in accelerated MRI we decrease the number of measurements by undersampling in the Fourier-domain . This undersampling can be represented by a binary mask M that sets all frequency components not sampled to zero : k̃i =Mki , i = 1 , .. , N. We can write the overall forward map concisely as k̃ = A ( x∗ ) , whereA ( ⋅ ) is the linear forward operator and k̃ denotes the undersampled coil measurements stacked into a single column vector . The goal in accelerated MRI reconstruction is to recover the image x∗ from the set of k-space measurements k̃ . Note that—without making assumptions on the image x∗—it is in general impossible to perfectly recover the image , because we have fewer measurements than variables to recover . This recovery problem is known as compressive sensing . To make image recovery potentially possible , recovery methods make structural assumptions about x∗ , such that it is sparse in some basis or implicitly that it looks similar to images from the set of training images . 2.2 TRADITIONAL ACCELERATED MRI RECONSTRUCTION METHODS . Traditional compressed sensing recovery methods for accelerated MRI are based on assuming that the image x∗ is sparse in some dictionary , for example the wavelet transform . Recovery is then posed as a typically convex optimization problem : x̂ = arg min x ∥A ( x ) − k̃∥2 +R ( x ) , whereR ( ⋅ ) is a regularizer enforcing sparsity in a certain domain . Typical functions used in CS based MRI reconstruction are ` 1-wavelet and total-variation regularizers . These optimization problems can be numerically solved via iterative gradient descent based methods . 2.3 DEEP LEARNING BASED MRI RECONSTRUCTION METHODS . In recent years , several deep learning algorithms have been proposed and convolutional neural networks established new state-of-the-art in MRI reconstruction significantly surpassing the classical baselines . Encoder-decoder networks such as the U-Net ( Ronneberger et al. , 2015 ) and its variants were successfully used in various medical image reconstruction ( Hyun et al. , 2018 ; Han & Ye , 2018 ) and segmentation problems ( Çiçek et al. , 2016 ; Zhou et al. , 2018 ) . These models consist of two sub-networks : the encoder repeatedly filters and downsamples the input image with learned convolutional filters resulting in a concise feature vector . This low-dimensional representation is then fed to the decoder consisting of subsequent upsampling and learned filtering operations . Another approach that can be considered a generalization of iterative compressed sensing reconstructions consists of unrolling gradient descent iterations and mapping them to a cascade of sub-networks ( Zhang & Ghanem , 2018 ) . Several variations of this unrolled method have been proposed recently for MR reconstruction , such as i-RIM ( Putzky & Welling , 2019 ) , Adaptive-CS-Net ( Pezzotti et al. , 2019 ) , Pyramid Convolutional RNN ( Wang et al. , 2019 ) and E2E VarNet ( Sriram et al. , 2020 ) , which is at the top of the fastMRI Challenge Leaderboard at the time of the writing of this paper . Another line of work , inspired by the deep image prior ( Ulyanov et al. , 2018 ) focuses on using the inductive bias of convolutional networks to perform reconstruction without any training data ( Jin et al. , 2019 ; Darestani & Heckel , 2020 ; Heckel & Soltanolkotabi , 2020 ; Heckel & Hand , 2019 ; Van Veen et al. , 2018 ) . Those methods do perform significantly better than classical un-trained networks , but do not perform as well as neural networks trained on large sets of training data .

Summary: Review: This paper proposes data augmentation methods for medical imaging(especially for accelerated MRI) based on the MR physics. The augmentation includes both pixel preserving augmentations/general affine augmentations on both real and imaginary values in the image domain. Then, the augmented images are transformed to k-space domain and the k-space data are down-sampled for the input data generation for the accelerated MRI task. They claim that how to schedule p(the probability of applying combinations of augmentation) over the training is important and the schedules from p=0 and increasing over epochs shows best results, experimentally.