Data augmentation for deep learning based accelerated MRI reconstruction
Authors: Zalan Fabian
Reinhard Heckel
Mahdi Soltanolkotabi
Venue:  (2021)
Keywords: fast MRI
deep learning for inverse problems
data augmentation
unrolled networks
learning with limited data

Abstract: Deep neural networks have emerged as very successful tools for image restoration and reconstruction tasks. These networks are often trained end-to-end to directly reconstruct an image from a noisy or corrupted measurement of that image. To achieve state-of-the-art performance, training on large and diverse sets of images is considered critical. However, it is often difficult and/or expensive to collect large amounts of training images. Inspired by the success of Data Augmentation (DA) for classification problems, in this paper, we propose a pipeline for data augmentation for image reconstruction tasks arising in medical imaging and explore its effectiveness at reducing the required training data in a variety of settings. We focus on accelerated magnetic resonance imaging, where the goal is to reconstruct an image from a few under-sampled linear measurements. Our DA pipeline is specifically designed to utilize the invariances present in medical imaging measurements as naive DA strategies that neglect the physics of the problem fail. We demonstrate the effectiveness of our data augmentation pipeline by showing that for some problem regimes, DA can achieve comparable performance to the state of the art on the FastMRI dataset while using significantly fewer training data. Specifically, for 8-fold acceleration we achieve performance comparable to the state of the art with only $10\%$ of the training data for multi-coil reconstruction and with only $33\%$ of the training data for single-coil reconstruction. Our findings show that in the low-data regime DA is beneficial, whereas in the high-data regime it has diminishing returns.

Under review a conference paper at ICLR 2021
DATA AUGMENTATION FOR DEEP LEARNING BASED AC- CELERATED MRI RECONSTRUCTION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks have emerged very successful tools for image restoration and reconstruction tasks. These networks are often trained end-to-end to directly reconstruct an image from noisy or corrupted measurement of that image. To achieve state-of-the-are performance, training on large and diverse sets of images is considered critical. However, it is often difficult and/or expensive to collect large amounts of training images. Inspired by the success of Data Augmentation (DA) for classification problems, in this paper, we propose pipeline for data augmentation for image reconstruction tasks arising in medical imaging and explore its effective- ness at reducing the required training data in variety of settings. We focus on accelerated magnetic resonance imaging, where the goal is to reconstruct an image from a few under-sampled linear measurements. Our DA pipeline is specifically designed to utilize the invariances present in medical imaging measurements as naive DA strategies that neglect the physics of the problem fail. We demonstrate the effectiveness of our data augmentation pipeline by showing that for some problem regimes, DA can achieve comparable performance to the state of the art on the fastMRI dataset while using significantly fewer training data. Specifically, for 8-fold acceleration we achieve performance comparable to the state of the art with only 10% of the training data for multi-coil reconstruction and with only 33% of the training data for single-coil reconstruction. Our findings show that in the low-data regime DA is beneficial, whereas in the high-data regime it has diminishing returns.
INTRODUCTION
In magnetic resonance imaging (MRT), an extremely popular medical imaging technique, it is common to reduce the acquisition time by subsampling the measurements, because this reduces cost and increases accessibility of MRI to patients. However, since due to the subsampling, there fewer equations than unknowns, the signal is not uniquely identifiable from the measurements. To overcome this challenge there has been a flurry of activity over the last decade aimed at utilizing prior knowledge about the signal, in research area referred to as compressed sensing (Candes et al., 2006; Donoho, 2006).
Classical compressed sensing methods reduce the required number of measurements by utilizing prior knowledge about the images during the reconstruction process, often via a convex regularization that enforces sparsity in an appropriate transformation of the image. More recently, deep learning techniques have been used to enforce much more nuanced forms of prior knowledge (see Ongie et al. (2020) and references therein for an overview). The most successful of these approaches aim to directly learn the inverse mapping from the measurements large set of training data consisting of signal/measurement pairs. This approach often enables faster reconstruction of images, but more importantly, deep learning techniques yield significantly higher quality reconstructions. This in turn enables fewer measurements further reducing image acquisition times. For instance, in an accelcrated MRI competition known as FastMRI Challenge (Zbontar et al., 2018), all the top contenders used deep learning reconstruction techniques.
the image by training on
Contrary to classical compressive sensing approaches, however, deep learning techniques typically rely on large sets of training data often consisting of images along with the corresponding measure- ment. This is also true about the use of deep learning techniques in other areas such as computer vision and Natural Language Processing (NLP) were superb empirical success has been observed.

---

Under review as conference paper at ICLR 2021
While large datasets have been harvested and carefully curated by tech companies in areas such as vision and NL.,, this is not feasible in many scientific applications including MRI. It is difficult and expensive to collect the necessary datasets for a variety of reasons, including patient confidentiality requirements, cost and time of data acquisition, lack of medical data compatibility standards, and the rarity of certain diseases.
A common strategy to reduce reliance on training data in classification tasks is data augmentation. In a classification setting data augmentation consists of adding additional synthetic data obtained by performing invariant alterations to the data (e.g. flips, translations. rotations) which do not affect the labels. Such data augmentation techniques are commonly used in classification tasks to significantly increase the performance standard benchmarks such as ImageNet and CIFAR-10. More specific to medical imaging, data augmentation techniques have been successfully applied to registration, classification and segmentation of medical images. More recently, several studies (Zhao al., 2020b; Karras et al., 2020; Zhao et al., 2020a) have demonstrated that data augmentation can significantly reduce the data needed for GAN training for high quality image generation. In regression tasks such as image reconstruction, however, data augmentation techniques are less common and much more difficult to design in part due to the lack of the aforementioned invariances (e.g., measurements of a rotated image are not the same as measurements from the original image).
The goal of this paper is to explore the benefits of data augmentation techniques for accelerated MRI with limited training data. By carefully taking into account the physics behind the MRI acquisition process we design data augmentation pipeline, which we call MRAugment, that can successfully reduce the amount of training data required. Specifically, our contributions are follows:
We propose data augmentation technique tailored the physics of the MR reconstruction problem. We note that it is not obvious how to perform data augmentation in the context of inverse problems, because by changing image to enlarge the training set, do not automatically get corresponding measurement, contrary to classification problems, where the label is retained. In fact, we demonstrate that naive forms of data augmentation that do not properly take into account the underlying physics do not work.
We demonstrate the effectiveness of MRAugment on a benchmark accelerated MRI data set, specifically on the fastMRI (Zbontar et al., 2018) dataset. For 8-fold acceleration and multi-coil measurements (multi-coil measurements are the standard acquisition mode for clinical practice) we can achieve performance comparable to the state of the with only 10% of the training data. Similarly, again for 8-fold acceleration and single-coil experiments (an acquisition mode popular for experimentation) ARAuegent can achieve the performance of reconstructions methods trained on the entire data set while using only 33% of the training data.
We perform experiments showing that in low-data regime, where we have only limited training data MRAugment is beneficial, whereas in the high-data regime has diminishing returns. In particular we demonstrate that in low data regime (ss 1% of training data), data augmentation very significantly boosts reconstruction performance and captures diagnostically significant detail that is missed without data augmentation. In moderate data regime (s 10 33% of training data), MRAugment still achieves significant improvement reconstruction performance and may help to avoid hallucinations caused by overfitting without the use of DA.
Finally, while we focus on the top performing neural networks, MRAugment can seamlessly integrate with any deep learning model and therefore it can be useful a variety MRI problems.
2 BACKGROUND AND PROBLEM FORMULATION
In this section, we provide a brief background on accelerated MRI and formulate the problem.
MRI is medical imaging technique that exploits strong magnetic fields to form images of the anatomy. MRI is a prominent imaging modality in diagnostic medicine and biomedical research because it does not expose patients to ionizing radiation, contrary to competing technologies such as computed and positron emission tomography (CT and PET).
However, performing MR scan is time intensive, which is problematic for the following reasons. First, patients are exposed to long acquisition times in a confined space with high noise levels. Second, long acquisition times induce reconstruction artifacts caused through patient movement,
2

---

Under review conference paper at ICLR 2021
which sometimes requires patient sedation in particular in pediatric MRI (Vasanawala et al., 2010). Reducing the acquisition time can therefore increase both the accuracy of diagnosis and patient comfort. Furthermore, decreasing the acquisition time needed allows more patients receive sean using the same machine. This can significantly reduce patient cost, since each MRI machine comes with high cost to maintain and operate.
Since the invention of MR in the 1980s there has been tremendous research focusing reducing their acquisition time. The two main ideas are to i) perform multiple acquisitions simultaneously (Sodickson & Manning, 1997; Pruessmann et al., 1999; Griswold et al., 2002) and to ii) subsample the measurements, known as accelerated acquisition or compressive sensing (Lustig et al., 2008). Most modern scanners combine both techniques, and therefore we consider such setup.
2.1 ACCELERATED MRI ACQUISITION
In magnetic resonance imaging, measurements of patient's anatomy are acquired in the Fourier- domain, also called k-space, through receiver coils. In the single-coil acquisition mode, the k-space measurement k e C of complex-value ground truth image x: e C is given by
Fx"
where FF is the two-dimensional Fourier-transee transform, and 2 e C denotes additive noise arising in the measurement process. parallel MR imaging, multiple receiver coils are used, each of which captures different region of the image, represented by complex-valuee sensitivity map S. In this multi-coil setup, coils acquire k-space measurements modulated by their corresponding sensitivity maps:
k FS,x" Zi, i= i.
where N is the number of coils. Obtaining fully-sampled k-space data is time-consuming, and therefore accelerated MRI we decrease the number of measurements by undersampling in the Fourier-domain. This undersampling can be represented by binary mask M that sets all frequency components not sampled to zero:
Mhi i= I,.., N.
We write the overall forward map concisely as
k A=
where A (-) is the linear forward operator and & denotes the undersampled coil measurements stacked into a single column vector. The goal in accelerated MRI reconstruction is to recover the image xx from the set of k-space measurements k. Note that making assumptions on the image -- is in general impossible to perfectly recover the image, because we have fewer measurements than variables to recover. This recovery problem is known as compressive sensing. To make image recovery potentially possible, recovery methods make structural assumptions about x",, such that sparse in some basis or implicitly that it looks similar to images from the set of training images.
2.2 TRADITIONAL ACCELERATED MRI RECONSTRUCTION METHODS
Traditional compressed sensing recovery methods for accelerated MRI are based on assuming that the image x² is sparse in some dictionary, for example the wavelet transform. Recovery is then posed as a typically convex optimization problem:
arg min A k 2 R((x),
where R(-) is regularizer enforcing sparsity in certain domain. Typical functions used in CS based MRT reconstruction are waaaeeet and total-variation regularizers. These optimization problems can be numerically solved via iterative gradient descent based methods.
2.3 DEEP LEARNING BASED MRI RECONSTRUCTION METHODS
In recent years, several deep learning algorithms have been proposed and convolutional neural networks established new state-of-the-art in MRI reconstruction significantly surpassing the classical

---

Under review as a conference paper at ICLR 2021
baselines. Encoder-decoder networks such as the U-Net (Ronneberger et al., 2015) and its variants were successfully used in various medical image reconstruction (Hyun et al., 201 8; Han & Ye, 2018) and segmentation problems (Çiçek et al., 2016; Zhou et al., 2018). These models consist of two sub-networks: the encoder repeatedly filters and downsamples the input image with learned convolutional filters resulting in a concise feature vector. This low-dimensional representation is then fed to the decoder consisting of subsequent upsampling and learned filtering operations. Another approach that can be considered a generalization of iterative compressed sensing reconstructions consists of unrolling gradient descent iterations and mapping them to cascade of sub-networks (Zhang & Ghanem, 8). Several variations of this unrolled method have been proposed recently for MR reconstruction, such i-RIM (Putzky & Welling, 2019), Adaptive-CS-Net (Pezzotti et al., 2019), Pyramid Convolutional RNN (Wang et al., 2019) and E2E VarNet (Sriram et al., 2020), which the top of the fastMRI Challenge Leaderboard at the time of the writing of this paper.
Another line of work, inspired by the deep image prior (Ulyanov et al., 2018) focuses on using the inductive bias of convolutional networks to perform reconstruction without any training data (Jin et al., 2019; Darestani & Heckel, 2020; Heckel & Soltanolkotabi, 2020; Heckel & Hand, 2019; Van Veen et al., 2018). Those methods do perform significantly better than classical un-trained networks, but do not perform as well as neural networks trained on large sets of training data.
3 MRAUGMENT: DATA AUGMENTATION PIPELINE FOR MRI
In this section we propose our data augmentation technique, MR ARAumeen,, for MRI reconstruction. We emphasize that data augmentation in this setup and for inverse problems in general is substantially different from DA for classification problems. For classification tasks, the label of the augmented image trivially the same as that of the original image, whereas for inverse problems we have to generate both an augmented target image and the corresponding measurements. This is non-trivial as it is critical to match the noise statistics of the augmented measurements with those in the dataset.
We are given training data in the form of fully-sampled MRI measurements in the Fourier domain, and our goal is to generated new training examples consisting of a subsampled k-space measurement along with target image. MRAugment is model-agnostic in that the generated augmented training example can be used with any machine learning model and therefore can be seamlessly integrated with existing reconstruction algorithms for accelerated MRI, and potentially beyond MRI.
Our data augmentation pipeline, illustrated in Figure 1, generates a new example consisting of a subsampled k-space measurement k, along with a target image T. as follows. We are given training examples as fully-sampled k-space slices, which k = col( k1, k,,.., ..., kN) for notational convenience. From these, we obtain the individual coil images by applying the inverse Fourier transform as x F- k. We generate augmented individual coil images with an augmentation function D, specified later, as x. D(x). From the augmented images, we generate an undersampled measurement by applying the forward model as k. A (æ.). Both x and æ, are complex-valued: even though the MR scanner obtains measurements of a real-valued object, due noise the inverse the augmentation function has to generate complex-valuse images, which adds an extra layer of difficulty compared to traditional data augmentation techniques pertaining real-valued images (see Section 3.1 for further details). Finally, the real-valued ground truth image is obtained by combining the coil images Lo by pixel-wise -ut-suu--qqurrss (RSS) followed by center-cropping C:
stack into single column vector
of the measurement is complex- valued. Therefore
(V
F. = C (RSS(xx)) C
Σ a 2
In the following two sections we first argue why we generate individual coil images with the data augmentation function, and then discuss the design of the data augmentation function D itself.
3.1 DATA AUGMENTATION NEEDS TO PRESERVE NOISE STATISTICS
As mentioned before, augmenting complex-valued noisy images. This noise enters in the measurement process when we obtain the sally-sampled measurement of an image x" as ls = Fx + z. and is well approximated by i.i.d complex Gaussian noise, independent in the real and imaginary
4

---

Under review conference paper at ICLR 2021
complex real
F
M
Ko Fra
ka- MRk
k
Da
RSS (ata)
Zo
D
RSS
pixel preserving augmentations
interpolating augmentations
split D4/3
combine 94/3
Oppe
fxx
sugmentation scheduler
Figure Flowchart of MRAugment, our data augmentation pipeline for MRI.
parts of each pixel (Nishimura, 1996). Therefore, we write xx" z' where z' has the same distribution as z due to F being unitary. Since the noise distribution characteristic to the instrumental setup (in this case the MR scanner and the acquisition protocol), assuming that the training and test images are produced by the same setup, it is important that the augmentation function preserves the noise distribution of training images as much as possible. Indeed, large mismatch between training and test noise distribution leads to poor generalization (Knoll et al., 2019).
Let us demonstrate why it is non-trivial to generate augmented measurements for MRI through simple example. A natural but perhaps naive approach for data augmentation is to augment the real-valued target image instead of the complex valued æ:. This would allow us to directly obtain real augmented images from real target image just as in typical data augmentation. However, this approach leads to different noise distribution in the measurements compared to the test data due to the non-linear mapping from individual coil images to the real-valued target and works poorly as demonstrated in Section
In contrast, if we augment the individual coil images directly with a linear function D, which is main focus here, we obtain the augmented k-space data
k FDr FD(x" + = FDD + FD2',
where FDx" represents the augmented signal and the noise FDz' is still additive complex Gaussian. key observation is that in case of transformations such as translation, horizontal and vertical flipping and rotation the noise distribution exactly preserved. Moreover, for general linear transformations the noise is still Gaussian in the real and imaginary parts of each pixel. This example motivates our choice augment complex-vales -aauud images directly derived from the original k-space measurements and ii) consider simple transformations which preserve the noise distribution. Next we overview the types augmentations we propose in line with these key observations.
3.2 TRANSFORMATTONS USED FOR DATA AUGMENTATION
We apply the following two types of image transformation D in our data augmentation pipeline: Pixel preserving augmentations, that do not require any form interpolation and simply result in vertical and horizontal flipping,
permutation of pixels over the image. Such transformations translation by integer number of pixels and rotation by multiples 90". As we pointed out in Section 3.1, these transformations do not affect the noise distribution the measurements and therefore
5

---

Under review conference paper at ICLR 2021
suitable for problems where training and test data are expected to have similar noise characteristics. General affine augmentations, that can be represented by an affine transformation matrix and in general require resampling the transformed image at the output pixel locations. Augmentations in this group are: translation by arbitrary (not necessarily integer) coordinates, arbitrary rotations, scaling and shearing. Scaling can be applied along any of the two spatial dimensions. We differentiate between isotropic scaling, in which the same scaling factor is applied in both directions (s 1: zoom-in, s < 1: zoom-out) and anisotropic scaling in which different scaling factors (Sx, Sy) are applied along different axes.
Figure provides a visual overview of the types of augmentations applied in this paper. Numerous other forms of transformation may be used in this framework such exposure and contrast adjustment, image filtering (blur, sharpening) or image corruption (cutout, additive noise). However, in addition the noise considerations mentioned before that have be taken into account, some of these transformations are difficult to define for complex-valued images and may have subtle effects on image statistics. For instance, brightness adjustment could be applied the magnitude image, the real part only or both real and imaginary parts, with drastically different effects on the magnitude phase relationship of the image. That said, we hope to incorporate additional augmentations in our pipeline in the future after thorough study of how they affect the noise distribution.
Original
V-flip
H-flip
Rot. k90
Rotation
Transl.
Zoom-in Zoom-out Aniso sc. Shcaring
Figure 2: Transformations used in MRAugment applied to ground truth slice.
3.3 SCHEDULING AND APPLICATION OF DATA AUGMENTATIONS
With the different components in place we are now ready to discuss the scheduling and application of the augmentations, depicted in the bottom half of Figure Recall that MRAugment generates target image 2, and corresponding inder-sampled k-space measurement k2 from full k-space measurement. Which augmentation is applied and how frequently is determined by parameter p, the common parameter determining the probability of applying a transformation to the ground truth image during training, and the weights W (W1, W2, ..., wx) pertaining to the K different augmentations, controlling the weights of transformations relative to each other. We apply a given transformation 4 with probability p; = p- wy. The augmentation function is applied to the coil images, specifically the same transformation applied with the same parameters to the real and imaginary parts (R{(x1), (xx))) (.2((2) ((x))))))()))) (/z=)..... ..... Ri x. }. H(NN) }) of coil images. If transformation t, is sampled (recall that we select them with probabilities p:). we randomly select the parameters of the transformation from pre-defined range (for example, rotation angle in [0, 180%) To avoid aliasing artifacts, before transformations that require interpolation we first upsample the image and apply the transformations the upsampled image. Then the result is downsampled to the original size.
A critical question is how to schedule over training in order to obtain the best model. Intuitively, in initial stages of training no augmentation is needed, since the model can learn from the available (possibly limited) original training data. As training progresses the network learns to fit the original data points and their utility decreases over time. We found schedules starting from p and increasing over epochs to work best. The ideal rate of increase depends on both the model size and amount of available training data. We experimented with linear (ramp) and exponential scheduling and observed that linear functions work best when we have large amount of training data compared to the model size. contrast, steeper exponential scheduling helps in the low-data regime.
4 EXPERIMENTS
In this section explore the effectiveness of MRAugment in the context of accelerated MRI reconstruction in various regimes of available training data sizes on the fastMRI dataset. We present detailed description of this dataset in Appendix A.

---

Under review conference paper at ICLR 2021
Target
100%
1%+DA
1%
Target
100%
1%+DA
1%
Figure 3: Visual comparison of single-coil (top row) and multi-coil (bottom-row) reconstructions using varying amounts of training data with and without data augmentation. We achieve reconstruction quality comparable the state of the art but using I % of the training data. Without DA fine details are completely lost.
Experimental setup. We use the state-of-the-att End-to- tndEnd VarNet model (Sriram al., 2020), because it is as of now the best performing neural network on the fastMRI dataset. We measure performance terms of the structural similarity index measure (SSIM), which is the standard evaluation metric on the fastMRI dataset. We study the performance of MRAugment as a function of the size the training set. We construct different subsampled training sets by randomly sampling volumes of the original training dataset and adding all slices of the sampled volumes to the new subsampled dataset. The original training set consists of approximately 35k MRI slices in 973 volumes and subsample 1%, 10%, 33% and 100% of the original size. We measure performance on the original validation set separate from the training set. For experiments, we apply random masks by undersampling whole kspace lines in the phase encoding direction and including 4% of lowest frequency adjacent kspace lines in order to be consistent with baselines in (Zbontar et al., 2018). For both the bascline experiments and for MRAugment, we generate a new random mask for each slice on-the-fly while training by uniformly sampling kspace lines, but use the same fixed mask for each slice within the same volume on the validation set (different across volumes). This technique is standard for models trained on this dataset and not part of our data augmentation pipeline. For augmentation probability scheduling we use
p(t) = Pmaz I- (1-e -te/T)
(4.1)
where t is the current epoch, T denotes the total number of epochs, 5 and Pmazz 0.6 for 1% training data experiments and Pmaz = 0.55 in other data augmentation experiments. More experimental details can be found in Appendix B and the source code is attached in the supplementary.
Single-coil experiments. For single-coil acquisition we are able to exactly match the performance of the model trained on the full dataset using only third of the training data as depicted on the left in Fig. Moreover, with only 10% of the training data we achieve 99.5% of the SSIM trained on the full dataset. The visual difference between reconstructions with and without data augmentation becomes striking in the low-data regime. As seen in the top row of Fig. 3, the model without DA was unable to reconstruct any of the fine details and the results appear blurry with strong artifacts. Applying MRAugment significantly improves reconstruction quality both a quantitative and qualitative sense, visually approaching that obtained from training on the full dataset but using hundred times less data.
Multi-coil experiments. As depicted on the right in Fig. for multi-coil acquisition we closely match the state of the art while significantly reducing training data. More specifically, approach
7

---

Under review as conference paper at ICLR 2021
0.9
Target
10%+DA
10%
6.0
0.00
700
0.00
685
2.25
Figure 4: Single-coil (left) and multi-coil (right) Figure 5: Hallucinated features appear on recon- validation SSIM vs. # of training images.
structions without data augmentation.
the state of the SSIM within 0.6% using 10% of the training data and within 0.25% with 33% of training data. As seen in the bottom row of Fig. 3, when using only 1% of the training data we successfully reconstruct fine details comparable that obtained from training on the full dataset, while high frequencies are completely lost without DA. More visual comparisons for both single-coil and multi-coil reconstruction can be found in Appendix D.
Hallucinations. An unfortunate side-effect of deep learning based reconstructions may be the presence of hallucinated details. This is especially problematic in providing accurate medical diagnosis and lessens the trust of medical practicioners in deep learning. We observed that data augmentation has the potential benefit of eliminating hallucinations by preventing overfitting to training data, seen in Fig. 5.
Our empirical observations on data augmentation can be summarized as follows:
In the low data regime (sx 1% of training data), data augmentation very significantly boosts reconstruction performance. The improvement is large both in terms of raw SSIM and visual reconstruction quality. Using MRAugment, fine details are recovered that are completely missing from reconstructions without DA. This suggests that DA improves the value reconstructions for medical diagnosis, since health experts typically look for small features of the anatomy.
In the moderate data regime (ss 10 - 33% of training data), MRAugment still achieves significant improvement reconstruction SSIM. We want emphasize the significance of seemingly small differences in SSIM close to the state of the art and invite the reader to visit the FastMRI Challenge Leaderboard that demonstrates how close the best performing models are. Moreover, MR MRAugeent achieves visual improvement in fine details that may have diagnostic value. Fig. 5 shows that in some cases hallucinated details can avoided in this regime using data augmennattin.. Additional visual comparison of recovered details is provided in Appendix D).
Data augmentation applied to the full training dataset has diminishing returns. It does not improve performance if applied to the full dataset, but it does not degrade it either. This suggests that it may be possible to further improve upon the state of the art when using the entire dataset by combining data augmentation with more advanced transformations and larger models. We plan pursue this exciting direction in our future work.
Ablation studies. We performed ablation studies 1% training data in order to better understand which augmentations are useful. We use the multi-coil experiment with all augmentations as baseline and tune augmentation probability for other experiments such that the probability that certain slice is augmented by at least one augmentation is the same across all experiments. We depict results on the validation dataset in Table 1. Both pixel preserving and general (interpolating) affine transformations are useful and can significantly increase reconstruction quality. Furthermore, we observe that their effect complementary: they are helpful separately, but we achieve peak reconstruction SSIM when all applied together. Finally, the utility of pixel preserving augmentations seems to be lower than that of general affine augmentations, however they come with a negligible additional computational cost.
Furthermore, we investigate the effect of varying the augmentation probability scheduling function. The results on the validation dataset are depicted in Table 2, where exponential, & denotes the exponential scheduling function in (4.1), with Pmaz and constant, means we use a fixed augmentation probability & throughout training. We observe that scheduling starting from low augmentation probability and gradually increasing is better than constant probability, as initially the network does not benefit much from data augmentation as it can still learn from the original samples. Furthermore, too low too high augmentation probability both degrade performance. If the
8

---

Under review conference paper at ICLR 2021
Aug. scheduling none exponential, 0.3
Val. SSIM
0.8396
Augmentations
Val. SSIM
0.8565
none
0.8396
constant, 0.3
0.8588
pixel preserving only
0,8585
exponential, 0.6
0.8758
interpolating only all augmentations
0.8731
constant, 0.6
0.8611
0.8758
exponential, 0.8
0.8600
Table I: Comparison of peak validation SSIM applying various of augmentations on 1% of training data, multi-coil acquisition.
Table 2:
Comparison of peak validation SSIM using different augmentation probability sched- ules on 1% of training data, multi-coil acquisi- tion.
Target
10%+DA
10%+naive DA
0.05
883
0.00
18G-+aa00
0.55
100
(a) Naive data augmentation that does not preserve mcasurement noise distri- bution Icads significantly degraded reconstruction quality.
(b) Naive data augmentation degrades generalization per- formance.
Figure 6: Experimental results comparing MRAugment with naive data augmentation.
augmentation probability is too low, the network may overfit to training data more regularization is needed. On the other hand, too much data augmentation hinders reconstruction performance as the network rarely sees images close to the original training distribution.
Noise considerations. Finally, we would like to emphasize the importance applying DA in a way that takes into account the measurement noise distribution. When applied incorrectly, DA leads to significantly worse performance than not using any augmentation. We train a model using DA without considering the measurement noise distribution described in Section 3.1 by augmenting real-valued target images. We use the same exponential augmentation probability scheduling for MRAugment and the naive approach. As Fig. 6b demonstrates, reconstruction quality degrades training using the naive approach. This is due to the fact that as augmentation probability increases, the network is being fed less and less original images, whereas the poorly augmented images detrimental for generalization due to the mismatch in train and validation noise distribution. On the other hand, MRAugment clearly helps and validation performance steadily improves over epochs. Fig. 6a provides a visual comparison of reconstructions using naive DA and our data augmentation method tailored to the problem. Naive DA reconstruction exhibits high-frequency artifacts and low image quality caused by the mismatch in noise distribution. These drastic differences underline the vital importance of taking a careful, physics-imfee informed approach to data augmentation for MR reconstruction.
5 CONCLUSION
In this paper, we develop physics-based data augmentation pipeline for accelerated MR imaging. find that DA yields significant gains in a low-data regime which can be beneficial in applications where only little training data is available where the training data changes quickly. We also demonstrate that even with a moderate amount of training data, DA enables reconstructions that contain diagnostically significant detail and may help avoiding hallucinations caused overfitting without DA.

---

Under review conference paper at ICLR 2021
REFERENCES
Emmanuel J. Candes, Justin K. Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 59(8):1207- 220-11233 2006.
Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3D U- Net: learning dense volumetric segmentation from sparse annotation. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 424 432.. Springer, 2016.
Mohammad Zalbagi Darestani and Reinhard Heckel. Can Un-trained Neural Networks Compete with Trained Neural Networks at Image Reconstruction? arXiv:2007.02-77 [cs, eess, stat], 2020.
David L. Donoho. Compressed sensing. IEEE Transactions information theory, 52(4):1289-1336, 2006.
Mark Griswold, Peter Jakob, Robin M Heidemann, Mathias Nittka, Vladimir Jellus, Jianmin Wang, Berthold Kiefer, and Axel Haase. Generalized autocalibrating partially parallel acquisitions (GRAPPA). Magnetic Resonance Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 1202-11202-2100 2002.
Yoseob Han and Jong Chul Ye. Framing U-Net via deep convolutional framelets: Application to sparse-view CT. IEEE Transactions on Medical Imaging, 37(6):1418-1 1429, 2018.
Reinhard Heckel and Paul Hand. Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks. In International Conference on Learning Representations, 2019.
Reinhard Heckel and Mahdi Soltanolkotabi. Compressive sensing with un- un-trained neural networks: Gradient descent finds the smoothest approximation. In International Conference Machine Learning, 2020.
Chang Min Hyun, Hwa Pyung Kim, Sung Min Lee, Sungchul Lee, and Jin Keun Seo. Deep learning for undersampled MRI reconstruction. Physics in Medicine & Biology, 63(13):135007, 2018.
Kyong Hwan Jin, Harshit Gupta, Jerome Yerly, Matthias Stuber, and Michael Unser. Time-Dependent Deep Image Prior for Dynamic MRI. arXiv:1910.01684 [cs, eess], 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006. 066766 2020.
Florian Knoll, Kerstin Hammernik, Erich Kobler, Thomas Pock, Michael P Recht, and Daniel K Sodickson. Assessment of the generalization of learned image reconstruction and the potential for transfer learning. Magnetic Resonance in Medicine, 31(1):116-13) 2019.
Michael Lustig, David L. Donoho, Juan M. Santos, and John M. Pauly. Compressed sensing MRI. IEEE signal processing magazine, 25(2):72-82, 2008
Dwight George Nishimura. Principles of magnetic resonance imaging. Stanford University, 1996.
Gregory Ongie, Ajil Jalal, Christopher A. Metzler Richard G. Baraniuk, Alexandros G. Dimakis, and Rebecca Willett. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 2020.
Nicola Pezzotti, Elwin de Weerdt, Sahar Yousefi, Mohamed S Elmahdy, Jeroen van Gemert, Christophe Schülke, Mariya Doneva, Tim Nielsen, Sergey Kastryulin, Boudewijn PF Lelieveldt, et al. Adaptive-CS-Nel -etN fastmri with adaptive intelligence. arXiv preprint 1912. 2019.
Klaas P Pruessmann, Markus Weiger, Markus B Scheidegger, and Peter Boesiger. SENSE: sensitivity encoding for fast MRI. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance Medicine, 42(5):952-962. 1999.
10

---

Under review conference paper at ICLR 2021
Patrick Putzky and Max Welling, Invert learn invert. In Advances in Neural Information Processing Systems, pp. 446-456 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 234-241. Springer, 2015.
Daniel K Sodickson and Warren J Manning. Simultaneous acquisition of spatial harmonics (SMASH). fast imaging with radiofrequency coil arrays. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 38(4)5991-603, 1997.
Anuroop Sriram, Jure Zbontar, Tullie Murrell, Aaron Defazio, C Lawrrecc Zitnick, Nafissa Yakubova, Florian Knoll, and Patricia Johnson. End-to-end variational networks for accelerated MRI recon- struction. arXiv preprint arXiv:2004.06666 2020.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep Image Prior. In Conference on Computer Vision and Pattern Recognition, pp. 9446 9454, 2018
Dave Van Veen, Ajil Jalal, Mahdi Soltanolkotabi, Eric Price, Sriram Vishwanath, and Alexandros G Dimakis. Compressed sensing with deep image prior and learned regularization. arXiv preprint (cMiv:8806.06444 2018
Shreyas Vasanawala, Marcus T Alley, Brian Hargreaves, Richard Barth, John Pauly, and Michael Lustig. Improved pediatric MR imaging with compressed sensing. Radiology, 256(2): 607-616, 2010.
Puyang Wang, Eric Z Chen, Terrence Chen, Vishal M Patel, and Shanhui Sun. Pyramid convolutional RNN for MRI reconstruction. arXiv preprint 191..0:112.000433, 2019.
Jure Zbontar, Florian Knoll, Anuroop Sriram, Matthew J Muckley, Mary Bruno, Aaron Defazio, Mare Parente, Krzysztof J Geras, Joe Katsnelson, Hersh Chandarana, al. fastMRI: An open dataset and benchmarks for accelerated MRL. arXiv preprint 18110888 2018.
Jian Zhang and Bernard Ghanem. -STA-Net: Interpretable optimizatioon-nsppree deep network for image compressive sensing. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 1828-1837, 2018
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun- Yan Zhu, and Song Han. Differentiable Augmentation for Data-Efficient GAN Training. arXiv:2006. 10738, 2020a.
Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image Augmentations for GAN Training. XXi:22006.02595, 2020b.
Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: A nested u-net architecture for medical image segmentation. Deep Learning Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 3-11. Springer, 2018.
11

---

Under review as a conference paper at ICLR 2021
A
THE FASTMRI DATASET
The fastMRI dataset (Zbontar al., 2018) is large open dataset of knee and brain MRI volumes. The train and validation splits contain fully-sampled k-space volumes and corresponding target reconstructions for both (simulated) single-coil and multi-coil acquisition. The knee MRT dataset we focusing in this paper includes 973 train volumes (34742 slices) and 199 validation volumes (7135 slices). The target reconstructions are fixed size 320 320 center cropped images corresponding to the fully-sampled data of varying sizes. The under-sampling ratio is either 25%( 4x acceleration) or 12.5% (8x acceleration). k-space, that is columns in k-space sampled. A certain neighborhood of adjacent low-frequee oowfrequencyy lines are always included in the measurement. The size of this fully-sampled region is 8% of all frequencies in case of 4x acceleration and 4% in case of 8x acceleration.
sampling is performed along the phase encoding dimension in
B EXPERIMENTAL DETAILS
Here we provide details of the setup for our main experiments.
Datasets. We use the FastMRI (Zbontar et al., 2018) single-coil and multi-coil knee dataset for our experiments. For creating the sub-sampled datasets, we uniformly sample volumes from the training set, and add all slices from the sampled volumes. Our validation results are reported on the whole validation dataset. Images in the dataset have varying dimensions. Due to GPU memory considerations center-cropped the input images 640 x 368 pixels (which covers most of the images). We use random undersampling masks with 8x acceleration and 4% fully-sampled low- frequency band, undersampled in the phase encoding direction by masking whole kspace lines. We generate a new random mask for each slice on-the-fly while training, but use the same fixed mask for each slice within the same volume on the validation set (different across volumes).
Model. We train the default E2E-VarNe network from Sriram et al. (2020) with 12 cascades (approx. 30M parameters) for both the single-coil and multi-coil reconstruction problems. For single-coil data we remove the Sensitivity Map Estimation sub-network as sensitivity maps are not relevant in this problem.
Hyperparameters and training. We use an Adam optimizer with 0.0003 learning rate following Sriram et al. (2020). We train the baseline model on the full training dataset for 50 epochs. For the smaller, sub-sampled datasets we train for the same computational cost as the baseline, that is we train for N 50 epochs on 1/Nth of the training data. Without data augmentation, we observe a saturation in validation SSIM during this time. With data augmentation still observe improvement validation performance after the standard number of epochs. We report the best SSIM on the validation set throughout training. We train 4 GPUs for single-coil data and on 8 GPUs for multi-coil data. The batch size matches the number of GPUs used for training, since a GPU can only hold single datapoint,
trained 50% longer
Data augmentation parameters. The transformations and their corresponding probability weights and ranges of values are depicted in Table 3. We adjust the weights so that groups of transformations such as rotation (arbitrary, by & 90%), flipping (horizontal or vertical) or scaling (isotropic or anisotropic) have similar probabilities. For both the affine transformations and upsampling we use bicubic interpolation. Due to computational considerations we only use upsampling before transformations for the single-coil experiments.
C COMPARISON OF ADDITIONAL METRICS
In order to provide more in-depth comparison for our main experiment, here we provide results on PSNR as additional image quality metric. We observe significant and consistent improvement in PSNR when applying MRAugment (Fig. 7) with similar trends to SSIM: the improvement the most prominent in the low-data regime, but still significant in the moderate domain.
12

---

Under review conference paper at ICLR 2021
Transtormations
Range of values
W2
Horizontal flip
flipped/not flipped flipped/not flipped
0.5
Vertical flip Rotation by k 9090
0.5
k E (0, 1, 2
0.5
Rotation
[-180", 180
0.5
Translation
width: [-8%, 8%.. height: [12.5%% 12.5%]
1.0
Isotropic scaling
[0.75. 1.25]
0.5
Anisotropic scaling
[0.75. 1. 1.25] along each axes (-12.5",12.5")
0.5
Shearing
1.0
Table 3: Data augmentation configuration for all experiments.
31
30.5
Val. PSNR
Val. PSNR
30
35
29.5
34
- no DA
DA
29
DA
33
DA
28.5
0.35k
3.5k
11.5k
35k
0.35k
3.5k
11.5k
35k
of training examples
# of training examples
Figure 7: Single-coil (left) and multi-coil (right) validation PSNR vs. # of training images.
D MORE EXPERIMENTAL RESULTS
Additional reconstructions. In order to demonstrate that MRAugment works well across wide range of MR slices and the results depicted in Section are not cherry-picked, here we provide additional reconstructions with and without data augmentation. In multi-coil reconstructions the visual differences are more subtle, therefore we magnified regions with fine details for better comparison.
Complete version of Figure 3 extended with reconstructions with and without data augmentation for 10% and 33% training data are presented in Figures and 10 respectively.
Figures 11 and 12 provide with and without DA.
reconstructed slices randomly sampled from the validation dataset
Target
10%+DA
10%
Figure 8: MRAugment recovers additional fine details even in the moderate data regime.
Other models. Even though we demonstrated our DA pipeline on E2E-VarNet, the potential of our technique not limited to a specific model. We performed preliminary experiments on i-RIM Putzky & Welling (2019), another top performing model on single-coil MR reconstruction. We kept the hyperparameters proposed in Putzky Welling (2019) for the single-coil problem with modifications follows. Due computational considerations, decreased the number of invertible layers to
13

---

Under review conference paper at ICLR 2021
100%
10%
Target
33%+DA
10%+DA
1%+DA
Figure 9: Visual comparison of single-coil reconstructions depicted in Figure extended with additional images corresponding to various amount of training data.
100%
33%
10%
1%
Target
33%+DA
10%+DA
1%+DA
Figure 10: Visual comparison multi-coil reconstructions depicted Figure extended with additional images corresponding various amount of training data.
6 with [64, 128, 256, 256, 128, 64] hidden features inside the reversible blocks and [1, 4, 4, 2, kernel strides in each layer, resulting in model with 20M parameters. order to further reduce training time, we trained on volumes without fat suppression that take up 50% of the full fastMRI knee dataset. We refer to this new reduced dataset ass 'full' this section. Finally, we trained on 368x368 center crops of input images for each experiment. We used ramp scheduling with augmentation probability linearly increasing from 0 to Pmaz = 0.4. The acceleration factor and under-sampling mask the same in Section 4 As depicted in Fig. 13, experiments show that applying data augmentation to only 10% of the training data can match the performance of the model trained the full dataset.
14

---

Under review conference paper at ICLR 2021
Ground truth
100% training data
1% training data DA
1% training data
Figure 11: Visual comparison of single-coil reconstructions using varying amounts of training data with and without data augmentation.
0.78
SSIM
Vol.
10%
0.74
10RG+DA
100% buseline
0.72
1 poocss

---

Under review conference paper at ICLR 2021
Ground truth
100% training data
1% training data DA
1% training data
Figure 12:
Visual comparison of multi-coil reconstructions using varying amounts of training data with and without data augmentation.
Experimental results on the i-RIM network. We are able to achieve SSIM comparable to the 100% baseline with only 10% of the training data.
Figure 13:
E SENSITIVITY MAP PRESERVING AUGMENTATIONS
In the multi-coil case, our augmentation pipeline applics transformations to the underlying object modulated by the different coil sensitivity maps. In particular, the fully sampled measurement of the
16

---

Under review a conference paper at ICLR 2021
ith coil in the image domain takes the form
Tj Stx + 2/3
(E.1)
where 2 =
2/ is i.i.d Gaussian noise obtained via a unitary transform of the original measurement noise. Assuming linear augmentations, the augmented coil image from MRAugment be written as
La.i D(S + DD
Dzf,
(E.2)
where the additive noise is still Gaussian and the augmentation is applied to the coil sensitivity maps as well. However, the models we experimented with had no issues learning the proper mapping from transformed sensitivity maps our experimental results show.
is natural to ask data augmentation would be possible by directly augmenting the object. the coil sensitivities known estimated priori, may recover the object from the various coils as
N (5
N SS
x Σ
Σ Sj))
Σ
Sj
where ST is the complex conjugate of Sj and Σソا SJ I due to typical normalization (Sriram et al., 2020). Then, apply the augmentation as
N
N
Dx D( Σ51
Dx= + +D D
2055 Sj 5=1
Finally, we obtain the augmented coil images as
Tozi = SIr, = S,Dx + S Σ 5j4
(E.3)
Comparing this result to E.1, one may see that the additive noise has a very different distribution from the original noise distribution, even worse noise different augmented coil images are now correlated! Morcover, in this last step we modulated the additive noise with the sensitivity map, resulting in regions with low noise variance where the coil less sensitive and higher variance in more sensitive regions. This does not reflect the i.d noise distribution present on the original coil measurements. Finally, the sensitivity maps are typically not known and need to be estimated before we can apply this augmentation technique, which can introduce additional inaccuracies in the augmentation pipeline.
F ROBUSTNESS
0.89
0.88
Val. SSIM
0.87
0.86
no DA
DA
0.85
10
30
40
50
Epochs
Figure 14: Generalization performance of models trained on knee MRI data and evaluated on brain MRI data. Data augmentation improves validation performance unseen datasets.
17

---

Under review conference paper at ICLR 2021
We observed additional benefit of data augmentation, namely its potential improve the robustness of the trained model. First, we trained VarNet model on the complete fastMRI knee train dataset using the hyperparameters recommended in Sriram et (2020), and evaluated the network on the fastMRI brain validation dataset throughout training. We repeated the experiment with the same hyperparameters, but with MRAugment turned on (exponential scheduling, peak probability 0.4). The results can be seen in Fig. 14. The regularizing effect of data augmentation impedes the network to overfit to the training dataset, thus the resulting model more robust to shifts in test distribution. This results in higher reconstruction quality in terms of SSIM on unseen brain data when using MR AuAumeet..
18


Reviews:
[{"review_id": "zkY1023eHHO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "An augmentation method is proposed for MR image reconstruction, and is shown that significant results can be achieved with a small fraction of training data. \n\nComments \nOne of the major concern in image reconstruction problems is the availability of fully sampled data, while the dataset used in the problem provides this, how the proposed method will work in the absence of such data for generating the augmentations? \n\nFor MR reconstruction, the undersampling mask plays a significant role, while augmenting real images, what will be the effect on the mask. In particular, for 8x which masks are used in this work, are these similar to baseline or always used random sampling?\n\nFor results evaluation, SSIM is used, it is recommended to add results for either PSNR or NMSE for error quantification. \n\nHow are the samples for various data subsets selected? \n\nSince it is claimed that this can be extended to other MRI datasets, is it possible to add results for Brain MRI as well fo having diverse data and hence a better representation on the generalisation of the MRAugment model. For model training, any form of transfer learning is used or all models are trained form scratch? \n\nA comparison with state-of-the-art methods is missing and should be added. \n\nMinor comments\n\nThere are typos and grammar related mistakes which should be corrected. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ubdjsbBBq0u", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper gives a simple yet straightforward method to do data augmentation in MRI imaging. The paper is well written and the results quite promising on a rigorous test case.\n\nHowever, the methods proposed are in a sense rather obvious and the results do not give an improvement on the main practical test case. Throwing away data-post hoc sadly isn’t as strong of a motivation as showing an improvement on the full dataset.\n\nStill, the methods are novel, the presentation good and all steps are done rigorously so I can certainly see this being useful for a huge range of related applications. For this reason I’ll recommend acceptance.\n\nSome further comments include:\n\n* As noted by the authors, post-processing methods are an alternative to physics based approaches. An upside of them is that it’s trivial to do very aggressive data-augmentation (see e.g. Deep Bayesian Inversion which applied the whole menagerie). It could be interesting to have a standard U-Net baseline.\n* The “MRI is a Fourier transform” argument is well known to be a rather coarse approximation. It would be interesting to hear some debate on how a slight error in the forward model would impact this application.\n* Related to the above, MRI is a “trivial” inverse problem since the Fourier transform is unitary, basically a “rotation” in some high dimensional space. Comments on how to apply these methods to more advanced inverse problems like CT would be most welcome.\n* H and V flip looks wrongly labeled in Fig2.\n* How are the images e.g. rotated without padding artefacts?\n* In figure 12 the baseline should also show a training curve\n* How are the windows selected in e.g. figure 10? It feels like it’s slightly different between samples and also a bit too wide.\n* All figures that are not “images” should be in vector format. Notably figure 4.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZXeur66yH-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Example 2: \n \n\nReview: This paper proposes data augmentation methods for medical imaging(especially for accelerated MRI) based on the MR physics. The augmentation includes both pixel preserving augmentations/general affine augmentations on both real and imaginary values in the image domain. Then, the augmented images are transformed to k-space domain and the k-space data are down-sampled for the input data generation for the accelerated MRI task. They claim that how to schedule p(the probability of applying combinations of augmentation) over the training is important and the schedules from p=0 and increasing over epochs shows best results, experimentally.\n\n+ The importance of data augmentation is very important topic, especially in medical domain, because of their expensive cost for data gathering. It directly effects on the performance of deep learning algorithms. \n+ Overall, the paper is well written and might be easy to read by the readers who are not familiar with accelerated MRI.\n+ The results section is well structured. In particular, the results seem great and the proposed augmentation works well especially for the extreme cases such as using 1% of the training set.\n\nConcerns: \n- The key concern about the paper is the lack of experimentation to study the usefulness of the proposed method. \n- Considering the limited results, a deeper analysis of the proposed method would have been nice. The idea of augmentation in the MR image domain for each real and imaginary domain is a generic one, and many of MR researchers are already tried to augment the MR image on image domain and then after down-sampling for the accelerated MR task. I cannot clearly see the novelty of the proposed algorithm except for scheduling.\n-There are not enough ablation studies. For example, they are used variety of transformation for augmentation. The effects of each transformation should be different on the performance of the algorithm. Also, even the author divides the transforms into two categories: pixel preserving/general affine augmentation and the effects of two different augmentation can be helpful to the readers and researchers in the accelerated MRI deep learning community. However, it is missing.\n- In the 3rd section, the author tries to link the necessary of the augmentation pipeline for preserving noise statistics with their algorithm. However, the analysis and comparison is not well-provided. I think the ‘naïve DA’ which they provided (just augmentation on real valued image) is not a fair comparison experiment since it is the half of the data (real vs complexed value). The results of the other augmentation algorithm and/or the more analysis based on their theory are missing.\n- Also, the author mentioned about the physics of the data. However, all the augmentations they used(horizontal/vertical flips, rotations, translations, zoom-in/out, scaling, shearing) are the transformations which affects to the coil sensitivity map. Thus, the augmented train sets have the samples which have different coil sensitivity maps compared to the one which test sets have. I think they need to consider about these factors to claim about the physics based augmentation. (for example, 1. Calculate coil sensitivities of each coil, 2. Reconstruct the true magnetization X, 3. Apply augmentation on X and then apply the coil sensitivity map, 4. Get the k-space data and down-sampling.) \n- The results seem more sharp then the one which not use the proposed domain adaptation and the SSIM score is better too. However, it is important the opinions of the clinician in department of radiology. In some cases, the clinicians might see some special features on the MR images even it is blurred. Thus, if the deep learning algorithm might lost that features, they prefer the blurred one even the other is more sharp. The evaluations by the clinicians make the author’s claim stronger.\n- There should be some variation of sampling the train dataset (such as 1%). Multiple number of experiments and the presentation of the statistics of the results (mean/stddev)\n\nMinor comments: \n- The details about how they apply the scheduling of p(augmentation hyper-parameter) needs to be stated more.\n-Data specification is missing which is important especially in the accelerated MRI. It should be stated in the supplementary materials.\n- Not only for the sampling rate, but also details about the sampling pattern is important. Even if it is written in the paper of the model they used(End-to-End VarNet), it would be great to describe the details of the experiment.\n- Is it work for the super-resolution task? I mean that even if the task was acceleration of MRI by super-resolution (the sampling pattern is low-pass filtering mask and the deep model is trained for that), does the augmentation successfully work?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Vvkqncil3K_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nIn this paper, the authors design a data-augmentation pipeline for the domain of MRI reconstruction (specifically, by proposing sensible guidelines for augmenting k-space data when learning image reconstructions, to preserve the noise characteristics of the image data). They show that this pipeline works as you might expect data augmentation to work: it boosts results for small training sets and becomes increasingly less effective as the training set grows. However, while the problem domain is of interest, there are issues with the presented work.  \n\nOne of the most prominent weaknesses of the work as a whole is that there is little attempt to compare to sensible baselines. Table 1, with the chosen augmentation parameters, does not have any ablation or intuition about why the choices are made. It would be interesting to compare the effects of reducing this fairly large selection of augmentations. How far can we get with just flips and rotations, for instance? While the authors do compare against a 'naive' augmentation, these results are misleading. It isn't 'naive' to apply augmentation techniques from other domains: it is simply wrong! \n\nWhen the authors evaluate performance with different percentages of the training dataset withheld, the authors must clarify their method for subsampling. Is it random over all slices, or stratified by subject? If it is completely random, the same subject can have slices in both the included and excluded parts of the dataset, even though different slices from the same subject will be highly correlated.\n\nThe analysis of results tends to be highly qualitative. The authors point out that 'fine detail' can be obtained with just 1% of the training data, (Figure 3). Similarly, the lack of hallucinated features is not quantified but instead shown with a Figure with one example of reduced hallucination. While the images supporting these two results are visually appealing, there is no quantification of these differences, or attempts to find places where the reconstruction fails. In fact, it is likely that with some searching, instances could be found of hallucinated features, so suggesting that hallucination has been 'eliminated' via augmentation is an overclaim.  Overall, quantifying results of this sort would strengthen the work considerably. \n\nMore minor points: \n* The analysis is restricted to one dataset (fastMRI), on which it doesn't truly excel — it doesn't boost performance at all when using the full training set. Attempts to source other datasets would be welcome. \n* The nomenclature of the paper needs attention: specifically,  compressed/compressive sensing are both used. Stick to one, or mention them both in the introduction as being interchangeable. \n* The paper states that MRI does not use radiation on patients: correct this to 'avoids using ionizing radiation'. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9gVTGNqc5yS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to use data augmentation to improve accelerated MRI reconstruction when the amount of training data is limited. This is an important problem since MRI data is expensive to obtain. Traditional image augmentation methods can't be applied directly for this problem because MR images are complex valued. Further, the applied transformations need to preserve the noise distribution, without which model performance degrades significantly.\n\nThe key contribution of the paper is a carefully designed data augmentation policy inspired by the physics of the MR domain. Using this method the author obtains significant improvement in the low data regime for both single-coil and multi-coil reconstruction problems. The proposed method is also independent of the reconstruction model. The results shown in the paper are highly encouraging. The paper is clearly written and is easy to follow.\n\nSome things to clarify / improve the paper:\n\n1. How was the data subsampled? For the 1% experiments, did you choose 1% of slices or 1% of volumes? Since different slices of the same volume have a large amount of redundant information, these two methods are not equivalent. Sampling 1% of volumes is a more realistic scenario.\n2. What model was used for single-coil reconstruction? The authors mention using the E2E-VarNet model, but that model was designed specifically for multi-coil data.\n3. The paper showed that data augmentation improves SSIM as well as image quality when the amount of training data is small. However, in practice it is important for the model to generalize to different pathologies. It is unclear if the model trained with DA can reconstruct pathologies that were not present in training data. This is important because when the amount of training data is small, most pathologies will be missing from it. Please include examples of pathologies that were not observed during training.\n4. The experimental results in the paper clearly demonstrate the benefits of the proposed data augmentation method. However, the paper lacks sufficient ablation experiments that leaves a number of questions unanswered, for example: (a) are all of the augmentations used in the paper required? (b) how does the augmentation schedule proposed by the paper compare to other schedules like uniform augmentation?\n5. Did you use random masks during training? Training with random masks (either fully random masks or fixed-width masks with random starting offsets) has the effect of data augmentation without the use of explicit augmentations. If the study used fixed masks, then the effect of DA is probably being exaggerated. It would be good to include additional experiments with random masks.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers could not reach consensus here and legitimate concerns are raised on novelty and on empirical results, although this can be attributed to the important computation times required to run experiments on 3D MRI volumes. The authors have provided a comprehensive response to the reviews, the general feedback is that the work has merit but it fails to convince on its real contribution to the state-of-the-art. At this stage, I fear this work cannot be recommended for acceptance.
Decision: Reject