Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper gives a simple yet straightforward method to do data augmentation in MRI imaging. The paper is well written and the results quite promising on a rigorous test case.

However, the methods proposed are in a sense rather obvious and the results do not give an improvement on the main practical test case. Throwing away data-post hoc sadly isn’t as strong of a motivation as showing an improvement on the full dataset.

Still, the methods are novel, the presentation good and all steps are done rigorously so I can certainly see this being useful for a huge range of related applications. For this reason I’ll recommend acceptance.

Some further comments include:

* As noted by the authors, post-processing methods are an alternative to physics based approaches. An upside of them is that it’s trivial to do very aggressive data-augmentation (see e.g. Deep Bayesian Inversion which applied the whole menagerie). It could be interesting to have a standard U-Net baseline.
* The “MRI is a Fourier transform” argument is well known to be a rather coarse approximation. It would be interesting to hear some debate on how a slight error in the forward model would impact this application.
* Related to the above, MRI is a “trivial” inverse problem since the Fourier transform is unitary, basically a “rotation” in some high dimensional space. Comments on how to apply these methods to more advanced inverse problems like CT would be most welcome.
* H and V flip looks wrongly labeled in Fig2.
* How are the images e.g. rotated without padding artefacts?
* In figure 12 the baseline should also show a training curve
* How are the windows selected in e.g. figure 10? It feels like it’s slightly different between samples and also a bit too wide.
* All figures that are not “images” should be in vector format. Notably figure 4.

review_title: Data augmentation for deep learning based accelerated MRI reconstruction

final_score: rating: 6: Marginally above acceptance threshold
review_content: Example 2: 
 

Review: This paper proposes data augmentation methods for medical imaging(especially for accelerated MRI) based on the MR physics. The augmentation includes both pixel preserving augmentations/general affine augmentations on both real and imaginary values in the image domain. Then, the augmented images are transformed to k-space domain and the k-space data are down-sampled for the input data generation for the accelerated MRI task. They claim that how to schedule p(the probability of applying combinations of augmentation) over the training is important and the schedules from p=0 and increasing over epochs shows best results, experimentally.

+ The importance of data augmentation is very important topic, especially in medical domain, because of their expensive cost for data gathering. It directly effects on the performance of deep learning algorithms. 
+ Overall, the paper is well written and might be easy to read by the readers who are not familiar with accelerated MRI.
+ The results section is well structured. In particular, the results seem great and the proposed augmentation works well especially for the extreme cases such as using 1% of the training set.

Concerns: 
- The key concern about the paper is the lack of experimentation to study the usefulness of the proposed method. 
- Considering the limited results, a deeper analysis of the proposed method would have been nice. The idea of augmentation in the MR image domain for each real and imaginary domain is a generic one, and many of MR researchers are already tried to augment the MR image on image domain and then after down-sampling for the accelerated MR task. I cannot clearly see the novelty of the proposed algorithm except for scheduling.
-There are not enough ablation studies. For example, they are used variety of transformation for augmentation. The effects of each transformation should be different on the performance of the algorithm. Also, even the author divides the transforms into two categories: pixel preserving/general affine augmentation and the effects of two different augmentation can be helpful to the readers and researchers in the accelerated MRI deep learning community. However, it is missing.
- In the 3rd section, the author tries to link the necessary of the augmentation pipeline for preserving noise statistics with their algorithm. However, the analysis and comparison is not well-provided. I think the ‘naïve DA’ which they provided (just augmentation on real valued image) is not a fair comparison experiment since it is the half of the data (real vs complexed value). The results of the other augmentation algorithm and/or the more analysis based on their theory are missing.
- Also, the author mentioned about the physics of the data. However, all the augmentations they used(horizontal/vertical flips, rotations, translations, zoom-in/out, scaling, shearing) are the transformations which affects to the coil sensitivity map. Thus, the augmented train sets have the samples which have different coil sensitivity maps compared to the one which test sets have. I think they need to consider about these factors to claim about the physics based augmentation. (for example, 1. Calculate coil sensitivities of each coil, 2. Reconstruct the true magnetization X, 3. Apply augmentation on X and then apply the coil sensitivity map, 4. Get the k-space data and down-sampling.) 
- The results seem more sharp then the one which not use the proposed domain adaptation and the SSIM score is better too. However, it is important the opinions of the clinician in department of radiology. In some cases, the clinicians might see some special features on the MR images even it is blurred. Thus, if the deep learning algorithm might lost that features, they prefer the blurred one even the other is more sharp. The evaluations by the clinicians make the author’s claim stronger.
- There should be some variation of sampling the train dataset (such as 1%). Multiple number of experiments and the presentation of the statistics of the results (mean/stddev)

Minor comments: 
- The details about how they apply the scheduling of p(augmentation hyper-parameter) needs to be stated more.
-Data specification is missing which is important especially in the accelerated MRI. It should be stated in the supplementary materials.
- Not only for the sampling rate, but also details about the sampling pattern is important. Even if it is written in the paper of the model they used(End-to-End VarNet), it would be great to describe the details of the experiment.
- Is it work for the super-resolution task? I mean that even if the task was acceleration of MRI by super-resolution (the sampling pattern is low-pass filtering mask and the deep model is trained for that), does the augmentation successfully work?

review_title: Recommendation to Reject

final_score: rating: 6: Marginally above acceptance threshold
review_content: An augmentation method is proposed for MR image reconstruction, and is shown that significant results can be achieved with a small fraction of training data. 

Comments 
One of the major concern in image reconstruction problems is the availability of fully sampled data, while the dataset used in the problem provides this, how the proposed method will work in the absence of such data for generating the augmentations? 

For MR reconstruction, the undersampling mask plays a significant role, while augmenting real images, what will be the effect on the mask. In particular, for 8x which masks are used in this work, are these similar to baseline or always used random sampling?

For results evaluation, SSIM is used, it is recommended to add results for either PSNR or NMSE for error quantification. 

How are the samples for various data subsets selected? 

Since it is claimed that this can be extended to other MRI datasets, is it possible to add results for Brain MRI as well fo having diverse data and hence a better representation on the generalisation of the MRAugment model. For model training, any form of transfer learning is used or all models are trained form scratch? 

A comparison with state-of-the-art methods is missing and should be added. 

Minor comments

There are typos and grammar related mistakes which should be corrected. 
review_title: The authors present a data augmentation framework for MRI reconstruction and evaluate performance on a large scale publicly available data. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a method to use data augmentation to improve accelerated MRI reconstruction when the amount of training data is limited. This is an important problem since MRI data is expensive to obtain. Traditional image augmentation methods can't be applied directly for this problem because MR images are complex valued. Further, the applied transformations need to preserve the noise distribution, without which model performance degrades significantly.

The key contribution of the paper is a carefully designed data augmentation policy inspired by the physics of the MR domain. Using this method the author obtains significant improvement in the low data regime for both single-coil and multi-coil reconstruction problems. The proposed method is also independent of the reconstruction model. The results shown in the paper are highly encouraging. The paper is clearly written and is easy to follow.

Some things to clarify / improve the paper:

1. How was the data subsampled? For the 1% experiments, did you choose 1% of slices or 1% of volumes? Since different slices of the same volume have a large amount of redundant information, these two methods are not equivalent. Sampling 1% of volumes is a more realistic scenario.
2. What model was used for single-coil reconstruction? The authors mention using the E2E-VarNet model, but that model was designed specifically for multi-coil data.
3. The paper showed that data augmentation improves SSIM as well as image quality when the amount of training data is small. However, in practice it is important for the model to generalize to different pathologies. It is unclear if the model trained with DA can reconstruct pathologies that were not present in training data. This is important because when the amount of training data is small, most pathologies will be missing from it. Please include examples of pathologies that were not observed during training.
4. The experimental results in the paper clearly demonstrate the benefits of the proposed data augmentation method. However, the paper lacks sufficient ablation experiments that leaves a number of questions unanswered, for example: (a) are all of the augmentations used in the paper required? (b) how does the augmentation schedule proposed by the paper compare to other schedules like uniform augmentation?
5. Did you use random masks during training? Training with random masks (either fully random masks or fixed-width masks with random starting offsets) has the effect of data augmentation without the use of explicit augmentations. If the study used fixed masks, then the effect of DA is probably being exaggerated. It would be good to include additional experiments with random masks.

review_title: Physics aware data augmentation for MRI reconstruction

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
In this paper, the authors design a data-augmentation pipeline for the domain of MRI reconstruction (specifically, by proposing sensible guidelines for augmenting k-space data when learning image reconstructions, to preserve the noise characteristics of the image data). They show that this pipeline works as you might expect data augmentation to work: it boosts results for small training sets and becomes increasingly less effective as the training set grows. However, while the problem domain is of interest, there are issues with the presented work.  

One of the most prominent weaknesses of the work as a whole is that there is little attempt to compare to sensible baselines. Table 1, with the chosen augmentation parameters, does not have any ablation or intuition about why the choices are made. It would be interesting to compare the effects of reducing this fairly large selection of augmentations. How far can we get with just flips and rotations, for instance? While the authors do compare against a 'naive' augmentation, these results are misleading. It isn't 'naive' to apply augmentation techniques from other domains: it is simply wrong! 

When the authors evaluate performance with different percentages of the training dataset withheld, the authors must clarify their method for subsampling. Is it random over all slices, or stratified by subject? If it is completely random, the same subject can have slices in both the included and excluded parts of the dataset, even though different slices from the same subject will be highly correlated.

The analysis of results tends to be highly qualitative. The authors point out that 'fine detail' can be obtained with just 1% of the training data, (Figure 3). Similarly, the lack of hallucinated features is not quantified but instead shown with a Figure with one example of reduced hallucination. While the images supporting these two results are visually appealing, there is no quantification of these differences, or attempts to find places where the reconstruction fails. In fact, it is likely that with some searching, instances could be found of hallucinated features, so suggesting that hallucination has been 'eliminated' via augmentation is an overclaim.  Overall, quantifying results of this sort would strengthen the work considerably. 

More minor points: 
* The analysis is restricted to one dataset (fastMRI), on which it doesn't truly excel — it doesn't boost performance at all when using the full training set. Attempts to source other datasets would be welcome. 
* The nomenclature of the paper needs attention: specifically,  compressed/compressive sensing are both used. Stick to one, or mention them both in the introduction as being interchangeable. 
* The paper states that MRI does not use radiation on patients: correct this to 'avoids using ionizing radiation'. 

review_title: The authors propose a data augmentation pipeline specifically for MRI reconstruction. 

Meta-review: Reviewers could not reach consensus here and legitimate concerns are raised on novelty and on empirical results, although this can be attributed to the important computation times required to run experiments on 3D MRI volumes. The authors have provided a comprehensive response to the reviews, the general feedback is that the work has merit but it fails to convince on its real contribution to the state-of-the-art. At this stage, I fear this work cannot be recommended for acceptance.
Decision: Reject