Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper describes a method for accelerating MRI scans by proposing lines in k-space to acquire next. The proposals are based on posterior uncertainty estimates obtained from GAN-based reconstructions from parts of the k-space acquired thus far. The authors address an interesting and important problem of speeding up MRI scans and thus improving the subject's experience. The proposed method achieves better posterior uncertainty and SSIM scores than competing methods.



While the paper considers an important problem and takes a novel approach to solve it (using a GAN generative model to estimate uncertainty), I found that it may be particularly inaccessible to non-experts in the field of MRI image processing. Furthermore, several important methodology-related questions remain unanswered in the paper; and the experiments offered in the paper are insufficient for convincingly arguing the author’s claims.

Specifically, it is unclear whether GANs with their mode dropping behaviour are the right model choice for proposing reconstructions - they are likely to drop modes and by extension - yield overconfident uncertainty estimates. This can be expected to be particularly problematic for scans of images that differ from the training distribution, and is exacerbated by the fact that authors train the model only on scans from healthy subjects. Furthermore, because of the the GANs propensity to drop modes, it is also unclear whether the posterior variance numbers reported in the paper are directly comparable between the methods.

The method used for obtaining the uncertainty estimates from GAN samples implicitly makes the assumption that reconstructions follow a Guassian distribution with a diagonal covariance. This assumption is also made in a competing method of Zhang et. al (2019) that the authors do not compare against, and claim to improve upon methodologically (i.e. the authors state that the method of Zhang et. al (2019) cannot be used to produce uncertainty estimates in Fourier space). I am not convinced that the authors claims about the method differences are sufficiently substantiated (see more under major comments). And because the methods bear significant similarity to each other, an experimental comparison - which is currently missing - should be carried out (on open datasets that ideally include non-healthy subjects).

Finally, the paper teases fast(er) MRI in the title, but doesn’t touch on this topic in the text. This aspect should of the authors contribution be discussed at length, in particular comparing the Cartesian sampling strategy adopted by the authors to other strategies, as well as evaluating the feasibility of implementing the adaptive sampling strategy in an actual scanner (e.g. can the network be ran fast enough)?


==================
Major comments:
==================

1) In the proposed method the authors employ a procedure in which the currently sampled parts y of the k-space are fed to a generator network to obtain n_s reconstructions. These n_s sampled reconstructions are then averaged to obtain the empirical mean and variance, with the latter being used for estimating uncertainty. This procedure is potentially problematic for several reasons:

  a) First, taking the empirical mean and variance of the samples is in fact equivalent to assuming that the reconstructed image follows a Guassian distribution with a diagonal covariance. This is the same assumption the authors argue is not realistic when discussing the work of Zhang et al. (2019) in the end of Section 1. 

  b) In case of GANs, which can model multi-modal distributions this uncertainty estimation is even more problematic in cases when the samples originate from different modes. What do the mean and variance represent then?

  c) As the authors highlight in the discussion section of the paper in the paper, GANs are prone to mode collapse. This is also potentially problematic for their estimator - in case of mode collapse their method would underestimate uncertainty. The fact that authors are able to use only two sampled reconstructions to estimate the mean and variance with acceptable accuracy is consistent with the occurrence of mode collapse in their generator. Furthermore, because mode collapse may occur in the author’s model, it is unsurprising that their method yields the smallest posterior variances in Table 1. The authors should provide evidence that mode collapse either does not occur or would not affect these numbers.

  d) Finally, the authors use the empirical mean and sampled reconstructions to obtain the empirical mean and variance in Fourier (k-) space. Specifically, they argue that “This feature is specific to generative models, as getting samples from P_{X|y_\omega} allows to transform these to a different domain [...] this is not possible with methods that only provide point-wise estimates of the mean and the variance in image space, such as the one used by Zhang et. al (2019)”. I don’t think this is true - Fourier transform is a linear transformation, thus given a mean and a variance in image space it is possible to deduce analytically what the mean and covariance in Fourier space would be. This should be elaborated in the text, and the comparison to Zhang et. al (2019) should thus be extended further.


2) The proposed method, CLUDAS, was evaluated against existing methods on a single proprietary dataset consisting of only 100 images from healthy individuals. This is potentially problematic, for several reasons:

  a) Using a proprietary dataset doesn’t allow follow-up works to compare against the author’s method; or for comparing CLUDAS to 
 existing methods not considered in the paper; the methods should additionally be compared on a public dataset  and

  b) Applying the method only to a single (small) dataset does not allow for reasoning on how the method behaves in different data regimes. I strongly encourage the authors to apply their method on public datasets, for example on data used in Zhang et. al (2019) - this would then allow for comparing the two methods despite not having access to an implementation of Zhang et. al (2019).

  c) Since the data is used to train the GAN model is obtained from healthy individuals, it’s unclear whether it can be used to acquire data from subject that may potentially have aberrations in their scans - the GAN model would be expected to produce low uncertainty estimates for regions where these aberrations would lie and not propose acquiring parts of the k-space that could be used to resolve these aberrations. For similar reasons using a GAN that potentially drops modes (e.g. scans of unhealthy individuals, for example because they are less common in the training data) is also problematic. The authors should consider evaluating their method on a dataset that contains non-healthy subjects, and investigate the performance of the method when it’s evaluated on healthy subjects, but tested on unhealthy ones.

3) The title of the paper (“[...] for fast MRI”) suggests that the authors aim to accelerate the MRI data acquisition process.

  a) Yet they chose the work with the Carterian sampling (i.e. sampling lines in the k-space parallel to the x-axis), which arguably requires larger sections of the k-space to be sampled before a high-quality reconstruction can be obtained (e.g. see http://mriquestions.com/k-space-trajectories.html). Providing information on how this choice influences the speed of data acquisition (and thus subject’s comfort) is important in order to assess the applicability of the author’s method to real world scenarios. This information should be provided.

  b) The paper does not actually describe how MRI is sped up. Given that the acquisition budget (number of lines acquired in k-space) is fixed, and assuming that time per line is constant, it is unclear where the speed up comes in. More generally, the speed claims / aspect of the proposed method should be discussed in more detail - how is speed measured and evaluated? How does it compare to non-Cartesian sampling approaches.

  c) Finally, it’s unclear whether neural network inference can be made fast enough to allow for a real world application of the adaptive CLUDAS sampling - can the data be transferred fast enough from the scanner to do inference and propose the next line to scan without causing delays in the scanning process? This should be discussed in the paper.

4) Currently, the paper places a strong expectation of knowing about MRI and being familiar with MRI-specific terminology on the reader. This makes the work substantially less accessible to a wide audience with machine learning expertise as the common denominator. The authors should take steps towards making the text more accessible to non-(MRI) expert audience, for example by introducing some of the basic knowledge (e.g. the data acquisition and reconstruction processes in MRI) early on, departing from MRI-specific jargon (e.g. k-space, lines in k-space) in favour of ML terminology whenever possible, and taking care to define and possibly illustrate (strongly encouraged) the MRI-specific concepts (e.g. the k-space, lines in k-space, sampling masks). Some specific examples the authors should address follow.

  a) The k-space is defined only in passing as being the frequency domain. It’s unclear whether these lines (which correspond to sampling masks) in this domain are parallel to the x-axis. The math (e.g. Equation 1) and Figure 2 suggest that, but it’s not obvious.
x is referred to both as "model parameter" in Section 2 (and Equation 1) as well as the "ground truth image" later in the same section. This is confusing, especially because in case of GAN model parameter would typically refer parameters of the generator and discriminator.

  b) SSIM is not defined anywhere, but already used in the abstract.

  c) It could be made more clearly why undersampling is required in case of MRI. E.g. how/why does it correlation with patient comfort.

  d) “K-space sampling” is used already in the introduction, but not really defined.

  e) It’s unclear whether sampling, subsampling and undersampling all refer to the same concept or not.

  f) The term “sampling mask” is already used in the introduction, but not clear what it refers to.

  g) Use of the term “innovation” to refer v_t, which appears to be a one-hot vector marking the newly added line in k-space.

  h) The use of term “sampling decision” to refer to v_i.

  i) Unclear what eta in “Z ~ eta“ in Equation 5 refers to - it is not defined. Later in Section 2.2 it is stated that “z_i are independent samples form Z”. Is this the same as “z_i ~ eta”? If so, why the second layer of notation?

  k) In Introduction “[...] which is not feasible on a real problem without the ground truth available”. Unclear what the ground truth refers to, I assume it’s the ground truth image.

  l) In Section 2.1 data refers to x_i, with i=1,...,m; which I assume are images and thus contain real numbers. Yet Section 3 states “As our data are complex [...]”. This is confusing - is that different data?

  m) Sampling is used ambiguously in the paper - to refer to sampling in the k-space (e.g. sampling masks, sampling decisions v_1,...,v_n) and to refer to sampling reconstructions from the generator. This should be resolved to improve readability of the paper.

  n) Not entirely clear what k-space pixel-wise variances are. And what is the difference between spatial and pixels-wise variances is (Section 2.2).



==================
Minor comments:
==================

1. In the Introduction the authors argue that metrics such as MSE and SSIM “[..] do not align with what clinicians see as valuable.”, yet use these throughout the paper for evaluating and comparing methods. This decision should be explained.

2. In Section 2 (and beyond) images x are described as belonging to subspace C^p of complex numbers. While this is technically true, them to actually belong to the space of R^p. If so, this should be reflected in the text for the benefit of the readers. Furthermore, I don’t think dimensionality p is actually defined anywhere.

3. Compressive / compressed sensing abbreviations CS is defined twice in Section 1.

4. It’s not entirely clear why “The CS-inspired methods shift the burden from acquisition to reconstruction [...]”

5. Incorrect double quotes are used throughout the text (both are right quotes).

6.  In Introduction “[...] yielding an estimator which can be used to drive back the whole sampling process in a closed-loop fashion.” 
it is unclear what it means to “drive back a sampling process”. Perhaps “back” should not be there?

7. In some cases it is unclear what the use of double quotes conveys, e.g.

  a) In Section 2.1 “[...] being the “full” mask [...]”

  b) In Section 2 “[...] ground truth “complete” images [...]”

  c) In Section 5 “[...] these inverse problems “depend” from each other [...]”

  d) Multiple places in Appendix B.

8. The convention of using small letters x, y for data samples / instances and capital letters X, Y for random variables could be made explicit.

9. Section mostly provides background (rather than theory) and could be named accordingly.

10. In Section 2.1, t refers to “time”. It may be clear if it were instead referred to as the step of the sampling process or something similar - the use of “time” to refer to some discrete set of actions can be a little confusing.

11. Section 2.1 refers to “[...] the online reconstruction speed of DL [...]”. This should be explained further - why are deep learning based approaches to reconstruction faster? Does this depend only on having the right hardware accelerators? Also, I don’t think the abbreviation “DL” was introduced.

12. Equation 4 is referred to both as an “Equation” and as a “Problem”.

13. In Equation 6 there is a summation over j from v_i. It is my understanding that v_i is a one-hot vector and it’s unclear what this summation means. Presumably it is the summation over pixels covered by line v_i, but the notation doesn’t convey this. It could be nice to also explain the “1D” superscript in this equation.

14. In Section 2.2 “[...] this is why the approach of (Adler & Oktem, 2018) minimizes over distance for observation in Equation 4 [...]”. I couldn’t follow the part about minimizing over the distance for an observation. Please consider making this more clear.

15. In Equation 8, what does index i run over?

16. Minor typos / textual issues:

  a) In Introduction “[...] of the our estimator [...]”

  b) In Introduction “[...] and show that even using a few samples [...]” -> even when using?

  c) In Section 2.2, after Equation 5 “[...] where t After finding the optimal”.

  d) Inconsistent use of “closed-loop” and “closed loop”.

  e) In multiple places throughout the paper a double space appears to be used instead of a single one.

  f) Section 4.1 “as can be in Figure 1”

  g) Section 4.3 “samples art random”

  h) In Section 5 “[...] “depend” from each other” should be depend on each other?

  i) In Section 5 “[...] we we found that [...]”

17. Unclear what’s meant by “[...] using the aggregated variance as a loss function” in Section 2.2

18. Section to refers to i* (integer scalar) as a line. Previously it was v_i (vector).

19. In Section 2.2 the authors state “Once the generator has been trained until convergence [...]”. The authors optimize a generator in an adversarial fashion. To my knowledge, this training procedure is not guaranteed to converge and would typically oscillate around a stable solution. Could the authors please comment on what they mean by convergence in this case and how they guarantee that the generators they train converge.

20. For the posterior variance results in Table 1 it should be discussed whether all the methods obtain / compute the variances the same way.

21. The “data consistency layer” (Section 3) should be explained briefly. How does it enforce perfect consistency and what’s meant by consistency here?

22. It should be made clear what MSE is calculated between in Figure 1 and Table 1. I assume it’s between ground truth (all frequencies sampled) and reconstructed (only some frequencies samples) images.

23. It’s not entirely clear what’s meant by consistency in Section 4.1

24. What do yellow arrows signify in Figure 2?

25. I don’t think the abbreviation UQ was defined in Figure 2.

26. The masks in Figure 2 are such that is a certain line was chosen at lower sampling rate (left on the x-axis), it would also be chosen at higher sampling rate (right on the x-axis). This is somewhat unexpected since the budget of lines v_i to be acquired differs between sampling rates. Why is there such consistency?

27. In Table 1, in the leftmost column, the number in brackets (number of posterior samples?) should be defined.

28. In Table 1 and Section 4.3: LBC-M and LBC-U - the -M and -U suffixes should be explained. What are the differences between the methods?

29. In Section 4.3: what are the FE methods?

30. Appendix A.3 is mostly a copy-paste from the main text. Unnecessary duplication?

31. Axes in Figure 4 (Appendix A.2) are not labeled or described.

32. Loss in A.4 (Equation) does not match Equation 5. In the former the discriminator takes three arguments instead of two, and the arguments z1 and z2 are not described.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper is quite well written and the idea is novel. However, the results are rather weak. The authors present a method to perform adaptive MR compressed sensing, i.e. decide online which readout to sample next. They compare it to an offline learning method where one sampling pattern is optimized for a whole training set, then applied to test data. The offline method performs better in terms of MSE, which is the loss it was trained for, meaning that the authors have not demonstrated a gain in adapting the sampling pattern to individual scans.

The primary concern with the paper is not with the author’s contribution, but with serious flaws in [Adler 2018] that unfortunately snowball into this one. While the authors in [Adler 2018] do acknowledge issues with learning a variance, they misdiagnose the problem as mode collapse. Mode collapse is an optimization problem, where the training set contains variability but the generator fails to learn it due to the lack of an encoder. That is not the case here: all the variability of the training set is encapsulated in y, and for each y the target empirical posterior distribution is a Dirac. This is very similar to the calibration problem in classification [1], where classifiers become overconfident because they are trained to always output 0 or 1. If the generator does not learn a Dirac, it can only be because of regularization (either explicit or implicit in the model architecture) or optimization failure (either involuntary or voluntary with early stopping.) Tweaking the loss as advocated in [Adler 2018] does not fundamentally change the problem as long as the loss is minimal at the target empirical posterior. It may change the dynamic behavior and result in posteriors with more variance when combined with early stopping, but those variances are not calibrated, i.e. they have not been trained to match the variances of the true continuous posteriors. In order to learn the variances, one would have to either provide multiple posterior samples for each y during training (not practical in this case,) or perform some kind of calibration on the validation dataset as in [1], i.e. learn the mean and variance from different data, which effectively uses the network’s interpolation properties as a proxy for true random sampling.

However this flaw does not invalidate the practical approach developed in the paper, but it seriously undermines its qualifications as a rigorous, principled, Bayesian approach. It also makes the reporting of posterior variances as final quality metrics pretty much useless since they are not interpretable: does lower variance mean that the generator got better at estimating the missing information, or that it got worse at estimating the true posterior variance? I would suggest to at least remove the variances highlights from Table 1 and Table 4, and maybe scrap the data altogether. The paragraph on posterior estimation should also be updated to represent the whole scope of the problem. 

Section 2: Theory: suggest to remove “Without loss of generality”. Due to the known issues with variance estimation, having p(y | x) as a density instead of a Dirac could very well change the behavior of the generator.

Section 2.1 Adaptive masks. The whole first paragraph is somewhat misleading and should be revised. Real-time reconstruction is indeed possible without deep learning, see [2] for example. Furthermore, real-time reconstruction is not nearly fast enough for adaptive sampling. Real-time reconstruction means that reconstructing an image is at least as fast as scanning the whole image, i.e. in the order of 0.1 to 1s., but for adaptive sampling one must reconstruct at least as fast as the time between two successive readouts, i.e. in the order of 1 to 10 ms. Both [Jin 2019] and [Zhang 2019] only showed single-coil offline simulations with no indications of the reconstruction time and so do the authors.

Figure 1: I must be missing something here. How can the image-domain and Fourier-domain figures be different? The Fourier transform being orthogonal, norms and variances should be the same in both domains.

[1] C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, “On Calibration of Modern Neural Networks”, ICML 2017 70:1321-1330.
[2] M. Uecker, S. Zhang, and J. Frahm, “Nonlinear Inverse Reconstruction for Real-Time MRI of the Human Heart Using Undersampled Radial FLASH”, MRM 63:1456-1462 (2010).

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes an uncertainty driven acquisition for MRI reconstruction. Contrary to most previous approaches (which try to get best reconstruction for a fixed sampling pattern) the method incorporates an adaptive, on-the-fly masking building (which is similar in spirit to Zhang at al. 2019). The measurements to acquire are selected based on variance/uncertainty estimates coming from a conditional GAN model. This is mostly an "application" paper that is evaluated on one dataset.

Strengths:
- The paper studies an interesting problem of adaptive MRI reconstruction
- The review of MRI reconstruction techniques is well scoped

Weaknesses:
- The evaluation is rather limited and performed on one, proprietary, relatively small sized dataset
- Some simple baselines might be missing


I like the idea of adaptive sampling in MRI. However, I'd slightly lean towards rejection of the paper. My main concerns are as follows:

The presentation of the paper could be improved. At the moment, the Theory section describes background information, related work and problem definition as well as the contribution of the paper. Maybe braking the section into related work, background and methodology (where the main contribution is presented) sections would improve the paper readability. 

The paper uses a conditional GAN model (with a discriminator from Adler & Oktem, 2018 and a generator that is based on Schlemper et al. 2018). Making the methodological contribution to be rather limited.  The main difference w.r.t. the previous papers seem to be the last paragraph of  section 2.2 - the empirical variance estimation is performed in Fourier space. 

A simple baseline to compare might be to train a Unet-like model (e. g. Schlemper et al 2018) with a Gaussian observation model (outputting a mean and a variance per each pixel) and train it to minimize Gaussian NLL. At the test time, one could simply sample from the Gaussian model instead of taking just the argmax of the output. It might be the case that the assumption of gaussian image might be too simplistic, however, it would be interesting to show it experimentally. Note that when sampling from such model the empirical variance estimation could be performed is the Fourier space too.

The experimental evaluation is rather limited and the dataset used in the experimental section is small. Adding another dataset would make the paper stronger.

Other comments:

There is a mention on training dataset and testing dataset -- there is no mention on validation set. How were the hyperparamenters of the conditional GAN selected?

As acknowledged by the authors, this paper bears several similarities with the work of Zhang at al. 2019. However, the approach is not compared to Zhang et al. Including this comparison would make the paper stronger.

It is interesting to see that CLUDAS outperforms CLOMDAS in terms of SSIM. If I understand this part properly, CLOMDAS uses ground truth image to estimate MSE. Is it expected that CLUDAS would outperform CLOMDAS? 

Section 5, Adaptive vs. fixed mask: "We also have a simple generalization bound of the obtained mask, relaying on a simple application of Hoeffding's inequality." Could the authors add a citation or explain this part in more detail?


Some typos:
"...we aim make a series..."
".. define an closed-loop..."
"We choose adopt a greedy"
"... we we found that..." 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes an active data acquisition framework for magnectic resonance imaging. A generative adversarial network is used to estimate the posterior distribution of the latent MRI image in a closed-loop and greedy manner where the uncertainty of the posterior image is used to guide the process.

The paper is well written and easy to follow. The main contribution seems to be the combination of deep Bayesian inversion in Adler & Oktem, 2018 with an uncertainty driven sampling framework. Using uncertainty to drive data acquisition and exploration is not a new idea; the concept has been applied to reinforcement learning, active learning, Bayesian optimisation, as instantiations of a broad class of methods in experimental design. The experimental results suggest that the technique can reduce the amount of time required to obtain good quality images from MRI scans which can potentially have a big financial impact. The technique is compared to several variants of compressed sensing approaches demonstrating superior performance. 

My main concerns with the paper are:

1. The key idea of using uncertainty to guide sampling was also the main concept in Zhang et al. 2019. This submitted paper highlights differences in the models but does not provide an experimental comparison. Since both papers share the same concepts, this reviewer considers that a comparison is critical.

2. Deep Bayesian inversion approximates the posterior distribution by minimising the Wasserstein distance between the posterior  and a parametrised generator. I find the idea potentially powerful, with the advantage of learning a generative model as well, but wonder how this compares in theory and in practice to simpler stochastic variational inference and modern Hamiltonian MCMC. The min-max formulation is notoriously difficult to optmise and might lead to many local optima and instabilities.

3. Given the complexity of learning GANs and the sensitivity to initialization, results should contain more information such as the std of the MSE for several runs of the algorithm.    

review_title: Official Blind Review #3

Meta-review: The author responses and notes to the AC are acknowledged.  A fourth review was requested because this seemed like a tricky paper to review, given both the technical contribution and the application area.  Overall, the reviewers were all in agreement in terms of score that the paper was just below borderline for acceptance.  They found that the methodology seemed sensible and the application potentially impactful.  However, a common thread was that the paper was hard to follow for non-experts on MRI and the reviewers weren't entirely convinced by the experiments (asking for additional experiments and comparison to Zhang et al.).  The authors comment on the challenge of implementing Zhang is acknowledged and it's unfortunate that cluster issues prevented additional experimental results.  While ICLR certainly accepts application papers and particularly ones with interesting technical contribution in machine learning, given that the reviewers  struggled to follow the paper through the application specific language it does seem like this isn't the right venue for the paper as written.  Thus the recommendation is to reject.  Perhaps a more application specific venue would be a better fit for this work.  Otherwise, making the paper more accessible to the ML audience and providing experiments to justify the methodology beyond the application would make the paper much stronger.
Decision: Reject