Closed loop deep Bayesian inversion:  Uncertainty driven acquisition for fast MRI
Authors: Thomas Sanchez
Igor Krawczuk
Zhaodong Sun
Volkan Cevher
Venue:  (2020)
Keywords: Deep Bayesian Inversion
accelerated MRI
uncertainty quantification
sampling mask design

Abstract: This work proposes a closed-loop, uncertainty-driven adaptive sampling frame- work (CLUDAS) for accelerating magnetic resonance imaging (MRI) via deep Bayesian inversion. By closed-loop, we mean that our samples adapt in real- time to the incoming data. To our knowledge, we demonstrate the first genera- tive adversarial network (GAN) based framework for posterior estimation over a continuum sampling rates of an inverse problem. We use this estimator to drive the sampling for accelerated MRI. Our numerical evidence demonstrates that the variance estimate strongly correlates with the expected MSE improvement for dif- ferent acceleration rates even with few posterior samples. Moreover, the resulting masks bring improvements to the state-of-the-art fixed and active mask designing approaches across MSE, posterior variance and SSIM on real undersampled MRI scans.

Under review as a conference paper at ICLR 2020
CLOSED LOOP DEEP BAYESIAN INVERSION: UN- CERTAINTY DRIVEN ACQUISITION FOR ACCELERATED MRI
Anonymous authors Paper under double-blind review
ABSTRACT
This work proposes a closed loop, uncertainty-driven adaptive sampling frame- work (CLUDAS) for accelerating magnetic resonance imaging (MRI) via deep Bayesian inversion. By closed loop, we mean that our samples adapt in real-time the incoming data. To our knowledge, we demonstrate the first generative adver- sarial network (GAN) based framework for posterior estimation over continuum sampling rates of inverse problem. We use this estimator to drive the sampling for accelerated MRI. Our numerical evidence demonstrates that the variance es- timate strongly correlates with the expected mean squared error (MSE) improve- ment for different acceleration rates even with few posterior samples. Moreover, the resulting masks bring improvements to the state-of-the-a fixed and active mask designing approaches across MSE, posterior variance and structural similar- ity metric on real undersampled MRI scans.
INTRODUCTION
Myriad of applications in control, data processing, and learning- from platform navigation to data mining and from channel estimation to compressive sensing (CS)-involve linear projection of signals or data points into diver-dimensional space. In this dimensionality reduction, the measure- ment the sensing matrix determines how much information we acquire per measurement, the ensuing computational ease of processing (since algorithms use the sensing matrix and its adjoint as subroutines), and the recovery guarantees.
In a resource constrained setting, these utilities create a Pareto trade-off wherein improving one worsens another. To impact all these fronts simultaneously, adaptive sensing (or sequential exper- imental design, active learning, etc.) aims to close the loop between the data acquisition and the inference, for instance, by exploiting information collected in past samples to adjust the future sam- pling process. While adaptive procedures promise great improvements over non-adaptive methods, they computationally demanding for real-time online response.
In the context magnetic resonance imaging (MRI), the dimensionality reduction process (i.e., undersampling in the Fourier domain, often referred to k-space) directly correlates with patient comfort, as it results in shorter scanning times. For the last decade, approaches motivated by com- pressed sensing have enabled successful reconstruction from highly accelerated (i.e., subsampled) data (see Lustig et al. (2007); Ravishankar & Bresler (2011b); Lingala & Jacob (2013): Otazo et al. (2015); Jin al. (2016) and references therein). While compressed sensing prescribed fully random sampling (Candès et al., 2006; Donoho, 2006) for recovery, most CS-inspired approaches to MRI departed from this paradigm and relied on the heuristies of variable-density sampling (VDS) (Lustig et al., 2007).
There, the sampling pattern (or mask) is picked at random from a probability distribution that rea- sonably imitates the energy distribution in Fourier space, whereas fully random sampling of the Fourier space ignores this important structure in the signal, which leads to practically poor results (Lustig et al., 2008). VDS appears heuristic middle ground for sampling pattern incorpo- rate the structure of energy distribution Fourier space, while preserving the benefits of incoherent sampling. In VDS, the probability distribution considered has traditionally been parametric (Lustig

---

Under review as a conference paper at ICLR 2020
et al., 2007; Chauffert et al., 2014; Boyer et al., 2016) constructed from data (Knoll et al., 2011; Vellagoundar & Machireddy, 2015; Bahadir et al., 2019).
The CS-inspired methods shift the burden from acquisition to reconstruction, as most of these meth- ods are iterative, preventing online reconstruction of accelerated data. This slow reconstruction rendered the problem of optimizing Fourier space sampling prohibitively expensive, little work was devoted to general method of designing sampling patterns for generic sampling methods (Gözcü al., 2018).
In recent years, deep learning applied to MRI enabled high quality reconstruction for unprecedented acceleration rates (Wang al., 2016; Schlemper et al., 2017; Hammernik et al., 2018), as well as near-online reconstruction times. This also led to a freshly renewed interest in the problem of optimizing the Fourier space sampling pattern (Bahadir et al., 2019; Weiss al., 2019; Sherry al., 2019; Jín et al., 2019; Zhang et al., 2019), although most of the research energy is still focused on developing more efficient reconstruction methods.
In its current stage, deep learning applied to MRI suffers from three main drawbacks: (i) as men- tioned, most of the research energy has been devoted to more efficient reconstruction methods, despite recent results showing that the sampling masks used have significant effect on the quality of reconstruction (Gözeü et al., 2018; Gözeüi et al., 2019), (ii) assessing the reliability the predic- tion of reconstructed image is difficult for clinician, due to the black-box nature of deep learning methods and (iii) the commonly used metrics for assessing the quality of the reconstruction (e.g. MSE, PSNR, SSIM) do not align with what clinicians see as valuable (Cheng et al., 2019).
A recent work of Adler & Öktem (2018) successfully demonstrated that a conditional Wasserstein GAN (cWGAN) (Arjovsky et al., 2017) given undersampled measurements in a tractable fashion while only relying on samples from the joint distribution. This result provides key opportunity address all three drawbacks in the context of MRI by using the posterior variance of the reconstruction as an uncertainty estimate, which is a more natural criterion for image quality.
be used to learn the posterior distribution of images
To this end, we show that a conditional WGAN can be trained on a continuum of inverse problems on various sampling rates, yielding an estimator of the posterior variance in the Fourier domain that can be used to drive the whole sampling process in a closed loop fashion. Despite the model being trained only for reconstruction and not sampling, the resulting variance estimator can reliably be used to guide closed loop sampling procedure, thus providing patient-adapted sampling masks.
In particular, we demonstrate that the generated masks that minimize the uncertainty estimates in an online fashion reach similar reconstruction MSE as compared to the state-of-ther of-the-art art fixed-mast approaches like Gözcü et al. (2018); Gözcü et al. (2019). While these approaches explicitly focus on minimizing MSE, we show that CLUDAS naturally outperforms them in terms of key visual metrics such as SSIM (Wang et al., 2004) without being trained on these metrics.
In addition, we investigate the reliability of our estimator in wide range of undersampling regimes and show that even when using a few samples from the cWGAN posterior, the variance estimate is reliable enough to be used to drive the design of the mask. This makes it feasible to use CLUDAS in a closed loop adaptive setting, where our approach is competitive with an approach using an MSE- oracle on the testing set (which is not feasible real problem without the ground truth available), and also beats strong open loop adaptive baselines while being easier to train and easier to apply. Contributions.
We show how to train a cWGAN to generate posterior samples across a continuum of sampling rates; We solve in Bayesian fashion not only single inverse problem, but a continuum of inverse problems.
We demonstrate the first posterior-basee mask design method for MRL.
We propose use the variance of the posterior distribution of images given measurements as quality metric for mask performance.
We show that despite the network being trained as a reconstruction method and not as a sampling method, our adaptive approach CLUDAS is competitive in both settings and strong contender for being used in active settings as it matches computationally expensive state-of-the-ane approaches.
2

---

Under review as a conference paper at ICLR 2020
Implications. We contend that uncertainty driven sampling framework extend to many similar problems where one wants to reduce acquisition times, such as atomic force microscopy (Abramovitch et al., 2007), transmission electron microscopy (Kovarik et al., 2016), trajectory optimization for ultrasound acquisitions (Malinen et al., 2005). Our results open the door of lever- aging Bayesian experimental design methods for designing adaptive sampling patterns in clinical settings and beyond.
Related works. We especially want highlight the work of Zhang et al. (2019), which bears several similarities with our method. It is important to note that their approach is not generative, as they only have point estimates of the mean and some learned uncertainty metric. Moreover, they assume the reconstructed image to be normally distributed with a diagonal covariance, a practically unrealistic assumption, which not required in Adler Öktem (2018).
2 NOTATION AND PROBLEM SETTING
Notation. Throughout this paper, we will refer to vectors as boldface lowercase letters x, y, and assume that they correspond N x N images that are vectorized to p dimensional space. In particular, we will assume that these vectors belong to some appropriately defined spaces N, y C Cº. We use vectors in C² as MR images are inherently complex. Boldface uppercase letters X, Y will in general refer to random variables (formally, they random vectors), with the exception of and P.. which will respectively refer to the discrete Fourier transform operator and the sampling operator, that will be defined below. Finally, will use E O 1)" be p dimensional binary vector, and we will refer to it interchangeably as a sampling, subsampling or undersampling pattern/mask. Finally, we will refer the distribution of random variable X Px, and extend the nottation to conditioned random variables such as Px,y = Px|Y=='
Problem setting. An inverse problem is the task of recovering the ground truth x X from measurements y. In the case of MRI, we consider the following acquisition model
Fx c.
(1)
where yu, E y' corresponds to the measurements obtained with given sampling mask w and where the sampling operator (Pw). = i E w, otherwise. w S [p] := [I,...,p] p is a set containing the sampled locations. Note that while the ground truth x is image living in the image space X,
the measurements yw live in the Fourier space. The Fourier space is often referred to as k-space in the MRI literature.
In this paper, we restrict ourselves to the setting where w is composed of lines in the Fourier space, also known as Cartesian sampling in the MRI literature', and usually constrained by maximal number n of lines that can be acquired. F denotes the Fourier transform, x E X' is the ground truth image and E is white additive noise. Without loss of generality, neglecting basic sampling effects, such as magnetic field inhomogeneity and spin relaxation, we assume e = 0 in the sequel.
As we will be working in Bayesian framework, we define a random variable X ~ Px from which ground truth, complete measurements are generated, distributed according to the unknown prior Px. From this we also define the distribution Px_ as well as the joint distribution Px,Y, and the posterior Px where Y., = P.. P.,.F..Th The posterior especially of interest as for fixed yo E Y. Pxiy (short for P represents the probability distribution of ground truths that are likely to have generated the observed data y., with a given mask w.
3 BACKGROUND
3.1 MASK DESIGN IN MRI
Fixed masks. The overwhelming majority of data-driven mask design approaches work in an open loop fashion: the sampling mask is built using training data and kept fixed at inference time. Even
'Tiis kind of structured acquisition originates from physical considerations and has the benefit of being casily implementable in practice.
3

---

Under review as a conference paper at ICLR 2020
the VDS paradigm, that prescribes sampling mask at random from parametric distribution, uses a fixed mask that is tuned in an ad-hoe fashion when applied clinically (Jaspan et al., 2015).
Formally, consider training set of original data {x, Jm that are assumed to originate from an unknown prior distribution Px. We are constrained to a maximal sampling budget n and want to find a mask that minimizes given loss function e (e.g. MSE, SSIM) these training samples.
The abstract problem of our interest then can be written as follows
min Ex-Px [&(X, Y = P.....] WEA
where A denotes the constrained set of masks w that are made of lines (cf. section ) and that respect the maximal sampling budget, i.e. |w/| I n (here, | denotes the cardinality of the set w). Formally, let us define S as a set of subsets of {1..... P)} that contains the N possible lines given vectorized image of dimension p = N2. A is defined as
A {w S : U U,S [,S, S n2)
which means that is constructed as union of the elements of a subset S of all possible lines S, with the additional constraint that the overall mask will respect the sampling budget n. However, the finite amount of samples requires solve the empirical risk minimization version of Equation 2, namely minweA 1/2 Σ; y = Fx,). Given that m is large enough and that the training and testing data do originate from Px. statistical learning theory guarantees that a mask performing well on the training set will adequately generalize.
In the literature, two trends are noticeable. A first body of works focused on constructing a good distribution from which to sample Ravishankar & Bresler (201 Vellagoundar & Machireddy (2015); Bahadir et al. (2019); Sherry et al. (2019). Other approaches tried to directly design fixed sampling mask that performs well on training data (Seeger et al., 2010; Gözcü et al., 2018; Gözeü et al., 2019; Haldar & Kim, 2019). More recent approaches tried to jointly train the mask with a deep network Weiss et al. (2019); Bahadir et al. (2019).
Adaptive masks. Until the re-birth of deep learning. most reconstruction methods relying on CS suffered from long reconstruction times, due to their iterative nature. However, recent works leverag- ing the online reconstruction speed of deep learning achieved mask designs in a closed loop fashion, ie., developing algorithms that could be used in online fashion to adapt to patients.
For a fixed, unknown data x, the adaptive approach aims at leveraging the information from the already measured frequencies to guide what should be acquired next. Instead of fixed mask w, are building up partial masks wt as union of individual Fourier space lines it with w = wy being the largest mask satisfying [WT| S n. The optimization now happens runtime as multistage problem which has to choose each new element Un such that
min ((x, = P..,Fx: WT U.,,) ae /0/
that is, we want to take at each time step t the sample that will allow to get the lowest final error, but being constrained by our previous acquisitions and the fact that we cannot look into the future to know where we should sample. The problem requires developing online sampling method that uses the partial information Yw, at each to decide on its next action.
Two approaches have been proposed for this problem in the literature. Jin al. (2019) took self-supervised learning approach, where sampling network learns to imitate a Monte-Carlo tree search method and predicts V2 Ug.... Ut-1 for all t. Zhang et al. (2019) leveraged adversarial training to jointly train a reconstruction algorithm and an evaluator that gives scores to the quality of reconstructed lines in Fourier space. The sampling procedure simply iteratively added the mask the lines with the lowest reconstruction score.
Other types of sampling. While our work here focuses on Cartesian sampling, which is by far the most widely used trajectory in MRI (Lustig et al., 2007), many other physically feasible trajectories have been investigated over the years for accelerated MRI. Radial trajectories have mainly been used in the context of dynamic MRI (Zhang et al., 2010a; Feng et al., 2014), and non-structured trajectories have also been explored and validated on real acquisitions (Lazarus et al., 2017). In particular, we note that some CS CS-based methods have shown online reconstruction times in the context of dynamic MRI (Zhang et al., 2010b) using radial trajectories, but such trajectories are rarely used in the context of static MRI.
4

---

Under review as a conference paper at ICLR 2020
4 METHODOLOGY
4.1 DEEP BAYESIAN INVERSION
In (Adler & Öktem, 2018), the authors propose a framework to estimate the posterior distribution Px i.e.. the distribution of original images x that are likely to have generated the observed data y. They formulate the problem as finding a parametrized generator Go- : y + Px that allows to minimize the Wasserstein distance with the unknown posterior Pxly over all the observations, namely minimizing
min BY [W(PX GY))) WEB
Here, Px is the space of probability measures on X. As this approach in not tractable in practice, they show that Equation 4 can equivalently be formulated as
min AE
max (xxxx)xxxx - x) Z-n
(5)
PED
After finding the optimal parameters (0", o²), the conditional generator G(z. : X' X y + x approximates the posterior distribution Pxy and different values of z yield different samples from
Pxly.
Note that Adler & Öktem (2018) applies this method to reconstruction data from ultra-low dose 3D helical computed tomography (CT). This differs from MRI several regards, but it is significant in case that we can generate different instances of Equation 4 by selecting different in Equation 1. Minimizing Equation+ 4 without giving w explicitely would amount to generating a posterior distri- bution from data y acquired when using any possible mask w sampled from a random vector Z with a possibly unknown distribution. This is why the approach of (Adler & Öktem, 2018 88 minimizes over the distance for observation in Equation and mask designs approaches consider minimization over original data (cf., equation 2).
Uncertainty estimation. Once the generator has been trained, we can sample from Go. (Z, yw) which approximates Pxly Let (&, = Ge (Z;, be samples of the posterior Pxly., where Z; are iid samples from Z and n, and the number of samples taken. Then, in (Adler & Öktem, 2018), the ground truth image x can be estimated by the empirical point-wise mean of these samples, namely X 1
R, Di=1 &, The corresponding empirical point-wise variance can be defined x² & X 22 where denotes the modulus.
Equally, one can estimate the ground truth Fourier spectrum Fx using the empirical average es- timator Fx. The empirical point-wise variance in the Fourier space can be obtained of -- Th [F FX12. This feature is specific to generative models, as getting samples from Pxiy_ allows to transform these to different domain, enabling to have simultaneous variance es- timates in both the image and Fourier spaces. This is not possible with methods that only provide point-wise estimates of the mean and the variance in image space, such as the one used by Zhang et al. (2019) or the direct estimation of Adler & Öktem (2018).
4.2 UNCERTAINTY DRIVEN SAMPLING
Due to the ability of constructing estimates of both the spatial and Fourier space pixel-wise vari- ances of Px y the approach of (Adler & Öktem, 2018) can be leveraged produce both fixed and adaptive sampling patterns by acquiring the frequencies with the highest empirical pixel-wise variance in the Fourier domain. As we constrained ourselves acquiring full lines in the Fourier domain, we will consider the total estimated variance in the Fourier space along the i-th line &; E S and define
u (yw)
Σ (x)) jE
Note that u. ID (yw) N contains the variances of the N possible lines in the Fourier domain. We will refer ID (yw) as the aggregated variance (along the z-dimension in Figure TODO).
5

---

Under review as a conference paper at ICLR 2020
Fixed sampling. Using the aggregated variance loss function, we can reformulate Equation as
Σ P
min
Exx
WEA
where samples {x)m= are obtained from an unknown prior distribution Px. and Y contains partial information on X through the model 1.
In practice, as Px is not available, one seeks to solve the empirical risk minimization (ERM) minweA m -= Ex (ul'y_... Fx.]] The aggregated point-wise variance of the posterior can be seen as a cost that one seeks to minimize on a training set, and consequently, can it can replace traditional cost functions such as the é 22-norm in most fixed sampling optimization method.
Adaptive sampling. Ideally, we aim at making a series of sampling decisions U1,..., eT to mini- mize the total final uncertainty for given ground truth image x once our sampling budget lines is spent, i.e., for each t,
min
u
ID (Ywz),
(8)
where Wr = UT=11 ut. Due to causality, we do not have access to this final posterior, or even the posterior of the mask which will result from choosing the next innovation u/. We can only make use of the partial observations You, corresponding the partial masks W/ = - us up the time step t. We choose to adopt a greedy approach to approximately solve
min Σ u (ywrue,) at each time step E
by simply choosing as vt the line i with the largest aggregated variance u!!- The overall flow is then: at each time t, (i) observe Yu,, (ii) select the line 02 = Us, where i* = argmax, u (y,.), (iii) update w = We U o2 and (iv) iterate until the cardinality constraint is met.
As no assumptions are made on the underlying distribution of the posterior, this application is only made possible by leveraging a generative framework that can estimate the posterior all sampling rates considered for widely different mask designs. It is rendered tractable by the fact that even two samples from the posterior allow to construct empirical variance estimate that can efficiently drive the sampling procedure.
5 IMPLEMENTATION
Training data. The data set used in the first three experiments (subsections) below consists of a proprietary dataset of 2D TI-weighted brain scans. In our experiments, use 100 slices of sizes 256x256 from five such subjects, 20 per subject. Three subjects (60 slices) were used for training the network, two subjects (30 slices) for testing. The data were then massively augmented with both rigid transformations and clastic deformations to counter overfitting as our dataset is very small, following the recommendations of (Ronneberger et al., 2015; Schlemper et al., 2018). Exact details on the dataset and augmentation methods used can be found in Appendix A.
Architecture For posterior sampling, we used the same diseriminator architecture as described in (Adler & Öktem, 2018). For the conditional generator, we used the cascading network of (Schlemper et al., 2018), where the data-consistency layer enforced perfect consistency. Perfect data consistency means that at the end of each block, one replaces the reconstructed value with the corresponding measured value in the Fourier space. This ensures that the reconstruction is consistent with the ob- servations where measurements were acquired. We used 3 CNN blocks, where each block contained 5 convolutional layers followed by ReLu.
As our data are complex, split the real and imaginary part as two channels and add two channels of Gaussian white noise to the conditional generator.
Training. We use the same loss as in (Adler & Öktem, 2018). The loss is reproduced in Ap- pendix A.4 for completeness. We use Adam (Kingma & Ba. 2014) with 81 = 0.5, B2 = 0.9, and learning rate 2 10-1 as in Adler & Öktem (2018) although do not use noisy linear cosine decay

---

Under review as a conference paper at ICLR 2020
out of simplicity. The model is trained for 10Sh backpropagations, which was chosen adhoc to account for the fact there are combinatorial numbers of masks being observed for each image (our reference point Adler & Öktem (2018) uses 5 10 for larger dataset).. For every 5 iterations, the generator was trained once and the discriminator was trained four times. In order to allow calculating the posterior throughout the sampling process, we generate observations of subsampled images various rates by randomly generating horizontal Cartesian masks for sampling rates E [0.025, 0.5], as described in detail Appendix A.2.
Metries. We will use mean squared-error (MSE), structural similarity (SSIM) (Wang et al., 2004) as well as the posterior variance for comparisons. MSE and SSIM are computed between the re- constructed image and the corresponding original, ground truth image. The posterior variance is estimated through the pixel-wise empirical variance estimate, and is averaged on the whole image to produce single scalar. This metric does not require reference.
6 EXPERIMENTS
Throughout our experiments, we use the empirical mean obtained from two posterior samples, as well the corresponding empirical standard deviation. We show exhaustively in appendix that while using 10 samples from the posterior improves the quality of reconstruction, it is sufficient to use the variance estimate from posterior samples the CLUDAS method.
6.1 CONSISTENCY OF THE UNCERTAINTY ESTIMATE
As can be Figure 1, the average MSE correlates well with the average posterior variance, both in image space and in Fourier space, which suggests that the posterior variance could serve as an ap- proximate MSE oracle. While training was only performed in the image domain, the generated samples have consistency both in Fourier and in image space, showing that the uncertainty-bart based approach does provide meaningful information on the error in the reconstruction. The consistency Fourier space crucial for the sampling procedure, as our sampling method leverages the vari- ance estimates in Fourier space, while the consistency in image space gives valuable information to interpret the reconstructed data, which are important for clinicians.
B00175
Fouttee sqaees
B00193
000135
Avcroge MSS in
0.00755
0.0004
600099
800033
0.3030
00302
0.0004
0.0000
0006
0.0010
(1148)
011011
030012
039023
CU0004
0.0006 BC006
Avenng posteciar varianze in spare
Average postesior vatiance Fourier
(b)
Figure 1: Image (left) (resp. Fourier (right)) space MSE against image (resp. Fourier) space empirical posterior variance constructed with and 50 posterior samples respectively. The coordinate of each point given by the image (resp. Fourier) space MSE averaged over reconstructed image and the pixel-wise image (resp. Fourier) space posterior variance averaged over this reconstructed image. The black lines represent the location obtained by averaging over a sampling rate over the whole testing set, with steps of 2.5% sampling rate. The light red and blue lines are the example of trajectory for given image of the testing when increasing the sampling rate.
6.2 RECONSTRUCTION QUALITY
In order to assess the reconstruction obtained by the adaptive masks, we define closed loop oracle MSE driven adaptive sampling method (CLOMDAS), which leverages MSE instead of uncertainty at inference time. While CLOMDAS not feasible in practice, it remains an interesting baseline showing how the mask design could be improved by having access the actual MSE at testing time.

---

Under review as a conference paper at ICLR 2020
Figure shows how the CLUDAS method compares against the CLOMDAS method sample from the testing set. CLUDAS is competitive with CLOMDAS at every sampling rate considered, without having access any oracle information. This behaviour is consistently observed on the whole testing set, as shown in Tables below.
10% sampling
20% sampling
30% sampling
CLOMDAS Ground truth
CLUDAS
CLOMDAS
CLUDAS
CLOMDAS
CLUDAS
Recon
Vac.
Var
Va
Var.
Var.
Variance
MSR 1.41
XSE
XIS
MRSE CLOA
MSR NL.7
U-masks Masss
MSE
7.om
Comparison of reconstruction quality and variance estimation quality for CLUDAS (U-masks), well the CLOMDAS (MSE-masks) for different sampling rates. The zoomed-in data taken the location highlighted by the yellow square upper- right image. The MSE-/U-mase show the evolution of the masks with increasing sampling rate (x-axis). The data are averaged on 2 testing samples
Figure 2:
6.3 COMPARISON WITH OTHER METHODS
We compare method to the following
Learning based compressed sensing (LBC) (Gözcü et al., 2018; Sanchez et al., 2019):
This method incrementally builds up w by computing an expected improvement of loss / at each step. This expected improvement is simply obtained by searching which line will add the largest improvement the next step, and once it has been found, permanently added to the mask. Then, the algorithm proceeds until the cardinality constraint [w] is met. When trained with MSE, we will refer to the method as LBC-M, and when trained to minimize variance, we will refer to it as LBC-V
Vellagoundar & Machireddy (2015): This method proposed the simple heuristic ap- proach of (i) constructing PDF from training data and (ii) sampling at random from the obtained PDF. Our implementation used the spectrum of the whole averaged training set for the PDF.
We were not able to compare our method to the closed loop method of Zhang et al. (2019), since their code is not being publicly available the time of writing.
When comparing the performance of reconstruction of different mask designing methods on the modified generator of Adler and Öktem, we observe that the heuristic baseline Vellagoundar & Machireddy (2015) performs significantly worse at any sampling rate and for any metric. This is not surprising, as this method simply samples art random from a constructed PDF. Comparing the variations of the LBC methods, notice that increasing the number of averaged samples during the training phase of the mask is translated into a uniform improvment of the performance for any sampling rate. This is more exhaustively diseussed in Appendix B Focusing on the LBC methods
8

---

Under review as a conference paper at ICLR 2020
Metric
MSE x x10
Variance x x10
SSIM
Sampling rate
0.1
0.2
0.3
0.1
0.2
0.3
0.1
0.2
0.3
Vellagoundar (2)
3.73
1.34
0.47
1.12
0.59
0.30
0.67
0.80
0.87
LBC-V(2)
2.69
0.76
0.36
0.8
0.44
0.25
0.74
0.83
0.88
LBC-V(10)
1.68
0.55
0.32
0.65
0.36
0.22
0.77
0.86
0.89
LBC-M(2)
1.64
0.51
0.28
0.70
0.35
0.20
0.76
0.87
0.91
LBC-M(10)
1.43
0.50
0.28
0.63
0.34
0.20
0.79
0.87
0.91
CLUDAS(2) (Ours)
1.5
0.51
0.26
0.58
0.32
0.19
0.80
0.89
0.92
CLOMDAS(2)
1.39
0.49
0.26
0.61
0.34
0.21
0.81
0.90
0.93
Table MSE scaled by 10², posterior variance scaled by 10 and SSIM on test data for different undersampling rates and mask design algorithms. The reconstruction is computed as the average of 2 samples. The lowest value (highetst for SSIM) for each undersampling rate is in blue. The value in parentheses corresponds to the number of samples used to compute the average.
trained with 10 posterior sample averaging, we see that the LBC-M method outperforms the LBC-V method. worth noting that the LBCCM uses the full ground truth to build its mask, while the LBC-V method only leverages the variance estimation in Fourier domain. This again highlight the reliability of the uncertainty as a mask designing technique. This conclusion also supported by the CLUDAS approach remaining competitive with the CLOMDAS one, which is infeasible in practice, due to requiring oracle MSE calls at test time. Note also that our method does not require the heavy computational burden of generating the sampling mask ahead of time, and can immediately be used on-the-fly fly after training.
LBC-V
LBC-M
The CLUDAS method is the most effective at reducing pos- terior variance, even if LBC- and CLOMDAS((2) are close runner-ups. More surprisingly, our CLUDAS method is found to yield the best SSIM performance, a metric designed match the human perception of quality better than MSE.
10 samples samples
7 DISCUSSION AND FUTURE WORKS
Posterior distribution for continuum of sampling rates. Successfully modelling the continuum of sampling rates stems from the fact the these inverse problems depend on each other in a highly structured and regular fashion. This enabled us to successfully demonstrate for the first time that principled Bayesian approach for a closed loop mask optimization with rigorous variance estimates is feasible.
Figure 3: Comparison fixed masks obtained by the learning- based method of Gözcü et al. (2018). The horizontal axis shows the mask growing elc- ments are iteratively added
Generically trained generative reconstruction method. The current generative model was trained in generic fashion and not specifically to optimize the quality of masks designed through it. The ability for designing masks stems purely from training it as a rigorous Bayesian modelling of the continuum of inverse problems. This allows the posterior to be conditioned on incrementally collected informa- tion in closed loop fashion. However, our method could easily be incorporated in a reinforcement leamnin--bsee based framework aimed at jointly training reconstruction and sampling such as Jin et al. (2019). This would give the best of both worlds, giving principled uncertainty estimates to the RI. sampler, moving beyond greedy sampling and possibly speeding up the training of the reconstruc- tion method by focusing regions with less reliable varaince estimates instead of using masks sampled from distributions as in this work.
Limitations of the posterior estimation. We leveraged the approach of Adler & Öktem (2018), which the first of its kind to allow to construct posterior estimator from samples of the joint distribution. While it works well empirically, the authors did not provide any analysis guarantees on how well the generator captures the tails of the posterior distribution. Our observations suggest that unusual images, i.e. far away from the mean of the learned distribution might not be accurately

---

Under review as a conference paper at ICLR 2020
captured, i.e. the estimated variance is lower than expected. This could be due to the limited training data available, but might also an artifact of the cWGAN approach which tends to struggle with capturing weaker modes of their distributions. Specifically, the problematic examples might be an indication that while the loss shown in eq. (11) avoids mode collapse, there might still be some "mode deflation" leading to the network underestimating the diversity of the data distribution.
Adaptive vs fixed sampling. Adaptive and fixed sampling methods both have advantages and lim- itations from practical perspective. The main advantages of deployment of the obtained mask, as it simply needs to be programmed into a scanner. We also have a simple generalization bound of the obtained mask, relying on a simple application of Hoeffd- ing's s inequality. In contrast, adaptive methods are currently difficult to deploy it on scanners, as it would require hardware capable of running neural network guiding the sampling procedure. They are also harder to train and currently lack rigorous reliability guarantees. However, if successfully trained and deployed they avoid rigid assumptions about the problem and are able incorporate partial data into the acquisition process, which in turn leads improved performance. In this work, we found that using the adaptive method also increased robustness to the noise the quality criterion used to drive the mask
fixed mask approach lie in the ease
Reliability guarantees beyond the variance estimate presented in this work (i.e. quantifying the uncertainty of the uncertainty estimate) an important future direction of research.
Extensions to the current model. We showed that it is possible to use the uncertainty estimate to design fixed masks as in LBC-V, for settings where we are only interested in using the posterior variance as a natural criterion for reconstruction quality, e.g. masks for MRI systems where incor- porating a neural network scanning time not feasible. In this setting, there are low hanging fruits for improving the method by making use of the available ground truth information, i.e. jointly using posterior variance with other metrics which require ground truth.
REFERENCES
Daniel Abramovitch, Sean Andersson, Lucy Pao, and Georg Schitter. A tutorial on the mechanisms, dynamics, and control of atomic force microscopes. In 2007 American Control Conference, pp. 3488-3502. IEEE, 2007.
Jonas Adler and Ozan Öktem. Deep bayesian inversion. arXiv preprint 1xXiv:1811.059)) 2018
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875. 2017.
Wasserstein gan. arXiv preprint
Cagla Deniz Bahadir, Adrian V Dalca, and Ment R Sabuneu. Learning- based optimization of the under-sampling pattern in MRL. In International Conference on Information Processing in Medi- cal Imaging, pp. 780-792. Springer, 2019.
Claire Boyer, Nicolas Chauffert, Philippe Ciuciu, Jonas Kahn, and Pierre Weiss. On the generation of sampling schemes for magnetic resonance imaging. SIAM Journal Imaging Sciences. 9(4): 2039-2072 2016.
Mark Bydder, David Larkman, and Joseph V Hajnal. Combination of signals from array coils using image-based estimation of coil sensitivity profiles. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 47(3):539-548, 2002.
Emmanuel J Candès, Justin Romberg, and Terence Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans, on Inf. Theory. 52(2): 489-509, 2006.
Nicolas Chauffert, Philippe Ciuciu, Jonas Kahn, and Pierre Weiss. Variable density sampling with continuous trajectories. SIAM Journal on Imaging Sciences, 7(4):1962-1922, 2014.
Joseph Y Cheng, Feiyu Chen, Christopher Sandino, Morteza Mardani, John M Pauly, and Shreyas Vasanawala. Compressed sensing: From research to clinical practice with data-driven learning. arXiv preprint arXiv: 1903.0090 2019.
10

---

Under review as a conference paper at ICLR 2020
David L Donoho. Compressed sensing. IEEE transactions Information Theory, 5284))1289- 1306, 2006.
Li Feng. Robert Grimm, Kai Tobias Block, Hersh Chandarana, Sungheon Kim, Jian Xu, Leon Axel, Daniel K Sodickson, and Ricardo Otazo. Golden-angle radial sparse parallel MRI: Combination of compressed sensing, parallel imaging, and golden-angle radial sampling for fast and flexible dynamic volumetric MRI. Magnetic Resonance in Medicine, 72(3):707-777, 2014.
Baran Gözcüi Rabeeh K. Mahabadi, Yen-Huan Li, Efe Ilick, Tolga Cukur, Jonathan Scarlett, and Volkan Cevher. Learning-basee compressive MRI. IEEE Transactions on Medical Imaging, 2018.
Baran Gözcü, Thomas Sanchez, and Volkan Cevher. Rethinking sampling in parallel MRI: A data- driven approach. In 27th European Signal Processing Conference (EUSIPCO), 2019.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in neural information processing systems, pp-
5767-5777, 2017.
Justin P Haldar and Daeun Kim. Oedipus: An experiment design framework for sparsity-constrained mri. IEEE transactions on medical imaging, 2019.
Kerstin Hammenik, Teresa Klatzer, Erich Kobler, Michael P Recht, Daniel K Sodickson, Thomas Pock, and Florian Knoll. Learning a variational network for reconstruction of accelerated mri data. Magnetic resonance in medicine, 79(6):3055-3071) 2018.
Oren N Jaspan, Roman Fleysher, and Michael L Lipton. Compressed sensing mri: a review of the clinical literature. The British journal of radiology, 88(1056):20150487.22 2015.
Kyong Hwan Jin, Dongwook Lee, and Jong Chul Ye. A general framework for compressed sensing and parallel MRI using annihilating filter based low-rank Hankel matrix. JEEE Transactions Computational Imaging, 2(4):480 495,, 2016.
Kyong Hwan Jin, Michael Unser, and Kwang Moo Yi. Self-supervised deep active accelerated MRI. arXiv preprint arXiv:1901.00 1et.:0941999997 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6988. 2014.
Florian Knoll, Christian Clason, Clemens Diwoky, and Rudolf Stollberger. Adapted random sam- pling patterns for accelerated MRI. Magnetic resonance materials in physics, biology and medicine, 24(1):43-50, 2011.
Libor Kovarik, A Stevens, A Liyu, and Nigel D Browning. Implementing an accurate and rapid sparse sampling approach for low-dose atomic resolution stem imaging. Applied Physies Letters, 109(16):164102, 2016.
Carole Lazarus, Pierre Weiss, Nicolas Chauffert, Franck Mauconduit, Michel Bottlaender, Alexan- dre Vignaud, and Philippe Ciuciu. SPARKLING: Novel non-cartesian sampling schemes for accelerated 2D anatomical imaging at using compressed sensing. In 25th annual meeting of the International Society for Magnetic Resonance Imaging, 2017.
Sajan Goud Lingala and Mathews Jacob. Blind compressive sensing dynamic MRI. IEEE transac- tions medical imaging, (2(6):1132-111) 2013.
Michael Lustig, David Donoho, and John M Pauly. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magnetic Resonance in Medicine,
2007.
Michael Lustig. David L Donoho, Juan M Santos, and John M Pauly. Compressed sensing MRI. IEEE signal processing magazine, 25(2):72-82 2008
Matti Malinen, Tomi Huttunen, Jari Kaipio, and Kullervo Hynynen. Scanning path optimization for ultrasound surgery. Physics in Medicine & Biology, 50(15):3473, 2005.
11

---

Under review as a conference paper at ICLR 2020
Ricardo Otazo, Emmanuel Candès, and Daniel K Sodickson. Low-rank plus sparse matrix decom- position for accelerated dynamic MRI with separation of background and dynamic components. Magnetic Resonance in Medicine, 73(3):1125-111 2015.
Saiprasad Ravishankar and Yoram Bresler. Adaptive sampling design for compressed sensing MRI. In Engineering in Medicine and Biology Society, EMBC, 2011 I Anmual International Conference of the IEEE, pp. 3751-3755 IEEE, 2011a.
Saiprasad Ravishankar and Yoram Bresler. MR image reconstruction from highly undersampled k-space data by dictionary learning. IEEE Transactions on Medical Imaging, 30(5):1028-1048 2011b.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 231-241. Springer, 2015.
Thomas Sanchez, Baran Gözcti, Ruud B van Heeswijk, Efe Ilickk, Tolga Çukur, et al. Scal- able learning-based sampling optimization for compressive dynamic MRI. arXiv preprint arXiv: /900:00000000 2019.
Jo Schlemper, Jose Caballero, Joseph V Hajnal, Anthony Price, and Daniel Rueckert. A deep cas- cade of convolutional neural networks for mr image reconstruction. In International Conference on Information Processing in Medical Imaging, pp. 647-658. Springer, 2017.
Jo Schlemper, Jose Caballero, Joseph V Hajnal, Anthony N Price, and Daniel Rueckert. A deep cas- cade of convolutional neural networks for dynamic MR image reconstruction. IEEE Transactions on Medical Imaging, 37(2):991-503, 2018.
Matthias Seeger, Hannes Nickisch, Rolf Pohmann, and Bernhard Schölkopf. Optimization of k- space trajectories for compressed sensing by bayesian experimental design. Magn. Reson. Med., 63(1):116-126, 2010.
Ferdia Sherry, Martin Benning, Juan Carlos De los Reyes, Martin J Graves, Georg Maierhofer, Guy Williams, Carola-Bibiane Schönlieb, and Matthias J Ehrhardt. Learning the sampling pattern for MRI. arXiv preprint 1r0iv:1906. 087504 2019.
Patrice Y Simard, David Steinkraus, John C Platt, al. Best practices for convolutional neural networks applied to visual document analysis. In Icdar, volume 3, 2003.
Jaganathan Vellagoundar and Ramasubba Reddy Machireddy. A robust adaptive sampling method for faster acquisition of MR images. Magnetic resonance imaging, 33(5):635-645 2015.
Shanshan Wang, Zhenghang Su, Leslie Ying. Xi Peng, Shun Zhu, Feng Liang, Dagan Feng, and Dong Liang. Accelerating magnetic resonance imaging via deep learning. In 20/6 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp. 514-517. IEEE, 2016.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4:6000- 612. 2004.
Tomer Weiss, Sanketh Vedula, Ortal Senouf, Alex Bronstein, Oleg Michailovich, and Michael Zibulevsky. Learning fast magnetic resonance imaging. arXiv preprint arXiv:1905.0000 1905.09344, 2019.
Shuo Zhang, Kai Tobias Block, and Jens Frahm. Magnetic resonance imaging in real time: advances using radial flash. Journal of Magnetic Resonance Imaging, 31(1):001-109, 2010a.
Shuo Zhang, Martin Uecker, Dirk Voit, Klaus-Dietmar Merboldt, and Jens Frahm. Real-time cardio- vascular magnetic resonance at high temporal resolution: radial FLASH with nonlinear inverse reconstruction. Journal of Cardiovascular Magnetic Resonance, 12(1):39, 2010b.
Zizhao Zhang, Adriana Romero, Matthew J Muckley, Pascal Vincent, Lin Yang. and Michal Drozdzal. Reducing uncertainty in undersampled MRI reconstruction with active acquisition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2049- 2058, 2019.
12

---

Under review as a conference paper at ICLR 2020
A IMPLEMENTATION DETAILS
A.1 TRAINING DATA
The data set used in all experiments consists of 2D Te-weighted brain seans of seven healthy sub- jects, which were scanned with FLASH pulse sequence and a 12-channel receive-only head coil. In our experiments, we use 20 slices of sizes 256x256 from five such subjects, for a total of 100 slices. We use three subjects (60 slices) for training, two subjects (40 slices) for testing. Data from individual coils was processed via a complex linear combination, where coil sensitivities were esti- mated from 8x8 8 central calibration region of Fourier space Bydder al. (2002). The acquisition used a field of view (FOV) of 220 x 220 mm? and a resolution of 0.9 x 0.7 mm? The slice thickness was 4.0 mm. The imaging protocol comprised flip angle of 70%, TR/TE of 250.0/2.46 ms, with a sean time of 2 minutes and 10 seconds. Following the recommendations of (Ronneberger et al., 2015: Schlemper al., 2018), we then massively augmented the dataset counter overfitting.
We apply both rigid transformations and elastic deformations. Specifically, at training time, each image was dynamically augmented with randomly applied translation of + pixels along x- and y-axes, rotations of [0, 2,r), reflection along the æ-axis with 50% probability. We also apply elastic deformations using the implementation in Simard al. (2003) with a E [0, 40) and [5, 8).
A.2 GENERATING OBSERVATIONS AT VARIOUS RATES OF SUBSAMPLING
We also dynamically generate horizontal Cartesian masks from sampling rates [0.025, 0.5) by randomly selecting lines following suitably deformed Gaussians (see fig. 4).
Original Gaussian
Rescale
Sample center Final renormalization
Figure 4: Gaussian mask design used during training. We start with a plain Gaussian, rescale it to ensure selection in the tails.always sample the 4 highest energy lines in the center of the Fourier space, and finally renormalize and follow the modified Gaussian to sample the remaining lines.
This generation of training masks biased towards the lower end of the frequency spectrum, and also does not consider extreme acceleration rates beyond 0.025. Ideally, would use fully dom subsampling across the whole range of subsampling rates, ensuring equally reliable variance estimates. In practice, extreme acceleration rates unlikely be used and most of the information is found in the lower frequencies, meaning these would almost surely be selected first. The present method represents a reasonable tradeoff's between generalization across frequencies and training time.
A.3 ARCHITECTURE
For posterior sampling, used the same discriminator architecture described in (Adler & Öktem, 2018). For the conditional generator, we used the cascading network (Schlemper et al., 2018), where the data-consistency layer enforced perfect consistency. We used CNN blocks, where each block contained convolutional layers followed by ReLu.
As our data complex, split the real and imaginary part two channels and add two channels of Gaussian white noise to the conditional generator. This also means that the discriminator of (Adler & Öktem, 2018) needs to be adapted to 6 input channels instead of 3.
13

---

Under review as a conference paper at ICLR 2020
A.4 LOSS FUNCTION
The conditional WGAN must have its discriminator and generator trained alternatively, and Adler and Öktem proposed novel discriminator loss that empirically avoids mode collapse. This discrim- inator takes three inputs instead of two in Equation 5, and is the one that we use in practice in the experiments. The general GAN loss reads
Lw(0, -xx, Zi Z.
Do (Gn GaZz, Y),y)
(10)
( (L Y), (), + (Z Y), x))
- Do
Then, for a fixed 0, the discriminator loss can be formulated as
LD(0) Lw (0, + g0 (0, Φ) -10-3 Lanin (0, Φ) 10 6
(11)
where, Lygrdl is the gradient penalty term for the I-Lipschitz constraint introduced in Guirajani al. (2017). The drift loss is used to stabilize training. to prevent the discriminator from being shifted large values, as its performance is invariant constant shifts. If the discriminator has large constant, the overall loss (1I) will not be influenced by the constant, so the drift loss penalizes large the expected squared norm of the discriminator.
For the generator, given fixed Φ the loss defined as
(0) -E Y)~Px, [D (Z), Y), GZ Y), Y) - 10 11011
(12)
Z1 Zgg-r
We refer the reader the Appendix D.3 of (Adler Öktem, 2018) for the full discussion on the loss function.
B GREEDY VS. ADAPTIVE
As can be seen in Figure 7, in the fixed mask (greedy) setting we require ten samples from the posterior before the uncertainty estimation yields competitive masks, while in the adaptive setting two samples suffice. The reason for this can be understood from considering the decision process and information flow each algorithm, visualized in fig. 5. The adaptive algorithm uses only single uncertainty estimate to make decision, as it aggregates the estimate for each candidate u!" line in the Fourier space from a single pointwise uncertainty estimate. Due to the online nature the sampling, if this estimate overshot undershot, the algorithm will receive immediate feedback and can so revert to the mean, making it robust against a noisy estimator. In contrast to this, the greedy algorithm a) requires a separate uncertainty estimate for each candidate (since want to look into the future and choose the candidate line which yielded the greatest improvement) and b) is performed on the training set only and then fixed. This means there good chance for noisy estimator to spike in the uncertainty estimate and give the illusion of large improvements in uncertainty during the precomputation of the mask, and no way to compensate for mistakes test time. This means we need much more reliable estimator, increasing the number of posterior samples required.
14

---

Under review as a conference paper at ICLR 2020
CLUDAS acquisition decision (closed loop)
LBC sampling decision (open loop)
cand
candl
past
current
cand2
past
current
cand2
loss Levaluation to take decision
cand3
cand3
The adaptive sampling method (left) uses the loss heuristic é (MSE, uncertainty) currently ob- served information prior the observation and then choosing based this. In contrast to this, the greedy method wants to use largest expected improvement in l so has to evaluate in each possible future, effectively deciding a posteriori based on which yielded the actual greatest improvement.
Figure 5:
C ADDITIONAL RESULTS
C.I EFFECT OF INCREASING THE NUMBER OF SAMPLES IN THE AVERAGE
Comparing adaptive baselines with 10 averaged samples Figure 6, we see that the posterior vari- ance magnitude is not affected by the increase in the number of samples, while MSE is simply shifted almost uniformly for each data. While the larger number samples might affect how the variance is distributed on the image, it does not affect too much the variance obtained out of the sampling procedure.
Tables 2, and 4 show the second experiment reproduced with averages computed with 10 posterior samples. The results are consistent with those Section 6.3.
Velapeezder
Vellapedda
CLUDAS
(1))
Will Vilegandda (90)
Velogardes 1900
3052
10'
30
Sampling
Sampling
Figure 6: MSE (left) and image posterior variance (right) averaged the whole testing for different adaptive methods, average respectively on samples from the posterior distribution.
C.2 SUPPLEMENTARY RESULTS
Figure shows the performance across all sampling rates of the methods considered in the main paper.
15

---

Under review as a conference paper at ICLR 2020
Sampling rate
0.05
0.1
0.15
0.2
0.25
0.3
Vellagoundar (10)
10.39
3.30
1.68
1.34
0.70
0.47
LBC-V (2)
8.98
3.48
1.63
1.00
0.58
0.34
LBC-V (10)
4.96
2.15
1.00
0.54
0.38
0.27
LBC-M (2)
4.26
1.45
0.76
0.43
0.3
0.23
LBC-M (10)
4.44
1.26
0.68
0.42
0.29
0.23
CLUDAS (10)
3.8
1.34
0.78
0.43
0.29
0.22
CLUDAS (10)
3.29
1.18
0.65
0.41
0.29
0.22
Table 2: MSE scaled by 10 on test data for different undersampling rates and mask design algorithms. The reconstruction is computed as the average of 10 samples. The lowest MSE for each undersampling rate is in bold.
Sampling rate
0.05
0.1
0.15
0.2
0.25
0.3
Vellagoundar (10)
0.55
0.70
0.77
0.82
0.86
0.89
LBC-V(2)
0.55
0.7
0.78
0.82
0.86
0.89
LBC-V(I0)
0.66
0.76
0.83
0.87
0.88
0.9
LBC-M(2)
0.66
0.79
0.84
0.89
0.91
0.92
LBC-M(10)
0.67
0.81
0.85
0.88
0.91
0.92
CLUDAS(10)
0.69
0.82
0.87
0.90
0.92
0.93
CLOMDAS(10)
0.71
0.83
0.88
0.91
0.93
0.94
Table 3: SSIM test data for different undersampling rates and mask design algorithms. The reconstruction computed as the average 10 samples. The highest SSIM each undersampling rate is bold.
Sampling rate
0.05
0.1
0.15
0.2
0.25
0.3
Vellagoundar (10)
1.60
1.11
0.81
0.58
0.41
0.30
LBC-V(2)
1.47
0.95
0.68
0.54
0.40
0.27
LBC-V(10)
1.02
0.74
0.54
0.38
0.30
0.22
LBC-M(2)
1.20
0.70
0.51
0.35
0.25
0.20
LBC-M(10)
1.12
0.63
0.45
0.34
0.25
0.20
CLUDAS(IO)
0.90
0.58
0.42
0.32
0.24
0.19
CLOMDAS(1O)
0.98
0.62
0.46
0.34
0.27
0.21
Table 4: Estimated posterior variance scaled by 10 on test data for different undersampling rates and mask design algorithms. The reconstruction is computed as the average of 10 samples. The lowest estimated variance for each undersampling rate in bold.
16

---

Under review as a conference paper at ICLR 2020
LICAE
LUCS (00)
LICAU000
LBCSU
LBC-6GI
LBCASTO)
LBC-M00)
LBCMS2
CLORIDAS
CLONDASHE
Vellagasder
Vellgeasshe (.0)
1086
MSR
10
0.1
0.5
0.1
0.2
0.1
Sampling rate
Sampling rate
331
6.7
9811
60-
LOUANN
LBC-U (1))
0.4
LBC-N15
[BCMM)
LBC-312
0.2
CLATDABLIS
CLOMDANE2
CLOMMAS(1O)
Sampling
Sampling
LBC-U
LBC M(2)
LBC-3000
LBCA372
CEATDASOS
CIATDASUO)
Posteros 10
(T.OMMAN77)
10
CLAMBASIIC
Posteriu variance
Sampling rate
Sampling rate
(a)
(b)
Figure 7: Performance baselines for sampling rates considered MSE, SSIM Posterior variance
for (a) 2 averaged samples, (b) 10 averaged samples. The results are averaged across the whole testing set and several runs of each method was done to get error bars.
17


Reviews:
[{"review_id": "rJx9Am1OYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an active data acquisition framework for magnectic resonance imaging. A generative adversarial network is used to estimate the posterior distribution of the latent MRI image in a closed-loop and greedy manner where the uncertainty of the posterior image is used to guide the process.\n\nThe paper is well written and easy to follow. The main contribution seems to be the combination of deep Bayesian inversion in Adler & Oktem, 2018 with an uncertainty driven sampling framework. Using uncertainty to drive data acquisition and exploration is not a new idea; the concept has been applied to reinforcement learning, active learning, Bayesian optimisation, as instantiations of a broad class of methods in experimental design. The experimental results suggest that the technique can reduce the amount of time required to obtain good quality images from MRI scans which can potentially have a big financial impact. The technique is compared to several variants of compressed sensing approaches demonstrating superior performance. \n\nMy main concerns with the paper are:\n\n1. The key idea of using uncertainty to guide sampling was also the main concept in Zhang et al. 2019. This submitted paper highlights differences in the models but does not provide an experimental comparison. Since both papers share the same concepts, this reviewer considers that a comparison is critical.\n\n2. Deep Bayesian inversion approximates the posterior distribution by minimising the Wasserstein distance between the posterior  and a parametrised generator. I find the idea potentially powerful, with the advantage of learning a generative model as well, but wonder how this compares in theory and in practice to simpler stochastic variational inference and modern Hamiltonian MCMC. The min-max formulation is notoriously difficult to optmise and might lead to many local optima and instabilities.\n\n3. Given the complexity of learning GANs and the sensitivity to initialization, results should contain more information such as the std of the MSE for several runs of the algorithm.    \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJxsl2aa9r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a method for accelerating MRI scans by proposing lines in k-space to acquire next. The proposals are based on posterior uncertainty estimates obtained from GAN-based reconstructions from parts of the k-space acquired thus far. The authors address an interesting and important problem of speeding up MRI scans and thus improving the subject's experience. The proposed method achieves better posterior uncertainty and SSIM scores than competing methods.\n\n\n\nWhile the paper considers an important problem and takes a novel approach to solve it (using a GAN generative model to estimate uncertainty), I found that it may be particularly inaccessible to non-experts in the field of MRI image processing. Furthermore, several important methodology-related questions remain unanswered in the paper; and the experiments offered in the paper are insufficient for convincingly arguing the author’s claims.\n\nSpecifically, it is unclear whether GANs with their mode dropping behaviour are the right model choice for proposing reconstructions - they are likely to drop modes and by extension - yield overconfident uncertainty estimates. This can be expected to be particularly problematic for scans of images that differ from the training distribution, and is exacerbated by the fact that authors train the model only on scans from healthy subjects. Furthermore, because of the the GANs propensity to drop modes, it is also unclear whether the posterior variance numbers reported in the paper are directly comparable between the methods.\n\nThe method used for obtaining the uncertainty estimates from GAN samples implicitly makes the assumption that reconstructions follow a Guassian distribution with a diagonal covariance. This assumption is also made in a competing method of Zhang et. al (2019) that the authors do not compare against, and claim to improve upon methodologically (i.e. the authors state that the method of Zhang et. al (2019) cannot be used to produce uncertainty estimates in Fourier space). I am not convinced that the authors claims about the method differences are sufficiently substantiated (see more under major comments). And because the methods bear significant similarity to each other, an experimental comparison - which is currently missing - should be carried out (on open datasets that ideally include non-healthy subjects).\n\nFinally, the paper teases fast(er) MRI in the title, but doesn’t touch on this topic in the text. This aspect should of the authors contribution be discussed at length, in particular comparing the Cartesian sampling strategy adopted by the authors to other strategies, as well as evaluating the feasibility of implementing the adaptive sampling strategy in an actual scanner (e.g. can the network be ran fast enough)?\n\n\n==================\nMajor comments:\n==================\n\n1) In the proposed method the authors employ a procedure in which the currently sampled parts y of the k-space are fed to a generator network to obtain n_s reconstructions. These n_s sampled reconstructions are then averaged to obtain the empirical mean and variance, with the latter being used for estimating uncertainty. This procedure is potentially problematic for several reasons:\n\n  a) First, taking the empirical mean and variance of the samples is in fact equivalent to assuming that the reconstructed image follows a Guassian distribution with a diagonal covariance. This is the same assumption the authors argue is not realistic when discussing the work of Zhang et al. (2019) in the end of Section 1. \n\n  b) In case of GANs, which can model multi-modal distributions this uncertainty estimation is even more problematic in cases when the samples originate from different modes. What do the mean and variance represent then?\n\n  c) As the authors highlight in the discussion section of the paper in the paper, GANs are prone to mode collapse. This is also potentially problematic for their estimator - in case of mode collapse their method would underestimate uncertainty. The fact that authors are able to use only two sampled reconstructions to estimate the mean and variance with acceptable accuracy is consistent with the occurrence of mode collapse in their generator. Furthermore, because mode collapse may occur in the author’s model, it is unsurprising that their method yields the smallest posterior variances in Table 1. The authors should provide evidence that mode collapse either does not occur or would not affect these numbers.\n\n  d) Finally, the authors use the empirical mean and sampled reconstructions to obtain the empirical mean and variance in Fourier (k-) space. Specifically, they argue that “This feature is specific to generative models, as getting samples from P_{X|y_\\omega} allows to transform these to a different domain [...] this is not possible with methods that only provide point-wise estimates of the mean and the variance in image space, such as the one used by Zhang et. al (2019)”. I don’t think this is true - Fourier transform is a linear transformation, thus given a mean and a variance in image space it is possible to deduce analytically what the mean and covariance in Fourier space would be. This should be elaborated in the text, and the comparison to Zhang et. al (2019) should thus be extended further.\n\n\n2) The proposed method, CLUDAS, was evaluated against existing methods on a single proprietary dataset consisting of only 100 images from healthy individuals. This is potentially problematic, for several reasons:\n\n  a) Using a proprietary dataset doesn’t allow follow-up works to compare against the author’s method; or for comparing CLUDAS to \n existing methods not considered in the paper; the methods should additionally be compared on a public dataset  and\n\n  b) Applying the method only to a single (small) dataset does not allow for reasoning on how the method behaves in different data regimes. I strongly encourage the authors to apply their method on public datasets, for example on data used in Zhang et. al (2019) - this would then allow for comparing the two methods despite not having access to an implementation of Zhang et. al (2019).\n\n  c) Since the data is used to train the GAN model is obtained from healthy individuals, it’s unclear whether it can be used to acquire data from subject that may potentially have aberrations in their scans - the GAN model would be expected to produce low uncertainty estimates for regions where these aberrations would lie and not propose acquiring parts of the k-space that could be used to resolve these aberrations. For similar reasons using a GAN that potentially drops modes (e.g. scans of unhealthy individuals, for example because they are less common in the training data) is also problematic. The authors should consider evaluating their method on a dataset that contains non-healthy subjects, and investigate the performance of the method when it’s evaluated on healthy subjects, but tested on unhealthy ones.\n\n3) The title of the paper (“[...] for fast MRI”) suggests that the authors aim to accelerate the MRI data acquisition process.\n\n  a) Yet they chose the work with the Carterian sampling (i.e. sampling lines in the k-space parallel to the x-axis), which arguably requires larger sections of the k-space to be sampled before a high-quality reconstruction can be obtained (e.g. see http://mriquestions.com/k-space-trajectories.html). Providing information on how this choice influences the speed of data acquisition (and thus subject’s comfort) is important in order to assess the applicability of the author’s method to real world scenarios. This information should be provided.\n\n  b) The paper does not actually describe how MRI is sped up. Given that the acquisition budget (number of lines acquired in k-space) is fixed, and assuming that time per line is constant, it is unclear where the speed up comes in. More generally, the speed claims / aspect of the proposed method should be discussed in more detail - how is speed measured and evaluated? How does it compare to non-Cartesian sampling approaches.\n\n  c) Finally, it’s unclear whether neural network inference can be made fast enough to allow for a real world application of the adaptive CLUDAS sampling - can the data be transferred fast enough from the scanner to do inference and propose the next line to scan without causing delays in the scanning process? This should be discussed in the paper.\n\n4) Currently, the paper places a strong expectation of knowing about MRI and being familiar with MRI-specific terminology on the reader. This makes the work substantially less accessible to a wide audience with machine learning expertise as the common denominator. The authors should take steps towards making the text more accessible to non-(MRI) expert audience, for example by introducing some of the basic knowledge (e.g. the data acquisition and reconstruction processes in MRI) early on, departing from MRI-specific jargon (e.g. k-space, lines in k-space) in favour of ML terminology whenever possible, and taking care to define and possibly illustrate (strongly encouraged) the MRI-specific concepts (e.g. the k-space, lines in k-space, sampling masks). Some specific examples the authors should address follow.\n\n  a) The k-space is defined only in passing as being the frequency domain. It’s unclear whether these lines (which correspond to sampling masks) in this domain are parallel to the x-axis. The math (e.g. Equation 1) and Figure 2 suggest that, but it’s not obvious.\nx is referred to both as \"model parameter\" in Section 2 (and Equation 1) as well as the \"ground truth image\" later in the same section. This is confusing, especially because in case of GAN model parameter would typically refer parameters of the generator and discriminator.\n\n  b) SSIM is not defined anywhere, but already used in the abstract.\n\n  c) It could be made more clearly why undersampling is required in case of MRI. E.g. how/why does it correlation with patient comfort.\n\n  d) “K-space sampling” is used already in the introduction, but not really defined.\n\n  e) It’s unclear whether sampling, subsampling and undersampling all refer to the same concept or not.\n\n  f) The term “sampling mask” is already used in the introduction, but not clear what it refers to.\n\n  g) Use of the term “innovation” to refer v_t, which appears to be a one-hot vector marking the newly added line in k-space.\n\n  h) The use of term “sampling decision” to refer to v_i.\n\n  i) Unclear what eta in “Z ~ eta“ in Equation 5 refers to - it is not defined. Later in Section 2.2 it is stated that “z_i are independent samples form Z”. Is this the same as “z_i ~ eta”? If so, why the second layer of notation?\n\n  k) In Introduction “[...] which is not feasible on a real problem without the ground truth available”. Unclear what the ground truth refers to, I assume it’s the ground truth image.\n\n  l) In Section 2.1 data refers to x_i, with i=1,...,m; which I assume are images and thus contain real numbers. Yet Section 3 states “As our data are complex [...]”. This is confusing - is that different data?\n\n  m) Sampling is used ambiguously in the paper - to refer to sampling in the k-space (e.g. sampling masks, sampling decisions v_1,...,v_n) and to refer to sampling reconstructions from the generator. This should be resolved to improve readability of the paper.\n\n  n) Not entirely clear what k-space pixel-wise variances are. And what is the difference between spatial and pixels-wise variances is (Section 2.2).\n\n\n\n==================\nMinor comments:\n==================\n\n1. In the Introduction the authors argue that metrics such as MSE and SSIM “[..] do not align with what clinicians see as valuable.”, yet use these throughout the paper for evaluating and comparing methods. This decision should be explained.\n\n2. In Section 2 (and beyond) images x are described as belonging to subspace C^p of complex numbers. While this is technically true, them to actually belong to the space of R^p. If so, this should be reflected in the text for the benefit of the readers. Furthermore, I don’t think dimensionality p is actually defined anywhere.\n\n3. Compressive / compressed sensing abbreviations CS is defined twice in Section 1.\n\n4. It’s not entirely clear why “The CS-inspired methods shift the burden from acquisition to reconstruction [...]”\n\n5. Incorrect double quotes are used throughout the text (both are right quotes).\n\n6.  In Introduction “[...] yielding an estimator which can be used to drive back the whole sampling process in a closed-loop fashion.” \nit is unclear what it means to “drive back a sampling process”. Perhaps “back” should not be there?\n\n7. In some cases it is unclear what the use of double quotes conveys, e.g.\n\n  a) In Section 2.1 “[...] being the “full” mask [...]”\n\n  b) In Section 2 “[...] ground truth “complete” images [...]”\n\n  c) In Section 5 “[...] these inverse problems “depend” from each other [...]”\n\n  d) Multiple places in Appendix B.\n\n8. The convention of using small letters x, y for data samples / instances and capital letters X, Y for random variables could be made explicit.\n\n9. Section mostly provides background (rather than theory) and could be named accordingly.\n\n10. In Section 2.1, t refers to “time”. It may be clear if it were instead referred to as the step of the sampling process or something similar - the use of “time” to refer to some discrete set of actions can be a little confusing.\n\n11. Section 2.1 refers to “[...] the online reconstruction speed of DL [...]”. This should be explained further - why are deep learning based approaches to reconstruction faster? Does this depend only on having the right hardware accelerators? Also, I don’t think the abbreviation “DL” was introduced.\n\n12. Equation 4 is referred to both as an “Equation” and as a “Problem”.\n\n13. In Equation 6 there is a summation over j from v_i. It is my understanding that v_i is a one-hot vector and it’s unclear what this summation means. Presumably it is the summation over pixels covered by line v_i, but the notation doesn’t convey this. It could be nice to also explain the “1D” superscript in this equation.\n\n14. In Section 2.2 “[...] this is why the approach of (Adler & Oktem, 2018) minimizes over distance for observation in Equation 4 [...]”. I couldn’t follow the part about minimizing over the distance for an observation. Please consider making this more clear.\n\n15. In Equation 8, what does index i run over?\n\n16. Minor typos / textual issues:\n\n  a) In Introduction “[...] of the our estimator [...]”\n\n  b) In Introduction “[...] and show that even using a few samples [...]” -> even when using?\n\n  c) In Section 2.2, after Equation 5 “[...] where t After finding the optimal”.\n\n  d) Inconsistent use of “closed-loop” and “closed loop”.\n\n  e) In multiple places throughout the paper a double space appears to be used instead of a single one.\n\n  f) Section 4.1 “as can be in Figure 1”\n\n  g) Section 4.3 “samples art random”\n\n  h) In Section 5 “[...] “depend” from each other” should be depend on each other?\n\n  i) In Section 5 “[...] we we found that [...]”\n\n17. Unclear what’s meant by “[...] using the aggregated variance as a loss function” in Section 2.2\n\n18. Section to refers to i* (integer scalar) as a line. Previously it was v_i (vector).\n\n19. In Section 2.2 the authors state “Once the generator has been trained until convergence [...]”. The authors optimize a generator in an adversarial fashion. To my knowledge, this training procedure is not guaranteed to converge and would typically oscillate around a stable solution. Could the authors please comment on what they mean by convergence in this case and how they guarantee that the generators they train converge.\n\n20. For the posterior variance results in Table 1 it should be discussed whether all the methods obtain / compute the variances the same way.\n\n21. The “data consistency layer” (Section 3) should be explained briefly. How does it enforce perfect consistency and what’s meant by consistency here?\n\n22. It should be made clear what MSE is calculated between in Figure 1 and Table 1. I assume it’s between ground truth (all frequencies sampled) and reconstructed (only some frequencies samples) images.\n\n23. It’s not entirely clear what’s meant by consistency in Section 4.1\n\n24. What do yellow arrows signify in Figure 2?\n\n25. I don’t think the abbreviation UQ was defined in Figure 2.\n\n26. The masks in Figure 2 are such that is a certain line was chosen at lower sampling rate (left on the x-axis), it would also be chosen at higher sampling rate (right on the x-axis). This is somewhat unexpected since the budget of lines v_i to be acquired differs between sampling rates. Why is there such consistency?\n\n27. In Table 1, in the leftmost column, the number in brackets (number of posterior samples?) should be defined.\n\n28. In Table 1 and Section 4.3: LBC-M and LBC-U - the -M and -U suffixes should be explained. What are the differences between the methods?\n\n29. In Section 4.3: what are the FE methods?\n\n30. Appendix A.3 is mostly a copy-paste from the main text. Unnecessary duplication?\n\n31. Axes in Figure 4 (Appendix A.2) are not labeled or described.\n\n32. Loss in A.4 (Equation) does not match Equation 5. In the former the discriminator takes three arguments instead of two, and the arguments z1 and z2 are not described.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxcSGnntB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an uncertainty driven acquisition for MRI reconstruction. Contrary to most previous approaches (which try to get best reconstruction for a fixed sampling pattern) the method incorporates an adaptive, on-the-fly masking building (which is similar in spirit to Zhang at al. 2019). The measurements to acquire are selected based on variance/uncertainty estimates coming from a conditional GAN model. This is mostly an \"application\" paper that is evaluated on one dataset.\n\nStrengths:\n- The paper studies an interesting problem of adaptive MRI reconstruction\n- The review of MRI reconstruction techniques is well scoped\n\nWeaknesses:\n- The evaluation is rather limited and performed on one, proprietary, relatively small sized dataset\n- Some simple baselines might be missing\n\n\nI like the idea of adaptive sampling in MRI. However, I'd slightly lean towards rejection of the paper. My main concerns are as follows:\n\nThe presentation of the paper could be improved. At the moment, the Theory section describes background information, related work and problem definition as well as the contribution of the paper. Maybe braking the section into related work, background and methodology (where the main contribution is presented) sections would improve the paper readability. \n\nThe paper uses a conditional GAN model (with a discriminator from Adler & Oktem, 2018 and a generator that is based on Schlemper et al. 2018). Making the methodological contribution to be rather limited.  The main difference w.r.t. the previous papers seem to be the last paragraph of  section 2.2 - the empirical variance estimation is performed in Fourier space. \n\nA simple baseline to compare might be to train a Unet-like model (e. g. Schlemper et al 2018) with a Gaussian observation model (outputting a mean and a variance per each pixel) and train it to minimize Gaussian NLL. At the test time, one could simply sample from the Gaussian model instead of taking just the argmax of the output. It might be the case that the assumption of gaussian image might be too simplistic, however, it would be interesting to show it experimentally. Note that when sampling from such model the empirical variance estimation could be performed is the Fourier space too.\n\nThe experimental evaluation is rather limited and the dataset used in the experimental section is small. Adding another dataset would make the paper stronger.\n\nOther comments:\n\nThere is a mention on training dataset and testing dataset -- there is no mention on validation set. How were the hyperparamenters of the conditional GAN selected?\n\nAs acknowledged by the authors, this paper bears several similarities with the work of Zhang at al. 2019. However, the approach is not compared to Zhang et al. Including this comparison would make the paper stronger.\n\nIt is interesting to see that CLUDAS outperforms CLOMDAS in terms of SSIM. If I understand this part properly, CLOMDAS uses ground truth image to estimate MSE. Is it expected that CLUDAS would outperform CLOMDAS? \n\nSection 5, Adaptive vs. fixed mask: \"We also have a simple generalization bound of the obtained mask, relaying on a simple application of Hoeffding's inequality.\" Could the authors add a citation or explain this part in more detail?\n\n\nSome typos:\n\"...we aim make a series...\"\n\".. define an closed-loop...\"\n\"We choose adopt a greedy\"\n\"... we we found that...\" \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylJ7Uoh9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is quite well written and the idea is novel. However, the results are rather weak. The authors present a method to perform adaptive MR compressed sensing, i.e. decide online which readout to sample next. They compare it to an offline learning method where one sampling pattern is optimized for a whole training set, then applied to test data. The offline method performs better in terms of MSE, which is the loss it was trained for, meaning that the authors have not demonstrated a gain in adapting the sampling pattern to individual scans.\n\nThe primary concern with the paper is not with the author’s contribution, but with serious flaws in [Adler 2018] that unfortunately snowball into this one. While the authors in [Adler 2018] do acknowledge issues with learning a variance, they misdiagnose the problem as mode collapse. Mode collapse is an optimization problem, where the training set contains variability but the generator fails to learn it due to the lack of an encoder. That is not the case here: all the variability of the training set is encapsulated in y, and for each y the target empirical posterior distribution is a Dirac. This is very similar to the calibration problem in classification [1], where classifiers become overconfident because they are trained to always output 0 or 1. If the generator does not learn a Dirac, it can only be because of regularization (either explicit or implicit in the model architecture) or optimization failure (either involuntary or voluntary with early stopping.) Tweaking the loss as advocated in [Adler 2018] does not fundamentally change the problem as long as the loss is minimal at the target empirical posterior. It may change the dynamic behavior and result in posteriors with more variance when combined with early stopping, but those variances are not calibrated, i.e. they have not been trained to match the variances of the true continuous posteriors. In order to learn the variances, one would have to either provide multiple posterior samples for each y during training (not practical in this case,) or perform some kind of calibration on the validation dataset as in [1], i.e. learn the mean and variance from different data, which effectively uses the network’s interpolation properties as a proxy for true random sampling.\n\nHowever this flaw does not invalidate the practical approach developed in the paper, but it seriously undermines its qualifications as a rigorous, principled, Bayesian approach. It also makes the reporting of posterior variances as final quality metrics pretty much useless since they are not interpretable: does lower variance mean that the generator got better at estimating the missing information, or that it got worse at estimating the true posterior variance? I would suggest to at least remove the variances highlights from Table 1 and Table 4, and maybe scrap the data altogether. The paragraph on posterior estimation should also be updated to represent the whole scope of the problem. \n\nSection 2: Theory: suggest to remove “Without loss of generality”. Due to the known issues with variance estimation, having p(y | x) as a density instead of a Dirac could very well change the behavior of the generator.\n\nSection 2.1 Adaptive masks. The whole first paragraph is somewhat misleading and should be revised. Real-time reconstruction is indeed possible without deep learning, see [2] for example. Furthermore, real-time reconstruction is not nearly fast enough for adaptive sampling. Real-time reconstruction means that reconstructing an image is at least as fast as scanning the whole image, i.e. in the order of 0.1 to 1s., but for adaptive sampling one must reconstruct at least as fast as the time between two successive readouts, i.e. in the order of 1 to 10 ms. Both [Jin 2019] and [Zhang 2019] only showed single-coil offline simulations with no indications of the reconstruction time and so do the authors.\n\nFigure 1: I must be missing something here. How can the image-domain and Fourier-domain figures be different? The Fourier transform being orthogonal, norms and variances should be the same in both domains.\n\n[1] C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, “On Calibration of Modern Neural Networks”, ICML 2017 70:1321-1330.\n[2] M. Uecker, S. Zhang, and J. Frahm, “Nonlinear Inverse Reconstruction for Real-Time MRI of the Human Heart Using Undersampled Radial FLASH”, MRM 63:1456-1462 (2010).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The author responses and notes to the AC are acknowledged.  A fourth review was requested because this seemed like a tricky paper to review, given both the technical contribution and the application area.  Overall, the reviewers were all in agreement in terms of score that the paper was just below borderline for acceptance.  They found that the methodology seemed sensible and the application potentially impactful.  However, a common thread was that the paper was hard to follow for non-experts on MRI and the reviewers weren't entirely convinced by the experiments (asking for additional experiments and comparison to Zhang et al.).  The authors comment on the challenge of implementing Zhang is acknowledged and it's unfortunate that cluster issues prevented additional experimental results.  While ICLR certainly accepts application papers and particularly ones with interesting technical contribution in machine learning, given that the reviewers  struggled to follow the paper through the application specific language it does seem like this isn't the right venue for the paper as written.  Thus the recommendation is to reject.  Perhaps a more application specific venue would be a better fit for this work.  Otherwise, making the paper more accessible to the ML audience and providing experiments to justify the methodology beyond the application would make the paper much stronger.
Decision: Reject