Closed loop deep Bayesian inversion:  Uncertainty driven acquisition for fast MRI

This work proposes a closed loop , uncertainty-driven adaptive sampling framework ( CLUDAS ) for accelerating magnetic resonance imaging ( MRI ) via deep Bayesian inversion . By closed loop , we mean that our samples adapt in real-time to the incoming data . To our knowledge , we demonstrate the first generative adversarial network ( GAN ) based framework for posterior estimation over a continuum sampling rates of an inverse problem . We use this estimator to drive the sampling for accelerated MRI . Our numerical evidence demonstrates that the variance estimate strongly correlates with the expected mean squared error ( MSE ) improvement for different acceleration rates even with few posterior samples . Moreover , the resulting masks bring improvements to the state-of-the-art fixed and active mask designing approaches across MSE , posterior variance and structural similarity metric on real undersampled MRI scans . 1 INTRODUCTION . Myriad of applications in control , data processing , and learning—from platform navigation to data mining and from channel estimation to compressive sensing ( CS ) —involve a linear projection of signals or data points into lower-dimensional space . In this dimensionality reduction , the measurement or the sensing matrix determines how much information we acquire per measurement , the ensuing computational ease of processing ( since algorithms use the sensing matrix and its adjoint as subroutines ) , and the recovery guarantees . In a resource constrained setting , these utilities create a Pareto trade-off wherein improving one worsens another . To impact all these fronts simultaneously , adaptive sensing ( or sequential experimental design , active learning , etc . ) aims to close the loop between the data acquisition and the inference , for instance , by exploiting information collected in past samples to adjust the future sampling process . While adaptive procedures promise great improvements over non-adaptive methods , they are too computationally demanding for real-time online response . In the context of magnetic resonance imaging ( MRI ) , the dimensionality reduction process ( i.e. , undersampling in the Fourier domain , often referred to as k-space ) directly correlates with patient comfort , as it results in shorter scanning times . For the last decade , approaches motivated by compressed sensing have enabled successful reconstruction from highly accelerated ( i.e. , subsampled ) data ( see Lustig et al . ( 2007 ) ; Ravishankar & Bresler ( 2011b ) ; Lingala & Jacob ( 2013 ) ; Otazo et al . ( 2015 ) ; Jin et al . ( 2016 ) and references therein ) . While compressed sensing prescribed fully random sampling ( Candès et al. , 2006 ; Donoho , 2006 ) for recovery , most CS-inspired approaches to MRI departed from this paradigm and relied on the heuristics of variable-density sampling ( VDS ) ( Lustig et al. , 2007 ) . There , the sampling pattern ( or mask ) is picked at random from a probability distribution that reasonably imitates the energy distribution in Fourier space , whereas fully random sampling of the Fourier space ignores this important structure in the signal , which leads to practically poor results ( Lustig et al. , 2008 ) . VDS appears as a heuristic middle ground for a sampling pattern to incorporate the structure of energy distribution in Fourier space , while preserving the benefits of incoherent sampling . In VDS , the probability distribution considered has traditionally been parametric ( Lustig et al. , 2007 ; Chauffert et al. , 2014 ; Boyer et al. , 2016 ) or constructed from data ( Knoll et al. , 2011 ; Vellagoundar & Machireddy , 2015 ; Bahadir et al. , 2019 ) . The CS-inspired methods shift the burden from acquisition to reconstruction , as most of these methods are iterative , preventing online reconstruction of accelerated data . This slow reconstruction rendered the problem of optimizing Fourier space sampling prohibitively expensive , so little work was devoted to general method of designing sampling patterns for generic sampling methods ( Gözcü et al. , 2018 ) . In recent years , deep learning applied to MRI enabled high quality reconstruction for unprecedented acceleration rates ( Wang et al. , 2016 ; Schlemper et al. , 2017 ; Hammernik et al. , 2018 ) , as well as near-online reconstruction times . This also led to a freshly renewed interest in the problem of optimizing the Fourier space sampling pattern ( Bahadir et al. , 2019 ; Weiss et al. , 2019 ; Sherry et al. , 2019 ; Jin et al. , 2019 ; Zhang et al. , 2019 ) , although most of the research energy is still focused on developing more efficient reconstruction methods . In its current stage , deep learning applied to MRI suffers from three main drawbacks : ( i ) as mentioned , most of the research energy has been devoted to more efficient reconstruction methods , despite recent results showing that the sampling masks used have a significant effect on the quality of reconstruction ( Gözcü et al. , 2018 ; Gözcü et al. , 2019 ) , ( ii ) assessing the reliability of the prediction of a reconstructed image is difficult for a clinician , due to the black-box nature of deep learning methods and ( iii ) the commonly used metrics for assessing the quality of the reconstruction ( e.g . MSE , PSNR , SSIM ) do not align with what clinicians see as valuable ( Cheng et al. , 2019 ) . A recent work of Adler & Öktem ( 2018 ) successfully demonstrated that a conditional Wasserstein GAN ( cWGAN ) ( Arjovsky et al. , 2017 ) can be used to learn the posterior distribution of images given undersampled measurements in a tractable fashion while only relying on samples from the joint distribution . This result provides a key opportunity to address all three drawbacks in the context of MRI by using the posterior variance of the reconstruction as an uncertainty estimate , which is a more natural criterion for image quality . To this end , we show that a conditional WGAN can be trained on a continuum of inverse problems on various sampling rates , yielding an estimator of the posterior variance in the Fourier domain that can be used to drive the whole sampling process in a closed loop fashion . Despite the model being trained only for reconstruction and not sampling , the resulting variance estimator can reliably be used to guide a closed loop sampling procedure , thus providing patient-adapted sampling masks . In particular , we demonstrate that the generated masks that minimize the uncertainty estimates in an online fashion reach similar reconstruction MSE as compared to the state-of-the-art fixed-mask approaches like Gözcü et al . ( 2018 ) ; Gözcü et al . ( 2019 ) . While these approaches explicitly focus on minimizing MSE , we show that CLUDAS naturally outperforms them in terms of key visual metrics such as SSIM ( Wang et al. , 2004 ) without being trained on these metrics . In addition , we investigate the reliability of our estimator in a wide range of undersampling regimes and show that even when using a few samples from the cWGAN posterior , the variance estimate is reliable enough to be used to drive the design of the mask . This makes it feasible to use CLUDAS in a closed loop adaptive setting , where our approach is competitive with an approach using an MSEoracle on the testing set ( which is not feasible on real problem without the ground truth available ) , and also beats strong open loop adaptive baselines while being easier to train and easier to apply . Contributions .. • We show how to train a cWGAN to generate posterior samples across a continuum of sampling rates ; We solve in a Bayesian fashion not only a single inverse problem , but a continuum of inverse problems . • We demonstrate the first posterior-based mask design method for MRI . • We propose to use the variance of the posterior distribution of images given measurements as quality metric for mask performance . • We show that despite the network being trained as a reconstruction method and not as a sampling method , our adaptive approach CLUDAS is competitive in both settings and a strong contender for being used in active settings as it matches computationally expensive state-of-the-art approaches . Implications . We contend that our uncertainty driven sampling framework can extend to many similar problems where one wants to reduce acquisition times , such as atomic force microscopy ( Abramovitch et al. , 2007 ) , transmission electron microscopy ( Kovarik et al. , 2016 ) , or trajectory optimization for ultrasound acquisitions ( Malinen et al. , 2005 ) . Our results open the door of leveraging Bayesian experimental design methods for designing adaptive sampling patterns in clinical settings and beyond . Related works . We especially want to highlight the work of Zhang et al . ( 2019 ) , which bears several similarities with our method . It is important to note that their approach is not generative , as they only have point estimates of the mean and some learned uncertainty metric . Moreover , they assume the reconstructed image to be normally distributed with a diagonal covariance , a practically unrealistic assumption , which is not required in Adler & Öktem ( 2018 ) . 2 NOTATION AND PROBLEM SETTING . Notation . Throughout this paper , we will refer to vectors as boldface lowercase letters e , x , y , and assume that they correspond to N × N images that are vectorized to a p dimensional space . In particular , we will assume that these vectors belong to some appropriately defined spaces X , Y ⊆ Cp . We use vectors in Cp as MR images are inherently complex . Boldface uppercase letters X , Y will in general refer to random variables ( formally , they are random vectors ) , with the exception of F and Pω which will respectively refer to the discrete Fourier transform operator and the sampling operator , that will be defined below . Finally , we will use ω ∈ O = { 0 , 1 } p to be a p dimensional binary vector , and we will refer to it interchangeably as a sampling , subsampling or undersampling pattern/mask . Finally , we will refer to the distribution of a random variable X as PX , and extend the nottation to conditioned random variables such as PX|y ≡ PX|Y=y . Problem setting . An inverse problem is the task of recovering the ground truth x ∈ X from measurements y ∈ Y . In the case of MRI , we consider the following acquisition model yω = PωFx + e , ( 1 ) where yω ∈ Y corresponds to the measurements obtained with a given sampling mask ω and where the sampling operator ( Pω ) ii = 1 if i ∈ ω , 0 otherwise . ω ⊆ [ p ] : = { 1 , . . . , p } is a set containing the sampled locations . Note that while the ground truth x is an image living in the image space X , the measurements yω live in the Fourier space . The Fourier space is often referred to as k-space in the MRI literature . In this paper , we restrict ourselves to the setting where ω is composed of lines in the Fourier space , also known as Cartesian sampling in the MRI literature1 , and usually constrained by a maximal number n of lines that can be acquired . F denotes the Fourier transform , x ∈ X is the ground truth image and e ∈ Y is a white additive noise . Without loss of generality , neglecting basic sampling effects , such as magnetic field inhomogeneity and spin relaxation , we assume e = 0 in the sequel . As we will be working in a Bayesian framework , we define a random variable X ∼ PX from which ground truth , complete measurements are generated , distributed according to the unknown prior PX . From this we also define the distribution PYω as well as the joint distribution PX , Yω and the posterior PX|Yω , where Yω = PωFX.The posterior is especially of interest as for a fixed yω ∈ Y , PX|yω ( short for PX|Yω=yω ) represents the probability distribution of ground truths that are likely to have generated the observed data yω with a given mask ω .

Summary: The paper proposes an uncertainty driven acquisition for MRI reconstruction. Contrary to most previous approaches (which try to get best reconstruction for a fixed sampling pattern) the method incorporates an adaptive, on-the-fly masking building (which is similar in spirit to Zhang at al. 2019). The measurements to acquire are selected based on variance/uncertainty estimates coming from a conditional GAN model. This is mostly an "application" paper that is evaluated on one dataset.