Closed loop deep Bayesian inversion:  Uncertainty driven acquisition for fast MRI

This work proposes a closed loop , uncertainty-driven adaptive sampling framework ( CLUDAS ) for accelerating magnetic resonance imaging ( MRI ) via deep Bayesian inversion . By closed loop , we mean that our samples adapt in real-time to the incoming data . To our knowledge , we demonstrate the first generative adversarial network ( GAN ) based framework for posterior estimation over a continuum sampling rates of an inverse problem . We use this estimator to drive the sampling for accelerated MRI . Our numerical evidence demonstrates that the variance estimate strongly correlates with the expected mean squared error ( MSE ) improvement for different acceleration rates even with few posterior samples . Moreover , the resulting masks bring improvements to the state-of-the-art fixed and active mask designing approaches across MSE , posterior variance and structural similarity metric on real undersampled MRI scans . 1 INTRODUCTION . Myriad of applications in control , data processing , and learning—from platform navigation to data mining and from channel estimation to compressive sensing ( CS ) —involve a linear projection of signals or data points into lower-dimensional space . In this dimensionality reduction , the measurement or the sensing matrix determines how much information we acquire per measurement , the ensuing computational ease of processing ( since algorithms use the sensing matrix and its adjoint as subroutines ) , and the recovery guarantees . In a resource constrained setting , these utilities create a Pareto trade-off wherein improving one worsens another . To impact all these fronts simultaneously , adaptive sensing ( or sequential experimental design , active learning , etc . ) aims to close the loop between the data acquisition and the inference , for instance , by exploiting information collected in past samples to adjust the future sampling process . While adaptive procedures promise great improvements over non-adaptive methods , they are too computationally demanding for real-time online response . In the context of magnetic resonance imaging ( MRI ) , the dimensionality reduction process ( i.e. , undersampling in the Fourier domain , often referred to as k-space ) directly correlates with patient comfort , as it results in shorter scanning times . For the last decade , approaches motivated by compressed sensing have enabled successful reconstruction from highly accelerated ( i.e. , subsampled ) data ( see Lustig et al . ( 2007 ) ; Ravishankar & Bresler ( 2011b ) ; Lingala & Jacob ( 2013 ) ; Otazo et al . ( 2015 ) ; Jin et al . ( 2016 ) and references therein ) . While compressed sensing prescribed fully random sampling ( Candès et al. , 2006 ; Donoho , 2006 ) for recovery , most CS-inspired approaches to MRI departed from this paradigm and relied on the heuristics of variable-density sampling ( VDS ) ( Lustig et al. , 2007 ) . There , the sampling pattern ( or mask ) is picked at random from a probability distribution that reasonably imitates the energy distribution in Fourier space , whereas fully random sampling of the Fourier space ignores this important structure in the signal , which leads to practically poor results ( Lustig et al. , 2008 ) . VDS appears as a heuristic middle ground for a sampling pattern to incorporate the structure of energy distribution in Fourier space , while preserving the benefits of incoherent sampling . In VDS , the probability distribution considered has traditionally been parametric ( Lustig et al. , 2007 ; Chauffert et al. , 2014 ; Boyer et al. , 2016 ) or constructed from data ( Knoll et al. , 2011 ; Vellagoundar & Machireddy , 2015 ; Bahadir et al. , 2019 ) . The CS-inspired methods shift the burden from acquisition to reconstruction , as most of these methods are iterative , preventing online reconstruction of accelerated data . This slow reconstruction rendered the problem of optimizing Fourier space sampling prohibitively expensive , so little work was devoted to general method of designing sampling patterns for generic sampling methods ( Gözcü et al. , 2018 ) . In recent years , deep learning applied to MRI enabled high quality reconstruction for unprecedented acceleration rates ( Wang et al. , 2016 ; Schlemper et al. , 2017 ; Hammernik et al. , 2018 ) , as well as near-online reconstruction times . This also led to a freshly renewed interest in the problem of optimizing the Fourier space sampling pattern ( Bahadir et al. , 2019 ; Weiss et al. , 2019 ; Sherry et al. , 2019 ; Jin et al. , 2019 ; Zhang et al. , 2019 ) , although most of the research energy is still focused on developing more efficient reconstruction methods . In its current stage , deep learning applied to MRI suffers from three main drawbacks : ( i ) as mentioned , most of the research energy has been devoted to more efficient reconstruction methods , despite recent results showing that the sampling masks used have a significant effect on the quality of reconstruction ( Gözcü et al. , 2018 ; Gözcü et al. , 2019 ) , ( ii ) assessing the reliability of the prediction of a reconstructed image is difficult for a clinician , due to the black-box nature of deep learning methods and ( iii ) the commonly used metrics for assessing the quality of the reconstruction ( e.g . MSE , PSNR , SSIM ) do not align with what clinicians see as valuable ( Cheng et al. , 2019 ) . A recent work of Adler & Öktem ( 2018 ) successfully demonstrated that a conditional Wasserstein GAN ( cWGAN ) ( Arjovsky et al. , 2017 ) can be used to learn the posterior distribution of images given undersampled measurements in a tractable fashion while only relying on samples from the joint distribution . This result provides a key opportunity to address all three drawbacks in the context of MRI by using the posterior variance of the reconstruction as an uncertainty estimate , which is a more natural criterion for image quality . To this end , we show that a conditional WGAN can be trained on a continuum of inverse problems on various sampling rates , yielding an estimator of the posterior variance in the Fourier domain that can be used to drive the whole sampling process in a closed loop fashion . Despite the model being trained only for reconstruction and not sampling , the resulting variance estimator can reliably be used to guide a closed loop sampling procedure , thus providing patient-adapted sampling masks . In particular , we demonstrate that the generated masks that minimize the uncertainty estimates in an online fashion reach similar reconstruction MSE as compared to the state-of-the-art fixed-mask approaches like Gözcü et al . ( 2018 ) ; Gözcü et al . ( 2019 ) . While these approaches explicitly focus on minimizing MSE , we show that CLUDAS naturally outperforms them in terms of key visual metrics such as SSIM ( Wang et al. , 2004 ) without being trained on these metrics . In addition , we investigate the reliability of our estimator in a wide range of undersampling regimes and show that even when using a few samples from the cWGAN posterior , the variance estimate is reliable enough to be used to drive the design of the mask . This makes it feasible to use CLUDAS in a closed loop adaptive setting , where our approach is competitive with an approach using an MSEoracle on the testing set ( which is not feasible on real problem without the ground truth available ) , and also beats strong open loop adaptive baselines while being easier to train and easier to apply . Contributions .. • We show how to train a cWGAN to generate posterior samples across a continuum of sampling rates ; We solve in a Bayesian fashion not only a single inverse problem , but a continuum of inverse problems . • We demonstrate the first posterior-based mask design method for MRI . • We propose to use the variance of the posterior distribution of images given measurements as quality metric for mask performance . • We show that despite the network being trained as a reconstruction method and not as a sampling method , our adaptive approach CLUDAS is competitive in both settings and a strong contender for being used in active settings as it matches computationally expensive state-of-the-art approaches . Implications . We contend that our uncertainty driven sampling framework can extend to many similar problems where one wants to reduce acquisition times , such as atomic force microscopy ( Abramovitch et al. , 2007 ) , transmission electron microscopy ( Kovarik et al. , 2016 ) , or trajectory optimization for ultrasound acquisitions ( Malinen et al. , 2005 ) . Our results open the door of leveraging Bayesian experimental design methods for designing adaptive sampling patterns in clinical settings and beyond . Related works . We especially want to highlight the work of Zhang et al . ( 2019 ) , which bears several similarities with our method . It is important to note that their approach is not generative , as they only have point estimates of the mean and some learned uncertainty metric . Moreover , they assume the reconstructed image to be normally distributed with a diagonal covariance , a practically unrealistic assumption , which is not required in Adler & Öktem ( 2018 ) . 2 NOTATION AND PROBLEM SETTING . Notation . Throughout this paper , we will refer to vectors as boldface lowercase letters e , x , y , and assume that they correspond to N × N images that are vectorized to a p dimensional space . In particular , we will assume that these vectors belong to some appropriately defined spaces X , Y ⊆ Cp . We use vectors in Cp as MR images are inherently complex . Boldface uppercase letters X , Y will in general refer to random variables ( formally , they are random vectors ) , with the exception of F and Pω which will respectively refer to the discrete Fourier transform operator and the sampling operator , that will be defined below . Finally , we will use ω ∈ O = { 0 , 1 } p to be a p dimensional binary vector , and we will refer to it interchangeably as a sampling , subsampling or undersampling pattern/mask . Finally , we will refer to the distribution of a random variable X as PX , and extend the nottation to conditioned random variables such as PX|y ≡ PX|Y=y . Problem setting . An inverse problem is the task of recovering the ground truth x ∈ X from measurements y ∈ Y . In the case of MRI , we consider the following acquisition model yω = PωFx + e , ( 1 ) where yω ∈ Y corresponds to the measurements obtained with a given sampling mask ω and where the sampling operator ( Pω ) ii = 1 if i ∈ ω , 0 otherwise . ω ⊆ [ p ] : = { 1 , . . . , p } is a set containing the sampled locations . Note that while the ground truth x is an image living in the image space X , the measurements yω live in the Fourier space . The Fourier space is often referred to as k-space in the MRI literature . In this paper , we restrict ourselves to the setting where ω is composed of lines in the Fourier space , also known as Cartesian sampling in the MRI literature1 , and usually constrained by a maximal number n of lines that can be acquired . F denotes the Fourier transform , x ∈ X is the ground truth image and e ∈ Y is a white additive noise . Without loss of generality , neglecting basic sampling effects , such as magnetic field inhomogeneity and spin relaxation , we assume e = 0 in the sequel . As we will be working in a Bayesian framework , we define a random variable X ∼ PX from which ground truth , complete measurements are generated , distributed according to the unknown prior PX . From this we also define the distribution PYω as well as the joint distribution PX , Yω and the posterior PX|Yω , where Yω = PωFX.The posterior is especially of interest as for a fixed yω ∈ Y , PX|yω ( short for PX|Yω=yω ) represents the probability distribution of ground truths that are likely to have generated the observed data yω with a given mask ω .

Summary: The paper describes a method for accelerating MRI scans by proposing lines in k-space to acquire next. The proposals are based on posterior uncertainty estimates obtained from GAN-based reconstructions from parts of the k-space acquired thus far. The authors address an interesting and important problem of speeding up MRI scans and thus improving the subject's experience. The proposed method achieves better posterior uncertainty and SSIM scores than competing methods.