On the benefits of deep RL in accelerated MRI sampling

1 INTRODUCTION . Magnetic resonance imaging ( MRI ) is a non-invasive , non-ionizing medical imaging method which has been widely adopted in clinical settings due to its with unmatched quality in soft tissue contrast . However , MRI suffers from long scanning times , which limits patient comfort , imaging quality as well as throughput ( Zbontar et al. , 2019 ) . These issues have been a longstanding research direction in the community , and one prominent way to shorten scanning times is acquiring less measurements and use a reconstruction method to recover the full image , a setting known as accelerated MRI . Spurred by the great gains in reconstruction quality ( Muckley et al. , 2020 ) , recent works have moved away from using generic , non-learning based heuristics , such as variable-density sampling ( VDS ) ( Lustig et al. , 2007 ) to select the measurements to acquire . Instead , there has been a growing interest in strong , learning-based methods that tailor sampling policies to the reconstruction method and achieve even greater accelerations , and as a result , further speeding up imaging . The state-of-the-art approaches rely mostly on the idea of pairing a reconstruction model with a patientadaptive sampling model , where the former estimates a clean image from partial measurements , and the latter selects measurements that it predicts as likely to improve the reconstruction quality . The selection policy can be trained relying on techniques from reinforcement-learning ( Jin et al. , 2019 ; Pineda et al. , 2020 ; Bakker et al. , 2020 ) , heuristics such as a model trying to estimate the Fourier space error of the locations to be acquired ( Zhang et al. , 2019 ) or end to end via the straight-through estimator ( Van Gorp et al. , 2021 ; Yin et al. , 2021 ) . Other approaches choose to directly parameterize a fixed mask and do not learn a policy via neural networks ( Bahadir et al. , 2019 ; Huijben et al. , 2020b ; Weiss et al. , 2019 ) . In this work , we focus our investigation on the benefits of the policies trained using reinforcement learning , namely the contributions of Pineda et al . ( 2020 ) and Bakker et al . ( 2020 ) . We reproduce their results and extend their evaluation with additional baselines and experimental settings , i.e . change of hyperparameters and preprocessing . Deep RL approaches have drawn attention because the policies can in principle offer two important benefits : i ) long term planning and ii ) adapting to the currently acquired image at test time . The work of Pineda et al . ( 2020 ) however seems to indicate that long term planning could be the most important component in deep RL , as their results show that a non-adaptive , long term planning policy model trained on the dataset can perform as well as an adaptive , long-term planning policy . On the contrary , the contribution of Bakker et al . ( 2020 ) Sampling Method Learning-based Adaptive Long horizon . highlights the importance of adaptivity , as a greedy policy , that does not do long-term planning is found to closely match policies that do long term planning . This means that at time of writing there are two peer reviewed SotA papers in apparent conflict with another . Our results synthesize this apparent conflict : We observe that a simple , easy-to-train method that does not rely on deep RL , does not attempt long term planning and is by definition not adaptive can perform as well as the state-of-the-art approaches of Pineda et al . ( 2020 ) ; Bakker et al . ( 2020 ) . The fixed sampling policies are obtained by optimizing a fixed policy greedily optimization on a training set , following the Learning-based Compressed Sensing ( LBCS ) approach of ( Gözcü et al. , 2018 ; Sanchez et al. , 2019 ) . This trend can be consistently observed on the fastMRI single-coil knee dataset ( Zbontar et al. , 2019 ) , where we carry out evaluations across a variety of settings , whether full scale images , cropping the field of view , various mask designs used for training and various architectures . We observe that such small changes in the experimental pipeline can easily lead to reversals in the performance or in the conclusions , by having the RL approach matched or outperformed by LBCS . It is possible then to choose an experimental setting and performance metrics to support one ’ s desired conclusion : that RL outperforms LBCS , that LBCS outperforms RL or that their difference is not significant . Together with the observation that current SotA RL methods only add marginal value over the simple baselines at best , we hope that this work will highlight the urgent need for further discussions in the community about standardized metrics , strong baselines , and careful design of experimental pipelines to evaluate MRI sampling policies fairly . To ensure reproducibility and facilitate the use of strong baselines in future work we will release our code upon acceptance of this paper . 2 BACKGROUND . This work focuses on the following inverse problem , where we seek to recover a signal x ∈ CP from partial observations y ∈ CN , N P obtained by subsampling a unitary transform matrix A ∈ CP×P : yω = PωAx+ η , ( 1 ) where η ∈ CN is a noise vector , ω ⊆ [ P ] : = { 1 , . . . , P } is an index set of allowable sampling locations with cardinality N , Pω is a diagonal matrix such that ( Pω ) ii = 1 if i ∈ ω , 0 otherwise . Pω and ω are referred to as the ( sampling ) mask , as they control what locations are acquired in the original signal . This problem is inherently ill-posed , due to N P , and our first goal will be to construct an estimate x̂ω ; θ = fθ ( yω , ω ) of the original signal x , where fθ is a reconstruction method parameterized by θ . 2.1 MRI FUNDAMENTALS . In MRI , observations are obtained in the Fourier space , also referred to as k-space . The acquisition of data happens sequentially , but the physical constraints of an MRI acquisition make it more efficient to observe entire columns or rows at once , a setting known as Cartesian sampling ( Zbontar et al. , 2019 ) . Non-Cartesian sampling is also possible ( see for instance Weiss et al . ( 2019 ) ; Lazarus et al . ( 2019 ) for recent references ) , but some sampling methods like e.g . the pixel level sampling in Bahadir et al . ( 2019 ) ; Yin et al . ( 2021 ) are not practical for 2D MRI , as they do not allow for a reduction of scanning time compared sampling full trajectories . The acquisition of a column or row is known as a readout , and the complete procedure consists in acquiring N readouts sequentially . A full acquisition would require P readouts , and acquiring only N of P lines accelerates the scan by a factor P/N , adequately named acceleration factor , the inverse Sampling mask . of which is known as the sampling rate , and the spectrum obtained with N out of P readouts is referred to as undersampled . The observations obtained by taking an inverse Fourier transform lead to an aliased image , and require processing through a reconstruction method . In MRI , the Fourier space is typically structured , containing the bulk of energy in the low-frequencies located around the center of the space , and less around the high ones . While Compressed Sensing ( CS ) prescribes uniform sampling Donoho ( 2006 ) ; Candès et al . ( 2006 ) , the structure of Fourier space made it necessary to leverage heuristics that assign more probability to low-frequencies in order to reflect the energy distribution . This approach is known as variable density-sampling ( VDS ) ( Lustig et al. , 2007 ) . 2.2 SAMPLING OPTIMIZATION . The success of deep-learning approaches to MRI have led the broader medical imaging community to re-think the problem of optimizing the undersampling patterns in a data-driven fashion as well ( Gözcü et al. , 2018 ; Sanchez et al. , 2019 ; Zhang et al. , 2019 ; Jin et al. , 2019 ; Pineda et al. , 2020 ; Huijben et al. , 2020a ) , instead of relying on heuristics such as variable density sampling ( Lustig et al. , 2007 ) . The ideal sampling algorithm would tailor the mask to each instance of x ∼ p ( x ) solving minω : |ω|≤N ` ( x , x̂θ ( yω = PωAx ) ) , ( 2 ) which is not realizable since this requires using the unknown ground truth signal x at testing time and is computationally intractable due to the combinatorial nature of the problem . Two main approaches have been explored to circumvent this problem . Fixed ( open-loop ) sampling . A majority of data-driven mask design approaches use fixed masks ( Ravishankar & Bresler , 2011 ; Gözcü et al. , 2018 ; Sanchez et al. , 2019 ; Bahadir et al. , 2019 ; Wu et al. , 2019 ; Huijben et al. , 2020b ) . The subsampling mask is constructed ahead of time - either via a heuristic or by using training data - and kept fixed at inference time . Formally , the problem of choosing the subsampling pattern corresponds to finding a subset ω that satisfies arg minω : |ω|≤N Ex∼p ( x ) [ ` ( x , x̂θ ( yω = PωAx ) ) ] , ( 3 ) where we are constrained with a maximal sampling budget N and want to find a mask that minimizes a given loss function ` . The true risk is substituted with the empirical one , estimated from training samples , and x̂θ ( yω = PωAx ) is an approximation of the ground truth obtained by reconstruction . Adaptive ( closed-loop ) sampling . In contrast to Equation 3 , adaptive sampling generates a dynamic sampling mask using a heuristic Hφ evaluated at test time . For a fixed , unknown data sample x , we use information of the previously obtained measurements yωt−1 to determine which candidate v ⊂ [ P ] , v 6∈ ωt−1 should be acquired at time t. These methods solve at test time , for each individual target x , the following sequence of problems : vt ∈ arg minv : v∈ [ P ] Hφ ( v , x̂ωt−1 ; θ , ωt−1 ) for t = 1 , . . . , N ( 4 ) where ωt = { ωt−1 , vt } is defined in a nested fashion at each step . The heuristic score can be specifically trained for sampling leveraging reinforcement-learning framework ( Bakker et al. , 2020 ; Pineda et al. , 2020 ; Jin et al. , 2019 ) , trained to directly estimate the current error and simply acquiring measurements where the error is estimated to be currently largest ( Zhang et al. , 2019 ) , or be derived from available heuristics like posterior variance as in ( Ji et al. , 2008 ; Haupt et al. , 2009 ; Sanchez et al. , 2020 ) . 3 THE MRI DATA PROCESSING PIPELINE . Section 2 outlined the mathematical setting underlying accelerated MRI , but does not capture practical considerations that can greatly affect the performance of methods as well as their applicability in the real world . In this section , we will step through the stages of the MRI data processing pipeline as shown in Figure 2 . Note that this pipeline is independent of the sampling method and simply operates on a given mask . As we will show in Section 4.1 , seemingly trivial changes at even a single of these stages can affect the results of the evaluation and lead to changes in the ordering of the performance of different policies , an observation that we will refer to as reversal . We mark sections where we observed changes leading to reversals with ∗ and sections here they only shifted the results but do not lead to reversals with italics . We also discuss some caveats at these different steps in Appendix B Data sources and storage . In all cases , the ground truth signal x is initially acquired as a complex signal in k-space , generally using multiple coils . The fastMRI dataset ( Zbontar et al. , 2019 ) provides this raw multicoil data , as well as a simulated single-coil k-space which we use throughout this work . Preprocessing∗ . Due to computational constraints , it is common to resize the images by cropping and/or resizing or use magnitude images over the raw data . Cropping and resizing changes the ground truth distribution , which as seen in Table 3 can also lead to reversals in the final results . Sampling . While in the real world the data is actually acquired by sampling k-space ( prospective sampling ) , in practice , the acquisition is generally simulated by retrospectively undersampling fully sampled Cartesian data following Equation 1 . When training reconstructor and sampling method separately , training data are generally constructed with random masks that sample a certain fraction of center frequencies , and then the rest from a random distribution . There has been considerable variation in defining the parameters of these distributions , but to this day , no systematic study of their effect on the reconstruction quality have been carried out . Post processing and reconstruction . After sampling , the observation will be processed according to the implementation details of the reconstruction algorithm ( e.g. , normalized or standardized ) and the reconstruction is computed . We discuss the impact of postprocessing in Appendices B and C.2 . Evaluation metrics ∗ . Finally to judge the results , the most common metrics are peak signal-to-noise ratio ( PSNR ) , mean squared error ( MSE ) , normalized mean squared error ( NMSE ) and structural similarity index metric ( SSIM ) . All metrics tend to promote smooth images when used as a training loss ( Muckley et al. , 2020 ) but it is widely known that MSE , NMSE and PSNR focus on low-frequencies/high-energy components , which was also noted by Bakker et al . ( 2020 ) . However , while MSE , NMSE and PSNR only operate on image differences and treats every pixel independently , SSIM is computed from local averages ( Wang et al. , 2004 ; Zbontar et al. , 2019 ) . The metric is then reported as a curve plotted against sampling rate ( Zhang et al. , 2019 ) , its reciprocal , the acceleration rate ( Pineda et al. , 2020 ) or aggregated by computing the average performance , the performance at end of sampling , or the area under curve ( AUC ) . Details on the AUC computation are provided in Appendix A.4 . As discussed throughout the experiments , the choice of metric and its aggregation can greatly impact the conclusions drawn from results .

Summary: The paper first surveys several recent Fast MRI Recon methods, including RL-based and greedy-policy-based ones, that reconstruct high quality images from highly undersampled data from the perspective of optimizing the sampling of k-space data and then concerns the RL-based methods. Further it considers a typical Fast MRI Recon framework that contains three main blocks of preprecessing (cropping and/or resizing, using magnitude only etc.), subsampling (random, LtH, LBCS, adaptive, etc.), and post-precessing and reconstruction, and evaluates its performance changes triggered by seemingly trivial changes at even a single of these stages in a series of experiments. It shows that the RL-based method, which adpatively computes the sampling mask, does not offer benefits over fixed masks.