On the benefits of deep RL in accelerated MRI sampling

1 INTRODUCTION . Magnetic resonance imaging ( MRI ) is a non-invasive , non-ionizing medical imaging method which has been widely adopted in clinical settings due to its with unmatched quality in soft tissue contrast . However , MRI suffers from long scanning times , which limits patient comfort , imaging quality as well as throughput ( Zbontar et al. , 2019 ) . These issues have been a longstanding research direction in the community , and one prominent way to shorten scanning times is acquiring less measurements and use a reconstruction method to recover the full image , a setting known as accelerated MRI . Spurred by the great gains in reconstruction quality ( Muckley et al. , 2020 ) , recent works have moved away from using generic , non-learning based heuristics , such as variable-density sampling ( VDS ) ( Lustig et al. , 2007 ) to select the measurements to acquire . Instead , there has been a growing interest in strong , learning-based methods that tailor sampling policies to the reconstruction method and achieve even greater accelerations , and as a result , further speeding up imaging . The state-of-the-art approaches rely mostly on the idea of pairing a reconstruction model with a patientadaptive sampling model , where the former estimates a clean image from partial measurements , and the latter selects measurements that it predicts as likely to improve the reconstruction quality . The selection policy can be trained relying on techniques from reinforcement-learning ( Jin et al. , 2019 ; Pineda et al. , 2020 ; Bakker et al. , 2020 ) , heuristics such as a model trying to estimate the Fourier space error of the locations to be acquired ( Zhang et al. , 2019 ) or end to end via the straight-through estimator ( Van Gorp et al. , 2021 ; Yin et al. , 2021 ) . Other approaches choose to directly parameterize a fixed mask and do not learn a policy via neural networks ( Bahadir et al. , 2019 ; Huijben et al. , 2020b ; Weiss et al. , 2019 ) . In this work , we focus our investigation on the benefits of the policies trained using reinforcement learning , namely the contributions of Pineda et al . ( 2020 ) and Bakker et al . ( 2020 ) . We reproduce their results and extend their evaluation with additional baselines and experimental settings , i.e . change of hyperparameters and preprocessing . Deep RL approaches have drawn attention because the policies can in principle offer two important benefits : i ) long term planning and ii ) adapting to the currently acquired image at test time . The work of Pineda et al . ( 2020 ) however seems to indicate that long term planning could be the most important component in deep RL , as their results show that a non-adaptive , long term planning policy model trained on the dataset can perform as well as an adaptive , long-term planning policy . On the contrary , the contribution of Bakker et al . ( 2020 ) Sampling Method Learning-based Adaptive Long horizon . highlights the importance of adaptivity , as a greedy policy , that does not do long-term planning is found to closely match policies that do long term planning . This means that at time of writing there are two peer reviewed SotA papers in apparent conflict with another . Our results synthesize this apparent conflict : We observe that a simple , easy-to-train method that does not rely on deep RL , does not attempt long term planning and is by definition not adaptive can perform as well as the state-of-the-art approaches of Pineda et al . ( 2020 ) ; Bakker et al . ( 2020 ) . The fixed sampling policies are obtained by optimizing a fixed policy greedily optimization on a training set , following the Learning-based Compressed Sensing ( LBCS ) approach of ( Gözcü et al. , 2018 ; Sanchez et al. , 2019 ) . This trend can be consistently observed on the fastMRI single-coil knee dataset ( Zbontar et al. , 2019 ) , where we carry out evaluations across a variety of settings , whether full scale images , cropping the field of view , various mask designs used for training and various architectures . We observe that such small changes in the experimental pipeline can easily lead to reversals in the performance or in the conclusions , by having the RL approach matched or outperformed by LBCS . It is possible then to choose an experimental setting and performance metrics to support one ’ s desired conclusion : that RL outperforms LBCS , that LBCS outperforms RL or that their difference is not significant . Together with the observation that current SotA RL methods only add marginal value over the simple baselines at best , we hope that this work will highlight the urgent need for further discussions in the community about standardized metrics , strong baselines , and careful design of experimental pipelines to evaluate MRI sampling policies fairly . To ensure reproducibility and facilitate the use of strong baselines in future work we will release our code upon acceptance of this paper . 2 BACKGROUND . This work focuses on the following inverse problem , where we seek to recover a signal x ∈ CP from partial observations y ∈ CN , N P obtained by subsampling a unitary transform matrix A ∈ CP×P : yω = PωAx+ η , ( 1 ) where η ∈ CN is a noise vector , ω ⊆ [ P ] : = { 1 , . . . , P } is an index set of allowable sampling locations with cardinality N , Pω is a diagonal matrix such that ( Pω ) ii = 1 if i ∈ ω , 0 otherwise . Pω and ω are referred to as the ( sampling ) mask , as they control what locations are acquired in the original signal . This problem is inherently ill-posed , due to N P , and our first goal will be to construct an estimate x̂ω ; θ = fθ ( yω , ω ) of the original signal x , where fθ is a reconstruction method parameterized by θ . 2.1 MRI FUNDAMENTALS . In MRI , observations are obtained in the Fourier space , also referred to as k-space . The acquisition of data happens sequentially , but the physical constraints of an MRI acquisition make it more efficient to observe entire columns or rows at once , a setting known as Cartesian sampling ( Zbontar et al. , 2019 ) . Non-Cartesian sampling is also possible ( see for instance Weiss et al . ( 2019 ) ; Lazarus et al . ( 2019 ) for recent references ) , but some sampling methods like e.g . the pixel level sampling in Bahadir et al . ( 2019 ) ; Yin et al . ( 2021 ) are not practical for 2D MRI , as they do not allow for a reduction of scanning time compared sampling full trajectories . The acquisition of a column or row is known as a readout , and the complete procedure consists in acquiring N readouts sequentially . A full acquisition would require P readouts , and acquiring only N of P lines accelerates the scan by a factor P/N , adequately named acceleration factor , the inverse Sampling mask . of which is known as the sampling rate , and the spectrum obtained with N out of P readouts is referred to as undersampled . The observations obtained by taking an inverse Fourier transform lead to an aliased image , and require processing through a reconstruction method . In MRI , the Fourier space is typically structured , containing the bulk of energy in the low-frequencies located around the center of the space , and less around the high ones . While Compressed Sensing ( CS ) prescribes uniform sampling Donoho ( 2006 ) ; Candès et al . ( 2006 ) , the structure of Fourier space made it necessary to leverage heuristics that assign more probability to low-frequencies in order to reflect the energy distribution . This approach is known as variable density-sampling ( VDS ) ( Lustig et al. , 2007 ) . 2.2 SAMPLING OPTIMIZATION . The success of deep-learning approaches to MRI have led the broader medical imaging community to re-think the problem of optimizing the undersampling patterns in a data-driven fashion as well ( Gözcü et al. , 2018 ; Sanchez et al. , 2019 ; Zhang et al. , 2019 ; Jin et al. , 2019 ; Pineda et al. , 2020 ; Huijben et al. , 2020a ) , instead of relying on heuristics such as variable density sampling ( Lustig et al. , 2007 ) . The ideal sampling algorithm would tailor the mask to each instance of x ∼ p ( x ) solving minω : |ω|≤N ` ( x , x̂θ ( yω = PωAx ) ) , ( 2 ) which is not realizable since this requires using the unknown ground truth signal x at testing time and is computationally intractable due to the combinatorial nature of the problem . Two main approaches have been explored to circumvent this problem . Fixed ( open-loop ) sampling . A majority of data-driven mask design approaches use fixed masks ( Ravishankar & Bresler , 2011 ; Gözcü et al. , 2018 ; Sanchez et al. , 2019 ; Bahadir et al. , 2019 ; Wu et al. , 2019 ; Huijben et al. , 2020b ) . The subsampling mask is constructed ahead of time - either via a heuristic or by using training data - and kept fixed at inference time . Formally , the problem of choosing the subsampling pattern corresponds to finding a subset ω that satisfies arg minω : |ω|≤N Ex∼p ( x ) [ ` ( x , x̂θ ( yω = PωAx ) ) ] , ( 3 ) where we are constrained with a maximal sampling budget N and want to find a mask that minimizes a given loss function ` . The true risk is substituted with the empirical one , estimated from training samples , and x̂θ ( yω = PωAx ) is an approximation of the ground truth obtained by reconstruction . Adaptive ( closed-loop ) sampling . In contrast to Equation 3 , adaptive sampling generates a dynamic sampling mask using a heuristic Hφ evaluated at test time . For a fixed , unknown data sample x , we use information of the previously obtained measurements yωt−1 to determine which candidate v ⊂ [ P ] , v 6∈ ωt−1 should be acquired at time t. These methods solve at test time , for each individual target x , the following sequence of problems : vt ∈ arg minv : v∈ [ P ] Hφ ( v , x̂ωt−1 ; θ , ωt−1 ) for t = 1 , . . . , N ( 4 ) where ωt = { ωt−1 , vt } is defined in a nested fashion at each step . The heuristic score can be specifically trained for sampling leveraging reinforcement-learning framework ( Bakker et al. , 2020 ; Pineda et al. , 2020 ; Jin et al. , 2019 ) , trained to directly estimate the current error and simply acquiring measurements where the error is estimated to be currently largest ( Zhang et al. , 2019 ) , or be derived from available heuristics like posterior variance as in ( Ji et al. , 2008 ; Haupt et al. , 2009 ; Sanchez et al. , 2020 ) . 3 THE MRI DATA PROCESSING PIPELINE . Section 2 outlined the mathematical setting underlying accelerated MRI , but does not capture practical considerations that can greatly affect the performance of methods as well as their applicability in the real world . In this section , we will step through the stages of the MRI data processing pipeline as shown in Figure 2 . Note that this pipeline is independent of the sampling method and simply operates on a given mask . As we will show in Section 4.1 , seemingly trivial changes at even a single of these stages can affect the results of the evaluation and lead to changes in the ordering of the performance of different policies , an observation that we will refer to as reversal . We mark sections where we observed changes leading to reversals with ∗ and sections here they only shifted the results but do not lead to reversals with italics . We also discuss some caveats at these different steps in Appendix B Data sources and storage . In all cases , the ground truth signal x is initially acquired as a complex signal in k-space , generally using multiple coils . The fastMRI dataset ( Zbontar et al. , 2019 ) provides this raw multicoil data , as well as a simulated single-coil k-space which we use throughout this work . Preprocessing∗ . Due to computational constraints , it is common to resize the images by cropping and/or resizing or use magnitude images over the raw data . Cropping and resizing changes the ground truth distribution , which as seen in Table 3 can also lead to reversals in the final results . Sampling . While in the real world the data is actually acquired by sampling k-space ( prospective sampling ) , in practice , the acquisition is generally simulated by retrospectively undersampling fully sampled Cartesian data following Equation 1 . When training reconstructor and sampling method separately , training data are generally constructed with random masks that sample a certain fraction of center frequencies , and then the rest from a random distribution . There has been considerable variation in defining the parameters of these distributions , but to this day , no systematic study of their effect on the reconstruction quality have been carried out . Post processing and reconstruction . After sampling , the observation will be processed according to the implementation details of the reconstruction algorithm ( e.g. , normalized or standardized ) and the reconstruction is computed . We discuss the impact of postprocessing in Appendices B and C.2 . Evaluation metrics ∗ . Finally to judge the results , the most common metrics are peak signal-to-noise ratio ( PSNR ) , mean squared error ( MSE ) , normalized mean squared error ( NMSE ) and structural similarity index metric ( SSIM ) . All metrics tend to promote smooth images when used as a training loss ( Muckley et al. , 2020 ) but it is widely known that MSE , NMSE and PSNR focus on low-frequencies/high-energy components , which was also noted by Bakker et al . ( 2020 ) . However , while MSE , NMSE and PSNR only operate on image differences and treats every pixel independently , SSIM is computed from local averages ( Wang et al. , 2004 ; Zbontar et al. , 2019 ) . The metric is then reported as a curve plotted against sampling rate ( Zhang et al. , 2019 ) , its reciprocal , the acceleration rate ( Pineda et al. , 2020 ) or aggregated by computing the average performance , the performance at end of sampling , or the area under curve ( AUC ) . Details on the AUC computation are provided in Appendix A.4 . As discussed throughout the experiments , the choice of metric and its aggregation can greatly impact the conclusions drawn from results .

Summary: The authors report on the use and current limitations of using deep reinforcement learning for accelerated sampling in MR imaging. They focus on two specific works that use deep RL for sampling: Pineda and Bakker. They mainly compare these methods against a previously described non-RL technique: Stochastic Learning-based Compressive Sampling (LBCS). This method trains a non-adaptive, greedy sampling policy that selects as a measurement candidate in each acquisition step the column that leads to the greatest average improvement over a sample from a training dataset.   The authors conclude that the ‘return on investment’ of adopting RL over LBCS is generally marginal compared to what changes in the modeling pipeline can make. Improving the reconstruction architecture or using masking regimes adapted to the sampling horizon yield more significant gains than using RL-based methods. The authors state that their results confirm the stated limitation in both Pineda and Bakker, namely that neither adaptivity nor long-term planning methods are currently worth the increased cost.