Zero-Shot Self-Supervised Learning for MRI Reconstruction
Authors: Burhaneddin Yaman
Seyed Amir Hossein Hosseini
Mehmet Akcakaya
Venue: ICLR 2022 Poster (2022)
Keywords: Zero-shot learning
Self-supervised learning
MRI Reconstruction
Transfer learning
Physics-guided deep learning

Abstract: Deep learning (DL) has emerged as a powerful tool for accelerated MRI reconstruction, but often necessitates a database of fully-sampled measurements for training. Recent self-supervised and unsupervised learning approaches enable training without fully-sampled data. However, a database of undersampled measurements may not be available in many scenarios, especially for scans involving contrast or translational acquisitions in development. Moreover, recent studies show that database-trained models may not generalize well when the unseen measurements differ in terms of sampling pattern, acceleration rate, SNR, image contrast, and anatomy. Such challenges necessitate a new methodology to enable subject-specific DL MRI reconstruction without external training datasets, since it is clinically imperative to provide high-quality reconstructions that can be used to identify lesions/disease for $\textit{every individual}$. In this work, we propose a zero-shot self-supervised learning approach to perform subject-specific accelerated DL MRI reconstruction to tackle these issues. The proposed approach partitions the available measurements from a single scan into three disjoint sets. Two of these sets are used to enforce data consistency and define loss during training for self-supervision, while the last set serves to self-validate, establishing an early stopping criterion. In the presence of models pre-trained on a database with different image characteristics, we show that the proposed approach can be combined with transfer learning for faster convergence time and reduced computational complexity.

Published as a conference paper at ICLR 2022
ZERO-SHOT SELF-SUPERVISED LEARNING FOR MRI RECONSTRUCTION
Burhaneddin Yaman ft Seyed Amir Hossein Hosseini Mehmet Akçakaya
i Department of Eleetrical&Computt Engineering, University of Minnesota } Center for Magnetic Resonance Research, University of Minnesota (yaman013, hosse049, akcakaya (ocakaya)@umil @umn.edd
ABSTRACT
Deep learning (DL) has emerged as a powerful tool for accelerated MRI recon- struction, but often necessitates a database of fully-sampled measurements for training. Recent self-supervised and unsupervised learning approaches enable training without fully-sampled data. However, a database of undersampled mea- surements may not be available in many scenarios, especially for scans involving contrast translational acquisitions in development. Moreover, recent studies show that setabase-trainee models may not generalize well when the unseen mea- surements differ in terms of sampling pattern, acceleration rate, SNR, image con- trast, and anatomy. Such challenges necessitate a new methodology to enable subject-specific DL MRI reconstruction without external training datasets, since it is clinically imperative to provide high-quality reconstructions that can be used identify lesions/disease for every individual. In this work, we propose zero- shot self-supervised learning approach to perform subject-specific accelerated DL MRI reconstruction to tackle these issues. The proposed approach partitions the available measurements from single scan into three disjoint sets. Two of these sets are used to enforce data consistency and define loss during training for self- supervision, while the last set serves to self-validate, establishing criterion. In the presence of models pre-trained on a database with different image characteristics, we show that the proposed approach can be combined with transfer learning for faster convergence time and reduced computational complexity.
early stopping
INTRODUCTION
Magnetic resonance imaging (MRI) a non-invasive, radiation-free medical imaging modality that provides excellent soft tissue contrast for diagnostic purposes. However, lengthy acquisition times in MRI remain a limitation. Accelerated MRI techniques acquire fewer measurements at a sub- Nyquist rate, and use redundancies in the acquisition system the images to remove the resulting aliasing artifacts during reconstruction. In clinical MRI systems, multi-coil receivers are used during data acquisition. Parallel imaging (PI) is the most elinically used method for accelerated MRI, and exploits the redundancies between these coils for reconstruction (Pruessmann et al.) 1999) Griswold al., 2002). Compressed sensing (CS) is another conventional accelerated MRI technique that exploits the compressibility of images in sparsifying transform domains (Lustig et al.) (007). and commonly used in combination with PI. However, PI and CS may suffer from noise and residual
artifacts at high acceleration rates (Robson et al. 2008 Sandino et al. 2020).
Recently, deep learning (DL) methods have emerged as an alternative for accelerated MRI due to their improved reconstruction quality compared to conventional approaches (Hammernik et al.) 2018 Knoll et al. (0220b AAçakaya et al., 2022). ). Particularly, physics-guided deep learning recon- struction (PG-DLR) approaches have gained interest due to their robustness and improved perfor- mance (Hammernik et al. 2018 Hosseini et al. 2020b). PG-DLR explicitly incorporates the physics of the data acquisition system into the neural network via procedure known as algorithm unrolling (Monaa et al. (021). This is done by unrolling iterative optimization algorithms that alternate be- tween data consistency (DC) and regularization steps for fixed number of iterations. Subsequently, PG-DLR methods are trained in a supervised manner using large databases of fully-sampled mea- surements (Hammernik et al. 2018 Aggarwal et al. 2019). More recently, self-supervised learning

---

Published as a conference paper at ICLR 2022
has shown that reconstruction quality similar to supervised PG-DLR can be achieved while training on a database of only undersampled measurements (Yaman et al. 2020).
While such database learning strategies offer improved reconstruction quality, acquisition of large datasets may often be infeasible. In some MRI applications involving time-varying physiological processes, dynamic information such as time courses of signal changes, contrast uptake or breath- ing patterns may differ substantially between subjects, making it difficult to generate high-quality databases of sufficient size for the aforementioned strategies. Furthermore, database training. in general, brings along concerns about robustness and generalization (Eldar et al. 2017 Knoll et al.) 2020cp. In MRI reconstruction, this may exhibit itself when there are mismatches between training and test datasets in terms of image contrast, sampling pattern, SNR, vendor, and anatomy. While it is imperative to have high-quality reconstructions that can be used to correctly identify lesions/disease for every individual, the fastMRI transfer track challenge shows that pretrained models fail to gen- eralize when applied to patients/scans with different distribution or acquisition parameters, with po- tential for misdiagnosis (Muckley et al. 2021). Finally, training datasets may lack examples of rare and/or subtle pathologies, increasing the risk of generalization failure (Knoll et al. 2019 2020c).
In this work, we tackle these challenges associated with database training, and propose a zero-shot self-supervised learning (ZS-SSL) approach, which performs subject-specific training of PG-DLR without any external training database. Succinctly, ZS-SSL partitions the acquired measurements into three types of disjoint sets, which are respectively used only in the PG-DLR neural network, in defining the training loss, and in establishing stopping strategy to avoid overfitting. Thus, our training is both self-supervised and self-validated. In cases where a database-pretraingg network is available, ZS-SSL leverages transfer learning (TL) for improved reconstruction quality and reduced computational complexity.
Our contributions can be summarized as follows:
We propose a zero-shot self-supervised method for learning subject-specific DL MRI re- construction from single undersampled dataset without any external training database. We provide a well-defined methodology for determining stopping criterion to avoid over- fitting in contrast to other single-image training approaches (Ulyanov et al., 2018).
We apply the proposed zero-shot learning approach to knee and brain MRI datasets, and show its efficacy in removing residual aliasing and banding artifacts compared to super- vised database learning.
We show our ZS-SSL can be combined with with TL in cases when a database-prettranned model is available to reduce computational costs.
We show that our zero-shot learning strategies address robustness and generalizability is- sues of trained supervised models in terms of changes in sampling pattern, acceleration rate, contrast, SNR, and anatomy at inference time.
2 BACKGROUND AND RELATED WORK
2.1 ACCELERATED MRI ACQUISITION MODEL
In MRI, raw measurement data is acquired in the frequency domain, also known k-space. In current clinical MRI systems, multiple receiver coils are used, where each is sensitive to different parts of the volume. In practice, MRI accelerated by taking fewer measurements, which are characterized by an undersampling mask that specifies the acquired locations in k-space. For a multi-coil MRI acquisition, the forward model is given as
y; + i (l)...,nc).
(I)
where x is the underlying image, y: is the acquired data for the 7th coil, Pn is the masking operator for undersampling pattern Q F is the Fourier transform, C2 is diagonal matrix characterizing the 7th coil sensitivity, n, is measurement noise for 1hh coil, and n. is the number of coils (Pruessmann al. 1999). This system can be concatenated across the coil dimension for a compact representation
yn E,x + n,
where yn is the acquired undersampled measurements across all coils, En is the forward encod- ing operator that concatenates F2 across i E {I,..., me). The general inverse problem for
2

---

Published as a conference paper at ICLR 2022
accelerated MRI is given
arg min live
E + R(x),
(3)
where the Iyn | EaxlE term enforces consistency with acquired data (DC) and R(-) is regularizer.
2.2 PG-DLR WITH ALGORITIIM UNROLLING
Several optimization methods are available for solving the inverse problem in 3 (Fesseer 2020
Variable-splitte via quadratic penalty is one of the approaches that can be employed cast Eq. (3) into two sub-problems as
()) arg min y (||x((111 =11
1 + R(z),
(4a)
arg min llyn Eax + /u|1x (0) 102
(4b)
where is the penalty parameter, ( is an intermediate variable and x(() is the desired image at iteration i. In PG-DLR, an iterative algorithm, in 4a) and (4b) is unrolled for fixed number of iterations (Liang et al. 2020). The regularizer sub-problem in Eq. (4a) is implicitly solved with
neural networks and the DC sub-problem in Eq. (4b) solved via linear methods such as gradient
descent (Hammernik et al. 2018) conjugate gradient (CG) (Aggarwal al. 2019).
There have been numerous works on PG-DLR for accelerated MRI (Schlemper et al. 2018 Ham- mernik et al.) 2018 Aggarwal et al. 2019 Liang et al.) 2020 Yaman al. 2020). Most of these works vary from each other on the algorithms used for DC and neural networks employed in the regularizer units. However, all these works require a large database of training samples.
2.3 SUPERVISED LEARNING FOR PG-DLR
In supervised PG-DLR, training is performed using database of fully-sampled reference data. Let yier be the fully- sampled k-space for subject n and f fya, E;; 0) be the output of the unrolled net- work for sampled k-space yd. where the network is parameterized by 0. End-to-end training
minimizes (Knoll et al. 2020b Yaman al. 2020)
min N
Clvin f Ei; 0)),
where N is the number of samples in the training database, E is the fully-sampled encoding operator that transform network output to k-space and C(.,.) is a loss function.
2.4 SELF-SUPERVISHR LEARNING FOR PG-DLR
Unlike supervised learning, self-supervised learning enables training without fully-sampled data by only utilizing acquired undersampled measurements (Yaman al. 2020). masking approach is used for self-supervision in this setting, where a subset A C 52 is set aside for checking prediction performance/loss calculation, while the remainder of points è 22 are used in the DC units of the PG-DLR network. End-to-end training is performed using the loss function
N
(yy,, ((:
min
(6)
FL
3 ZERO-SHOT SELF-SUPERVISED LEARNING FOR PG-DLR
As discussed in Section I] lack of large datasets in numerous MRI applications. as well as robustness and generalizability issues of pretrained models pose a challenge for the clinical translation of DL reconstruction methods. Hence, subject-specific reconstruction is desirable in clinical practice, since it is critical to achieve a reconstruction quality that can be used for correctly diagnosing every patient. While the conventional self-supervise for subject-specine learning, it leads to overfitting unless the training is stopped early (Hosseini et al, 2020a). This is similar other single-image learning strategies, such the deep image prior (I)P))
masking strategy, in Yaman al. 2020) can be applied
3

---

Published as a conference paper at ICLR 2022
Consistency (DC)
Training
Ext/t...!
Sero-Filed
Sensiiiity Mays
DC Maak
Tramnng
octpat
UnIT
Unrolled
Parametera.0
Unit
Acquled
Eur Sur
Network output
Inspace
Tra
Valicatton
Unroltee Networkk
Validation
Firall Recondtrucion
Black (mS)
AELL
Figure 1: An overview of the proposed zero-shot self- supervieee learning approach. a) Acquired measurements for the single scan partitioned into three sets: training (9) and loss mask (A) for self-supervision, and a self-validation mask for automated early stopping (T') b) The parameters, 0. of the unrolled MRI reconstruction network updated using θ and A in the data consistency (DC) units of the unrolled network and for defining loss, respectively. c) Concurrently, a k-space validation procedure is used to establish the stopping criterion by using S in the DC units and T to measure validation loss. d) Once the network training has been stopped due to an increasing trend in the k-space validation loss, the final reconstruction is performed using the relevant learned network parameters and all the acquired measurements in the DC unit.
zero-shot super-resolution [Ulyanov et al. 2018 Shocher et al. 2018). DIP-type approaches
shows that an untrained neural network can successfully perform instance-specific image restoration tasks such denoising, super-resolution, inpainting without any training data. However, such DIP- type techniques requires an early stopping for avoiding over-fitting, which is typically done with manual heuristic selection (Clyanov al. 2018 Hosseini al. 20200a Darestani et al. [02T). While this may work in research setting, having well-defned automated early stopping criterion critical fully harness the potential of subject-specific DL MRI reconstruction practice.
Early stopping regularization in tilabase--rainee setting is conventionally motivated through the bias-variance trade-off, in which a validation set is used as proxy identify the stopping criterion. Using the same bias-variance trade-off motivation, having a valida- tion set can aid in devising a stopping criterion, but this has not been feasible in existing zero-shot learning approaches, which either use all acquired measurements (Ulyanov al.||2211 2018 Senouf et al.) 2019) or partition them into two sets for training and defining loss (Hosseini et al. 2020a).. Hence, existing zero-shot learning techniques lack validation set identify the stopping criterion.
generalization error to
ZS-SSL Formulation and Training: We propose new ZS-SSL. partitioning framework to enable subject-specific self-supervised training and validation with a well-defined stopping criterion. We define the following partition for the available measurement locations from a single scan, Ω:
Ω QuALr,
where Ll denotes a disjoint union, iee. θ, A and I are pairwise disjoint (Figure [. Similar to Section 2.4] θ is used in the DC units of the unrolled network, and A is used define a k-space loss for the self-supervision of the network. The third partition I' i' a set of acquired k-space indices set aside for defining k-space validation loss. Thus, 7S-SSI. training is both self-supervised and self-validated
In general, since zero-shot learning approaches perform training using a single dataset, generation of multiple data pairs from this single dataset necessary self-supervise the neural network (Quan) et al. 2020). Hence, we generate multiple (0, A) pairs from the acquired locations Ω2 of the single scan. ZS- this is achieved by fixing the k-space validation partition I' C Ω and performing the retrospective masking on I multiple times. Formally, I is partitioned K times such that
ΩT O₂ L. Ag,
I (1,...,K))
(8)
4

---

Published as a conference paper at ICLR 2022
where Ak. O₂ and I' are pairwise disjoint, i.e. 2= -2=ru L As, A k. ZS-SSL training minimizes
K Σ (yyA: EA, (f(ye,: Ee,: 0 K
min
In the proposed ZS-SSL, this is supplemented by k-space self-validation loss, which tests the generalization performance of the trained network on the k-space validation partition T. For the ( epoch, where the learned network weights are specified by 0 this validation loss is given by:
<yyr. (r Eagr: )
Note that in (9), the network output is calculated by applying the DC units Ω θ U A, iee. all acquired points outside of T, to better assess its generalizability performance. Our key motivation is that while the training loss will decrease over epochs, the k-space validation loss will start increasing once overfitting is observed. Thus, we monitor the loss in (9) during training to define an early stopping criterion to avoid overfitting. Let L be the epoch in which training needs to be stopped. Then inference time, the network output is calculated fyn, E (())), i.c. all acquired points are used to calculate the network output.
ZS-SSL with Transfer Learning (TL): While pretrained models are very efficient in reconstruct- ing new unseen measurements from similar MRI scan protocols, their performance degrades signif- icantly when acquisition parameters vary (Muckley al. 2021). Moreover, retraining new model on a large database for each acquisition parameter,
may be very computationally expensive (Knoll et al. 2019). Hence, TL has been used for re- re-training DL models pre-trained on large databases to reconstruct MRI data with different characteristics (Knoll al. 2019). However, such transfer still requires another, often smaller, database for re-training. In contrast, in the presence of pre- pra-tr models, ZS-SSL can be combined with TL, referred to as ZS- SSL-TL, to reconstruct a single slice/instance with different characteristics by using weights of the pre-trained model for initialization. Thus, ZS-SSL-TL ensures that the pretrained model is adapted for each patient/subject, while facilitating faster convergence time and reduced reconstruction time.
4 EXPERIMENTS
4.1 DATASETS
We performed experiments on publicly available fully-sampled multi-coil knee and brain MRI from fastMRI database (Knoll et al., 2020a). Knee and brain MRI datasets contained data from 15 and 16 receiver coils, respectively. Fully-sample, datasets were retrospectively undersampled by keeping 24 lines of autocalibrated signal (ACS) from center of k-space. FastMRI database contains different contrast weightings. For knee MRI, we used coronal proton density (Cor-PD) and coronal proton density with fat suppression (Cor-PDFS), and for brain MRI, axial FLAIR (AX-FLAIR) and axial T2 (Ax-T2). Different types of datasets and undersampling masks used in this study are provided in Figure in the Appendix.
4.2 IMPLEMENTATION DETAILS
All PG-DLR approaches were trained end-to-end using 10 unrolled iterations. CG method and ResNet structure (Timofte et al. 2017) were employed in the DC and regularizer units of the unrolled network, respectively (Yaman et al. 2020). The ResNet comprised of a layer of input and output convolution layers, and 15 residual blocks (RB) each containing two convolutional layers, where the first layer is followed by ReLU and the second layer is followed by a constant multiplication Timofte et al. (017). All layers had a kernel size of 3 x 3, 64 channels. The real and imaginary parts of the complex MR images were concatenated prior to being input to the ResNet as 2-channel images. The unrolled network, which shares parameters across the unrolled iterations had total of 592,129 trainable parameters. Coil sensitivity maps were generated from the central 24x24 ACS using ESPIRiT (Uecker et al.) 2014). End-to-end training
performed with normalized f1- loss (Adam optimizer, LR = 5- 10 batch size = 1) (Yaman et al. 2020). Peak signal-to- noise ratio (PSNR) and structural similarity index (SSIM) used for quantitative evaluation.
5

---

Published as a conference paper at ICLR 2022
4.3 RECONSTRUCTION METHOD COMPARISONS
In this work, we focus on comparing training strategies for accelerated MRI reconstruction. Thus, the same network architecture from Section 4.2 for all training methods in all experiments. We note that the proposed ZS-SSL strategy is agnostic to the specifics of the neural network ar- chitecture. In fact, the number of network parameters is higher than the number of undersampled measurements available on a single slice, i.c. dimension of yo. As such, different neural networks may be used for the regularizer unit the unrolled network, but this is not the focus of our study.
Supervised PG-DLR: Supervised PG-DLR models for knee and brain MRI were trained on 300 slices from 15 and 30 different subjects, respectively. For each knee and brain contrast weighting, two networks were trained separately using random and uniform masks (Hammernik et al.) 2018) at an acceleration rate (R) of 4 (Knoll et al. 2020c). Trained networks were used for comparison and TL purposes. We note that random undersampling results in incoherent artifacts, whereas uniform undersampling leads to coherent artifacts that are harder to remove (Figure Z in Appendix) (Knoll et al. 2019). Hence, we focus on the more difficult problem of uniform undersampling, while presenting random undersampling results in the Appendix.
Self-Supervision via Data Undersampling (SSDU) PG-DLR: SSDU Yaman al. 2020) PG- DLR was trained using the same database approach as supervised PG-DLR, with the exception that SSDU performed training only using the undersampled data (See. 2.4).
DIP-Recon:
We employ a DIP-type subject-specifee MRI reconstruction that uses all acquired mea- surements in both DC and defining loss (Senout et al. 2019: Jafari et al. (021
L (ya,, (J En; 0))
(10)
We refer to the reconstruction from this training mechanism as DIP-Recon. DIP-Recon- refers to combining (1O) with TL. As mentioned, DiP-Recon does not have a stopping criterion, hence early stopping was heuristically determined (Figure 8 in the Appendix).
Parallel Imaging: We include CG-SENSE, which a commonly used subject-specific conventional Pl method (Pruessmann et al. 1999 2001), as the clinical baseline quality for comparison purposes.
4.4 AUTOMATED STOPPING AND ABLATION STUDY
The stopping criterion for the proposed ZS-SSL was investigated slices from the knee dataset. The k-space self-validation set I was selected from the acquired measurements Ω using a uniformly random selection with [|] = 0.2. The remaining acquired measurements R2 were retrospec- tively partitioned into disjoint 2-tuples multiple times based on uniformly random selection with the ratio p = | [ 0.4 Ak E {L.....' K} (Yaman al. 2020).
Figure 2h shows representative subject-specific training and validation loss curves at R = 4 for K E (1, 10, 25, 50, 1003. As expected, training loss decreases with increasing epochs for all K. The k-space validation loss for K = 1 decreases without showing clear breaking point for stopping. For K > 1, the validation loss forms an L-curve, and the breaking point of the L-curve is used as the stopping criterion. K 10 is used for the rest of the study, while noting K 25, 50 and 100
C4
E30
WAldein LANE
100
250
Epccte
Epects
Figure 2: a) Representative training and k-space validation loss curves for ZS-SSL with multiple K E {1, 10, 10,000 25, 50, 100} masks on Cor-PD knee MRI using uniform undersampling at = 4. For K K>1 1 the validation loss forms L-curve, whose breaking point (red arrows) dictates the auto- mated early stopping criterion for training. b) Loss curves for ZS-SSL with/without TL for K = 10 on Cor-PD knee MRI slice. ZS- ZSL with TL converges faster compared to ZS-SSL (red arrows).

---

Published as a conference paper at ICLR 2022
Ground Truth
CG SENSE
Supervised PG-DLR
PG-DLR
DiP-Recon
DiP-Recon-TL
Zs SSL-TL
AXFLAIR Brain MRI, Red ConPD Knee MiR,, Red
Figure 3: Reconstruction results on a representative test slice from a) Cor-PD knee MRI and b) Ax- FLAIR brain MRI at 4 with uniform undersampling. CG-SENSE, DIP-Recon, DIP-Recon-TT TI. suffer from noise amplification and residual artifacts shown with red arrows, especially in knee MRI due to the unfavorable coil geometry. Subject-specific ZS-SSL and ZS-SSL-TL achieve artifact-free and improved reconstruction quality, similar to the database-trained SSDU and supervised PG-DLR.
also show similar performance. Figure 2p shows loss curves on Cor-PD slice with and without transfer learning. ZS-SSL-TL, which uses pre-trained supervised PG-DLR parameters as initial starting parameters, converges faster in time compared to ZS-SSL, substantially reducing the total training time. Average computation times for single-instance reconstruction methods are presented in Table 2 in the Appendix. Similarly, corresponding reconstruction results for the loss curves in Figure 2 and b are provided Figure in the Appendix.
4.5 RECONSTRUCTION RESULTS
In the first set of experiments, we compare all methods for the case when the testing and training data belong to the same knee/brain MRI contrast weighting with the undersampling mask. These experiments aim to show the efficacy of the proposed approach in performing subject-specifie MRI reconstruction, while removing residual aliasing artifacts. We also note that this is the most favorable setup for database-trained supervised PG-DLR.
acceleration rate and
In the subsequent experiments, focus the reported generalization and robustness issues with database-trained PG-DLR methods (Knoll et al.) 2019; 2020c Defazio et al.) 22020] Muckley et al.) 2021]. We investigate banding artifacts, as well as in-domain and cross-domain transfer cases. For these experiments, we concentrate on ZS-SSL-TL, since ZS ZS-SSL has no prior domain information, and is inherently not susceptible to such generalizability issues.
Comparison of Reconstruction Methods: In these experiments, supervised and SSDU PG-DLR trained and tested using uniform undersampling R 44, representing perfect match for training and testing conditions. Figure 3p and show reconstruction results for Cor-PI) knee and Ax-FLAIR brain MRI datasets in this setting. CG-SENSE reconstruction suffers from significant residual arti- facts and noise amplification in Cor-PD knee and Ax-FLAIR brain MRIs, respectively. Similarly, both DIP-Recon and DIP-Recon-TL. suffer from residual artifacts and noise amplification. Super- vised PG-DLR achieves triffcctffre reconstruction. Both ZS-SSL and ZS-SSL-TL also perform artifact-free reconstruction with similar image quality. Table [ shows the average SSIM and PSNR values on 30 test slices. Similar observations apply when random undersampling is employed (Fig- ure [0 the Appendix). For the remaining experiments. we investigate the generalizability of database-pretraime pretrained models using supervised PG-DLR as baseline due to its higher performance,
Table 1: Average PSNR and SSIM values on 30 test slices.
Manics
UGSENNS
RE-DER SSDVU FG-DL
Z3.3L.
28-38L-11.
SSIM
PSNR
10.966
33.883
Ax-FI
PSNR
SSIM
369
0336
37,375
BUS4
36.761
03255
30.637
0.709
31,299
OXIS
36861
G03S
37.250
03337

---

Published as a conference paper at ICLR 2022
Ground Truth
Supervised PG-DLR
DIP-Recon-T TL
ZS-SSL-TL
Figure 4: Supervised PG-DLR suffers from banding artifacts (yellow arrows), while ZS-SSI.-TL. significantly alleviates these artifacts. DIP-Recon- TL suffers from clear noise amplification.
while noting SSDU PG-DLR, which is self-supervised database-trained model, may also be used as a baseline needed.
Banding artifacts appear in the form of streaking horizontal lines, and occur due to high acceleration rates and anisotropic sampling (Defazio al. 2020). These hinder radiological evaluation and regarded
Banding Artifacts:
barrier for the translation of DL reconstruction methods into clinical practice (Defazio et al.) 2020). This set of experiments explored training and testing on Cor-PDFS data, where database-trained PG-DLR reconstruction has been reported show such artifacts (De- fazio et al. 2020 Muckley et al.) 2021). Figure 4 shows reconstructions for a Cor-PDFS test slice. While MIP-Recon-T TL suffers from clearly visible noise amplification, supervised PG-DLR suffers from banding artifacts shown with yellow arrows. ZS-SSL- significantly alleviates these banding artifacts in the reconstruction. While supervised PG-DLR achieves slightly better SSIM and PSNR (Table in the Appendix), we note that the banding artifacts do not necessarily correlate with such metries, and are usually picked up expert readings (Defazio et al. 2020 Knoll al. 2020c).
In-Domain Transfer: In these experiments, we compared the in-domain generalizability of database-traines PG-DLR and subject-specific PG-DLR. For in-domain transfer, training and test datasets are of the same type of data, but may differ from each other in terms of acceleration and un- dersampling pattern (Figure the Appendix). In Figure supervised PG-DLR was trained with random undersampling and tested on uniform undersampling. both at R = 4. Supervised PG-DLR fails to generalize and suffers from residual aliasing artifacts (red arrows), consistent with previous reports (Knoll et al., 2019; Muckley et al.) 2021). Similarly, DIP-Recon-TL. suffers from artifacts and noise amplification. Proposed ZS-SSL-TE achieves an artifact-free and improved reconstruc- tion quality. In Figure 5b. supervised PG-DLR was trained with uniform undersampling R and tested on uniform undersampling at = 6. While both supervised PG-DLR and DIP- HeP-Recon-TL suffers from aliasing artifacts, 7S-SSL-TL successfully removes these artifacts. Average PSNR and SSIM values align with the observations (Table in the Appendix).
Ground Truth
Supervised PG-DLR
DIP-Recon-TL
ZS-SSL-TL
b)
Figure 5: Supervised PG-DLR trained with a) random mask and tested uniform mask, both = 4; b) uniform mask at R = and tested on R = uniform mask. Supervised PG-DLR and DIP- Recon-TL 'L suffer from visible artifacts (red arrows). ZS-SSL-TL yields artifact-free reconstruction.
8

---

Published as a conference paper at ICLR 2022
Ground Truth
Supervised PG-DLR
DIP-Recon-TL
ZS-SSL-TL
Figure 6: Using pre-trained a) Cor-PDFS (low-SNR) and b) Ax-FLAIR (brain MRI) models for Cor-PD. Supervised PG-DLR fails to generalize for both contrast/SNR and anatomy changes, suf- fering from residual artifacts (red arrows). DiP-Recon-TL. also shows artifacts. ZS-SSL-TL. suc- cessfully removes noise and artifacts for both cases.
In the last set of experiments, we investigated the cross-domain gen- eralizability of database-trained PG-DLR compared subject-specific trained PG-DLR. For cross- domain transfer, training and test datasets are of the different data characteristics and generally differ terms of contrast, SNR, and anatomy (Figure [I in the Appendix), Figure 6 shows results for the case when the testing contrast/SNR and anatomy differs from training contrast/SNR and anatomy, even though the same R = 4 uniform undersampling is used for both training and testing. In Fig- ure supervised PG-DLR and different contrast). In Figure 6b, supervised PG-DLR was trained on Ax-FLAIR (brain MRI) and tested Cor-PD (knee MRI). In both cases, supervised PG-DLR fails to generalize and has residual artifacts (red arrows). Similarly, DIP-Recon-TT TL suffers from artifacts and noise. ZS-SSL- TL achieves an artifact-free improved reconstruction. For both cross-domain transfer experiments, similar results were observed for brain MRI (Figure I2 the Appendix). Average PSNR and SSIM values match these observations (Table 3 in the Appendix)
Cross-Domain Transfer:
trained on Cor-PDES (low-SNR) but tested on Cor-PD (high-SNR
5 CONCLUSIONS
We proposed a zero-shot self-supervised deep learning method, ZS-SSL, for subject-specific accel- erated DL MRI reconstruction from single undersampled dataset. The proposed ZS-SSL partitions the acquired measurements from single scan into three types of disjoint sets, which are used only in the PG-DLR network, in defining the training loss, and in establishing a validation strategy for early stopping to avoid overfitting. In particular, we showed that with our training methodology and automated stopping criterion, subject-specific zero-shot learning of PG-DLR for MRI can be achieved even when the number of tunable network parameters is higher than the number of avail- able measurements. Finally, we also combined ZS-SSL with transfer learning, cases where a pre-trained model may be available, for faster convergence time and reduced reconstruction time. Our results showed that ZS-SSL methods perform similarly to database-trained supervised PG-DLR when training and testing data are matched, and they significantly outperform catabase-trainee meth- ods in terms of artifact reduction and generalizability when the training and testing data differ in terms of image characteristics and acquisition parameters. In fact, the subject-specifie nature of ZS-SSL ensures that it is agnostic to such changes in acquisition parameters. As such, the proposed work is able provide good reconstruction quality for each subject, and may have significant im- plications in the integration of DL reconstruction to clinical studies. We note that hyperparameters, such as learning rate may be adjusted based on the domain for further improvements. It is also noteworthy that the subject-specific ZS-SSI. eliminates the requirement for large training sets. This may also facilitate the use and appeal of DL reconstruction for recently developed acquisitions, as well pilot studies that are often performed to determine the acquisition parameters/acceelraatio rates of large-scale imaging studies, such as the Human Connectome Project (HCP) (Ugurbil et al.) 2013). Finally, while concentrated and results may inspire further work in related image restoration problems, as well as for generative models or data-driven problems without data consistency term.
physics-guided guided models in MRI reconstruction, ideas

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work was partially supported by NIH R0THL153146, P41EB027061, UO1EB025144, NSF CAREER CCF-1651825. There is no conflict of interest for the authors.
REFERENCES
Hemant Kumar Aggarwal, Merry P. Mani, and Mathews Jacob. Modl: Model-based deep learning architecture for inverse problems. IEEE Trans. Medical Imaging, 38(2):394 405, 2019.
Mehmet Akçakaya, Burhaneddin Yaman, Hyungjin Chung, and Jong Chul Ye. Unsupervised deep learning methods for biological image reconstruction and enhancement: An overview from signal processing perspective. IEEE Sig Proc Mag, 39(2):28 2022.
Jesper LR Andersson and Sotiropoulos Stamatios N. An integrated approach to correction for off- resonance effects and subject movement in diffusion imaging. Neuroimage, 125:1063-1078, 2016.
Tamer A Basha, Mehmet Akcakaya, Charlene Liew, Connie W Tsao, Francesca N Delling, Gifty Addae, Long Ngo, Warren J Manning, and Reza Nezafat. Clinical performance of high-resolution late gadolinium enhancement imaging with compressed sensing. Journal of Magnetic Resonance Imaging, 46(6):1829-188) 2017.
Mohammad Zalbagi Darestani, Akshay Chaudhari, and Reinhard Heckel. Measuring robustness in deep learning based compressive sensing. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 2433-2444 PMLR, 2021.
Aaron Defazio, Tullie Murrell, and Michael Recht. MRI banding removal via adversarial training. In Advances in Neural Information Processing Systems, volume 33, pp. 7660-7670, 2020.
Yonina C. Eldar, Alfred O. Hero III, Li Deng, Jeffrey A. Fessler, Jelena Kovacevic, H. Vincent Poor, and Steve J. Young. Challenges and open problems signal processing: Panel discussion summary from ICASSP. IEEE Signal Process. Mag., 34(6):8-23, 2017.
Jeffrey A. Fessler. Optimization methods for magnetic resonance image reconstruction: Key models and optimization algorithms. IEEE Signal Processing Magazine, 37(1):33 40, 2020.
Mark A. Griswold, Peter M. Jakob, Robin M. Heidemann, Mathias Nittka, Vladimir Jellus, Jianmin Wang, Berthold Kiefer, and Axel Haase. Generalized autocalibrating partially parallel acquisi- tions (GRAPPA). Magn Reson Med, 47:1202- 1210, 2002.
Kerstin Hammernik, Teresa Klatzer, Erich Kobler, Michael Recht, Daniel Sodickson, Thomas Pock, and Florian Knoll. Learning a variational network for reconstruction of accelerated MRI data. Magn Reson Med, 79:3055-3000 2018
Seyed Amir Hossein Hosseini, Burhaneddin Yaman, Steen Moeller, and Mehmet Akçakaya. High- fidelity accelerated MRI reconstruction by scan-specific fine-tuning of physics-based neural net- works. In EMBC, pp. 1481-1484. IEEE, 2020a.
Seyed Amir Hossein Hosseini, Burhaneddin Yaman, Steen Moeller, Mingyi Hong, and Mehmet Akçakaya. Dense recurrent neural networks for accelerated MRI: history-cognizant unrolling of optimization algorithms. IEEE J. Sel. Top. Signal Process., 14(6):1280-129)) 2020b.
Ramin Jafari, Pascal Spincemaille, Jinwei Zhang, Thanh Nguyen, Martin R. Prince, Xianfu Luo, Junghun Cho, Daniel Margolis, and Yi Wang. Deep neural network for water/fat separation: Supervised training, unsupervised training, and no training. Magn Reson Med, 85(4):2263-2277, 04 2021.
F. Knoll, J. Zbontar, A. Sriram, M. J. Muckley, M. Bruno, A. Defazio, M. Parente, K. J. Geras, J. Kat- snelson, H. Chandarana, Z. Zhang, M. Drozdzaly, A. Romero, M. Rabbat, P. Vincent, J. Pinkerton, D. Wang, N. Yakubova, E. Owens, C. L. Zimick, M. P. Recht, D. K. Sodickson, and Y. W. Lui. fastMRI: publicly available raw k-space and DICOM dataset knee images for accelerated MR image reconstruction using machine learning. Radiology AI, pp. el e190007, 2020a.
10

---

Published as a conference paper at ICLR 2022
Florian Knoll, Kerstin Hammernik, Erich Kobler, Thomas Pock. Michael P. Recht, and Daniel K. Sodickson. Assessment the generalization of learned image reconstruction and the potential for transfer learning. Magn Reson Med, 8168):116-128, 01 2019.
Florian Knoll, Kerstin Hammernik, Chi Zhang, Steen Moeller, Thomas Pock, Daniel K. Sodick- son, and Mehmet Akçakaya. Deep-learning methods for parallel magnetic resonance imaging reconstruction. IEEE Signal Processing Magazine, 37(1):128- 140, 2020b.
Florian Knoll, Tullie Murrell, Anuroop Sriram, Nafissa Yakubova, Jure Zbontar, Michael G. Rabbat, Aaron Defazio, Matthew Muckley, Daniel K. Sodickson, C. Lawrence Zitnick, and Michael Recht. Advancing machine learning for MR image reconstruction with an open competition: Overview the 2019 fastMRI challenge. Magn Reson Med, Jun 2020c.
Dong Liang, Jing Cheng, Ziwen Ke, and Leslie Ying. Deep magnetic resonance image reconstruc- tion: Inverse problems meet neural networks. IEEE Sig Proc Mag. 37(1):141-15) 2020.
Michael Lustig, David I.. Donoho, Juan M. Santos, and John M. Pauly. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn Reson Med, 58: 1911118299., 2007.
Vishal Monga, Yuelong Li, and Yonina C. Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Process.
44, 2021.
Matthew J. Muckley, Bruno Riemenschneider, Alireza Radmanesh, Sunwoo Kim, Geunu Jeong, Jingyu Ko. Yohan Jun, Hyungseob Shin, Dosik Hwang. Mahmoud Mostapha, Simon Arberet, Dominik Nickel, Zaccharie Ramzi, Philippe Ciuciu, Jean-Luc Starck, Jonas Teuwen, Dim- itrios Karkalousos, Chaoping Zhang, Anuroop Sriram, Zhengnan Huang, Nafissa Yakubova, Yvonne W. Lui, and Florian Knoll. Results of the 2020 fastmri challenge for machine learn- ing MR image reconstruction. IEEE Trans. Medical Imaging, 40(9):2306-231) 2021.
Klaas P. Pruessmann, Markus Weiger, Markus Scheidegger, and Peter Boesiger. SENSE: Sensi- tivity encoding for fast MRI. Magn Reson Med, 42:952-9611 1999.
Klaas P. Pruessmann, Markus Weiger, Peter Bornert, and Peter Boesiger. Advances in sensitivity encoding with arbitrary k-space trajectories. Magn Reson Med, 46:638-65 651, 2001.
Yuhui Quan, Mingqin Chen, Tongyao Pang, and Hui Ji. Self2self with dropout: Learning self- supervised denoising from single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2020.
Philip M. Robson, Aaron K. Grant, nanth J. Madhuranthakam, Riccardo Lattanzi, Daniel K. Sodick- son, and Charles A. McKenzie, Comprehensive quantification of signal-to-noise ratio and g-factor for image-based and k-space-based parallel imaging reconstructions. Magn Reson Med, 60(4): 895-907, Oct 2008
Christopher Sandino, Joseph Y Cheng, Feiyu Chen, Morteza Mardani, John M Pauly, and Shreyas S Vasanawala. Compressed sensing: From research to clinical practice with deep neu- ral networks: Shortening scan times for magnetic resonance imaging. IEEE Signal Processing Magazine, 37(1):117-127, 2020.
Jo Schlemper, Jose Caballero, Joseph V. Hajnal, Anthony N. Price, and Daniel Rueckert. A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Trans. Medical Imaging, 37(2):99-5033 2018.
Ortal Senouf, Sanketh Vedula, Tomer Weiss, Alex Bronstein, Oleg Michailovich, and Michael Zibulevsky. Self-supervised learning of inverse problem solvers in medical imaging. In Do- main Adaptation and Representation Transfer and Medical Image Learning with Less Labels and Imperfect Data, pp. 111-119. Springer, 2019.
Assaf Shocher, Nadav Cohen, and Michal Irani. "Zero-shot" super-resolution using deep internal learning. In Proc IEEE CVPR, June 2018.
11

---

Published as a conference paper at ICLR 2022
Radu Timofte, Eirikur Agustsson, Lue Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, Kyoung Mu Lee, Xintao Wang, Yapeng Tian, Ke Yu, Yulun Zhang, Shixiang Wu, Chao Dong, Liang Lin, Yu Qiao, Chen Change Loy, Woong Bac, Jac Jun Yoo, Yoseob Han, Jong Chul Ye, Jae-Seok Choi, Munchurl Kim, Yuchen Fan, Jiahui Yu, Wei Han, Ding Liu, Haichao Yu, Zhangyang Wang. Honghui Shi, Xinchao Wang. Thomas S. Huang, Yunjin Chen, Kai Zhang, Wangmeng Zuo, Zhimin Tang, Linkai Luo, Shaohui Li, Min Fu, Lei Cao, Wen Heng, Giang Bui, Truc Le, Ye Duan, Dacheng Tao, Ruxin Wang. Xu Lin, Jianxin Pang, Jinchang Xu, Yu Zhao, Xiangyu Xu, Jin-shan Pan, Deqing Sun, Yujin Zhang, Xibin Song, Yuchao Dai, Xueying Qin, Xuan- Mhan-Phung Huynh, Tiantong Guo, Hojjat Seyed Mousavi, Tiep Huu Vu, Vishal Monga, Cristóvão Cruz, Karen O. Egiazarian, Vladimir Katkovnik, Rakesh Mehta, Arnav Kumar Jain, Abhinav Agarwalla, Ch V. Sai Praveen, Ruofan Zhou, Hongdiao Wen, Che Zhu, Zhiqiang Xia, Zhengtao Wang, and Qi Guo. NTIRE 2017 challenge single image super- resolution: Methods and results. In CVPR Workshops, pp. 1110-1121 1111. IEEE Computer Society, 2017.
Martin Uecker, Peng Lai, Mark J. Murphy, Patrick Virtue, Michael Elad, John Pauly, Shreyas S. Vasanawala, and Michael Lustig. ESPIRiT-an eigenvalue approach to autocalibrating parallel MRI: where SENSE meets GRAPPA. Magn Reson Med, 71(3):990-1001, Mar 2014.
Kamil Ugurbil, Junqian Xu, Edward Auerbach, Steen Moeller, An Vu, Julio Martin Duarte- Carvajalino, Christophe Lenglet, Xiaoping Wu, Sebastian Schmitter, Pierre-Françoii van de Moortele, John Strupp, Guillermo Sapiro, Federico De Martino, Dingxin Wang, Noam Harel, Michael Garwood, Liyong Chen, David A. Feinberg, Stephen M. Smith, Karla L. Miller, Sta- matios N. Sotiropoulos, Saåd Jbabdi, Jesper L. R. Andersson, Timothy Edward John Behrens, Matthew F. Glasser, David C. Van Essen, and Essa Yacoub. Pushing spatial and temporal resolu- tion for functional and diffusion MRI in the human connectome project. Neurolmage, 80:80-104, 2013.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. Proc. IEEE CVPR, June 2018.
Luca Vizioli, Steen Moeller, Logan Dowdle, Mehmet Akçakaya, Federico De Martino, Essa Yacoub, and Kamil Uğurbil. Lowering the thermal noise barrier functional brain mapping with magnetic resonance imaging. Nature communications, 12(1):1-15, 2021.
Burhaneddin Yaman, Seyed Amir Hossein Hosseini, Steen Moeller, Jutta Ellermann, Kåmil Ugur- bil, and Mehmet Akçakaya. Self-supervised learning of physics-guided reconstruction neural networks without fully-sampled reference data. Magn Reson Med, 84:3172-3191, Dec 2020.
12

---

Published as a conference paper at ICLR 2022
A
APPENDIX
Cor-PD b)
Cor-PDFS
AX-FLAIR
Ax-T2
a)
c)
d)
e)
Uniform Mask
Zero-Filled
Random Mask
Zero-Filled
Figure 7: Different contrast weightings and anatomies used this study: a) Cor-PD, b) Cor-PDFS, c) AX-FLAIR, d) Ax-T2, well undersampling patterns: e) Uniform, f) Random mask. Zero- filled images generated by uniform and random undersampling masks have coherent and incoherent aliasing artifacts, respectively. Coherent aliasing artifacts are generally harder to remove than inco- herent artifacts.
Epoch
Epoch
Epoch
Epach
DIP-
PSNR-20.953
PSNR:28-526 SSIM:D 702
PS
PSNR-26.318
PSNR-28-259
SSIM:0.788
SSIM:O
SSIM:0.697
SSIM:O 634
SSIM:0.692
Figure 8: Cor-PD Knee MRI reconstruction results across different epochs for DIP-Recon using uniform undersampling at R = 4. At the 25th epoch, the reconstruction suffers from artifacts, with the zoom-in in area showing texture that does not resemble the ground truth. With more epochs, this aspect of the reconstruction improves, but the reconstruction starts to suffer from noise amplification as the number of epochs increases. Hence, the 50th epoch was used the experiments.
13

---

Published as a conference paper at ICLR 2022
Ground Truth
25-55L K-1
Z5-55L K-10
25-55L Km25
25-55L K-50
25- K=100
b) Ground Truth
25-551, K-10
25-55L-TL, K-10
Figure 9: a) and b) show reconstruction results corresponding to the loss curves in Figure 2 and b, respectively.
Cor-PD Knee MAI, Rag 3/
Ground Truth
CG-SENSE
Supervised PG-DLI
SSDU PG-DLR
DIP-Recon
DIP-Recon-TL
25-SSL
25--SL-TL
PSNR:36.698
PSNR-10/535 SSIM:0.954
1.8M908895
055M
SSIM:0.953
841
SSIM:G.
551M809958
AX-FLAIR Beain MRI B-g E
29
35
P5N6-
P5NR
PSNR-29:230
P5N
35
SSIM:0.793
S51M:0.928
SSIM:0-721
SSIM:0:927
Figure 10:
Reconstruction results from R 4 with random undersampling on representative test slices from a) Cor-PD knee MRT and b) Ax-FLAIR brain MRI. CG-SENSE, DIP-Recon and DIP- Recon-TL suffer from noise amplification. Supervised PG-DLR, SSDU PG-DLR, ZS-SSL and 7S- SSL-TL all show artifact-free reconstruction quality, with similar quantitative metrics.
Table Average reconstruction times for single-instance reconstruction methods. The computation times were measured on the machines equipped with 4 NVIDIA V100 GPUs (each with 32 GB memory). While CG-SENSE and DIP methods have lower computational times, their reconstruc- tion quality is severely degraded hindering clinical usage. ZS-SSL-TL (K 10) provides 8-fold faster convergence time compared to ZS RSLSSL (K = 10). We note that ZS-SSL methods recon- struction times may further be reduced by means of more compact architectures. Additionally, the increased computational times may be tolerable within the workflow, for instance in clinical settings such high-resolution functional or diffusion MRI, where it is challenging to have high-quality high high-resolution data, while post-reconttruction analyses readily take hours days (Andersson & Stamatios N 2016 Vizioli
where image readings done the next day (Basha al. 2017).
et al. 2021
CG-SENSE
DeP-Recon
DIP-Recon-TI
ZS-SSL
ZS-SSL-TL
Average Time (sec)
<<1
75
75
640
85
14

---

Published as a conference paper at ICLR 2022
In-Domain MRI Reconstruction
Ground-Trume
Training Mask
Network Input
Reconstruction
Unrolled Network
Training
DC
Different Masks
Parameters,
Ground-Truth
Test Mask
Network Input
Reconstruction
Inference
DC
Cross-Domain MRI Reconstruction
Ground- Truth
Training Mask
Network Input
Reconstruction
Unrolled
Training
Differente Contrase
Parameters, 0
Ground-Truth
Network Unrolled Network
Reconstruction
DC
Figure 11: Test datasets may differ from the training datasets in terms of sampling pattern, SNR, contrast and anatomy. Such differences lead to sub-optimal reconstructions in the test datasets, raising robustness and generalizability concerns for translation of trained MRI reconstruction models to clinical practice.
Ground Truth
Supervised PG-DLR
DiP-Recon-TL
ZS-SSL-TL
b)
Figure 12: a) Using pre-trained Ax-Flair for Ax-T2 reconstruction. b) Using a pre- pre-trained Cor-PD (knee MRI) for Ax-Flair (brain MRI) reconstructions. Supervised PG-DLR fails generalize when contrast, SNR and anatomy changes, with residual artifacts (red arrows). DIP-Recon-11 TL. also shows artifacts. ZS-SSL-TL. successfully removes noise and artifacts.
15

---

Published as a conference paper at ICLR 2022
Table 3: Average PSNR and SSIM values on 30 test slices for the experiments associated with Figures 4-6 (in the main text) and 12. We note that ZS-SSL-TL successfully fine-tunes the network for each new dataset/instance regardless of the starting pretrained model. Interestingly, there are cases where out-of-domain transfer has slightly higher metrics than in-domain transfer. In these cases, since the metries already high, the slight quantitative differences do not affect the overall quality. However, the main difference in these cases is in the convergence/stopping time, where in- domain transfer typically converges/stops in ~2-fold fewer iterations than out-of-domain transfer.
Metrics
Supervised PG-DLR
DIP RecoR-eo--- TL
ZS-SSL-
SSIM
0.873
0.530
0.861
Figure
Banding Artifacts
PSNR
36.365
26.924
36. T2T
Figure
In- on-Domain Transter
SSIM
0.949
0.836
0.951
Different Mask
PSNR
39.167
34.093
40,088
Figure
b:
In-Domain Transfer-
SSIM
0.937
0.792
0.940
Different Rates
PSNR
38.262
32.658
38.301
Figure 64: Cross-Domain Transfer
SSIM
0.931
0.859
0.949
Knee-Different Contrast
PSNR
37.566
34,855
39,855
Figure 6/:
Anatomy Change-
SSIM
0.936
0.890
0.957
Trained on Brain & Tested on Knee
PSNR
37.494
35.458
40.407
Figure 124: Cross-Domain Transfer
SSIM
0.929
0.834
0.950
Brain-Diffcrent Contrast
PSNR
35.578
32.655
38.767
Figure 12b: Anatomy Change
SSIM
0,929
0.806
0.936
Trained on Knee & Tested on Brain
PSNR
36.242
30.849
37.134
16


Reviews:
[{"review_id": "g5Ud-KrzIdq", "reviewer": "Reviewer_mBMk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the paper is pleasant to read and rigorous it does not offer a significant improvement over ref [Yaman et al., 2020] which by the way is regrettably not included in the set of benchmark methods to which the submission is compared in the result tables of the experimental section.\n\nI am not sure to understand the argument w.r.t. transfer learning and single dataset reconstruction. The proposed approach also retrains the model on a new dataset with patient specific properties. I think that what the authors mean is that this dataset does not need to be supervised. But this is a built-in property of the model from [Yaman et al., 2020] from which the authors are building upon.\n\nThe proposed approach as well as supervised PG-DLR seem to fail to reconstruct texture feature visible in ground truth images (the reconstructed ones are smoother). This might be a problem for diagnosis or for other trained ML based models that rely on texture features such as radiomics and take MR images as inputs. Is this a known issue in MRI reconstruction in general or in DL based reconstruction ?", "summary_of_the_paper": "This submission deals with MR images reconstruction in a context where the raw data is under-sampled. This data (which is in the Fourier domain) can be under-sampled to accelerate the imaging exam and thus improve clinical workflow and it is thus a very relevant research topic. \n\nThe submission relies on recent deep learning models that explicitly incorporate some physical aspects of MR image acquisition (multiple coils, coil sensitivity and Fourier transform) and achieve MRI reconstruction based on supervised training examples. The training examples do not need to be (under-sampled / fully sampled) pairs. Indeed, some entries of under-sampled data can be deleted to create an input for which correct reconstruction of the deleted entries can be required through a computable loss term.\n", "summary_of_the_review": "Pros : \n- the paper is well written and ideas are clearly explained and stated.\n- the paper is technically sound.\n\nCons : \n- the contribution is too incremental compared to [Yaman et al., 2020] as it consists in re-using the Fourier domain partitioning idea in order to have a validation loss allowing to detect overfitting and stop learning.\n- patient specific training demands more computation time and resources than usual clinical workflow.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_jsMaQWIkYm", "reviewer": "Reviewer_pt6r", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "####################################################\n\nStrengths\n\n+ Overall, this is a very interesting paper with impressive experimental results. The proposed algorithm yields high-quality reconstructions and the rationale behind designing such an algorithm is very well-motivated by the authors. Indeed, it is important to come up with reconstruction algorithms that do not require large training sets, especially in medical imaging where creating datasets is a tedious task.\n\n+ The idea of self validation is also interesting since it mimics the presence of a validation set for training and prevents the network from overfitting.\n\n+ The robustness investigations are appealing and very related to the problem considered. Moreover, from the robustness perspective, any effort toward single-instance reconstruction is of great value provided that it results in more robustness. Regarding robustness evaluations, please see the reviewer’s comment below.\n\n################################################\n\nWeaknesses/comments\n\n- The paper states several times that ZS-SSL-TL significantly reduces convergence time. Yet what is missing is a computational comparison among single-instance reconstruction methods such as traditional sparsity-based, DIP-based, ZS-SSL, ZS-SSL-TL. This is important since one of the critical challenges of single-instance reconstruction algorithms is their inefficiency at the inference. Adding such an experiment would also make the paper even stronger.\n\n- Table 2 in the appendix contains very interesting results. However, there are several surprising scores achieved by ZS-SSL-TL. For instance, how come for ``trained on brain & tested on knee`` ZS-SSL-TL achieves 40.4 dBs, yet according to Table 1, ``trained on knee & tested on knee`` achieves 40.1 dBs? This is counter-intuitive in that it suggests pre-training a model on ``brains`` is marginally better than pre-training it on ``knees`` if one wants to get a good PSNR on ``knees``!\n\n- Page 7, Section 4.5, Paragraph 2, last sentence: ``ZS-SSL has no prior domain information and is inherently not susceptible to such generalizability issues.`` Is there any evidence to suggest that this is the case? For example, DIP-based methods have been suggested to be susceptible to distribution shifts although there’s no ``prior domain information`` involved. But their hyper-parameters are tuned on a specific domain. Doesn’t ZS-SSL rely on any tunable hyper-parameters that differ from one domain to another?\n\n- The reviewer perceives the point made in Figure 2 regarding automatic early stopping, however, there are the following two comments in terms of the consistency of the results:\n\n    - Second plot from the left: Can the authors provide intuition on why the curves are uncorrelated w.r.t K? Specifically, at the convergence (epoch 300), the curve for k=10 is on top of k=1, the curve for k=25 is on top of k=10, but then somehow the curve for k=50 is below k=25. Moreover, the breakout point of k=50 has a sudden drop from the breakout point of k=25; does this mean k=100 would go down even further?\n\n    - The right-most plot: The reviewer is unsure what the authors mean by ``ZS-SSL with TL converges faster compared to ZS-SSL`` in the caption, in that the validation loss implies no benefit comes with combining ZS-SSL and TL and it’s not a matter of convergence. \n\n- Table 1 suggests that DIP-TL performs strangely poorly compared to PG-DLR and ZS-SSL-TL. Does this mean that the pretrained network used for DIP-TL performs poorly and DIP has not been able to improve its performance, and thus the low score has nothing to do with the DIP itself?\n", "summary_of_the_paper": "In order to eliminate the need for large training sets, one can consider a transition from (1) fully-supervised to (2) self-supervised methods, and then from (2) self-supervised methods to (3) single-instance reconstruction methods. \n\nIn the context of accelerated MRI reconstruction which is considered in this work, the above categories translate into models that are trained based on (1) having access to a fully-sampled dataset, (2) having access to a dataset of under-sampled measurements, and (3) having access to only one under-sampled measurement. The paper targets (3), that is proposing a zero-shot learning approach for accelerated MRI reconstruction.\n\nThe algorithm is based on the idea proposed in paper [1] combined with building a dataset for the given sample in order to eventually perform self-supervised training on the synthesized dataset. In order to prevent overfitting, the authors propose a way to do automatic early stopping.\n\n[1] Yaman, Burhaneddin, et al. \"Self-supervised physics-based deep learning MRI reconstruction without fully-sampled data.\" 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 2020.", "summary_of_the_review": "Decision: Accept (6)\n\nThe reviewer finds the paper of great interest to the community and the thorough experimental analysis of the proposed algorithm is the main motivation for accepting the paper. However, \n\n(1) several concerns/comments mentioned above, and \n\n(2) the fact that the major difference between the proposed algorithm and the prior work [1] is the self-validation step combined with dataset synthesis \n\nprevent the reviewer from giving a higher score.\n\n[1] Yaman, Burhaneddin, et al. \"Self-supervised physics-based deep learning MRI reconstruction without fully-sampled data.\" 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 2020.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9lopGO94IIA", "reviewer": "Reviewer_19v3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and clear with great illustrations and only a few typos. It applies the combined principles of zero-shot learning [1] plug-and-play [2] iterative architectures and transfer learning to solve the MRI reconstruction problem on single images.\n\nThe idea of zero-shot learning is not new, in the particular context of imaging, it has long been known that the denoising problem could be solved by training on the very same image one would seek to denoise [3,4]. Plug-and-play architectures combine a general-purpose denoiser with a Tikhonov-like least-square solver to solve arbitrary inverse problems. MRI is often regarded as a classical inverse problem in the same range as CT reconstruction or deblurring. The range of solutions provided to solving inverse problems in medical imaging is too large to mention [5].\n\nFrom the methodological point of view, I see very little novelty in this article.\n\nThere are many aspects to using zero-shot learned approaches for denoising, in particular the fact that their regularity is not established and so they can only be applied for a limited number of iterations, and the need to use early-stopping. One contribution of this article is their proposed condition for early stopping, but it is not studied mathematically.\n\n\n\n\n\n[1] Romera-Paredes B, Torr P. An embarrassingly simple approach to zero-shot learning. InInternational conference on machine learning 2015 Jun 1 (pp. 2152-2161). PMLR.\n[2] S. Venkatakrishnan, C. A. Bouman and B. Wohlberg, \"Plug-and-play priors for model based reconstruction\", IEEE Global Conference on Signal and Information Processing, pp. 945-948, 2013.\n[3] Huang DA, Kang LW, Wang YC, Lin CW. Self-learning based image decomposition with applications to single image denoising. IEEE Transactions on multimedia. 2013 Oct 7;16(1):83-93.\n[4] Liu C, Szeliski R, Kang SB, Zitnick CL, Freeman WT. Automatic estimation and removal of noise from a single image. IEEE transactions on pattern analysis and machine intelligence. 2007 Dec 18;30(2):299-314.\n[5] Senouf O, Vedula S, Weiss T, Bronstein A, Michailovich O, Zibulevsky M. Self-supervised learning of inverse problem solvers in medical imaging. InDomain adaptation and representation transfer and medical image learning with less labels and imperfect data 2019 Oct 13 (pp. 111-119). Springer, Cham.", "summary_of_the_paper": "This article propose a \"zero-shot\" method for MRI reconstruction, which is a well-studied inverse problem. The method is based on the ideas of deep image prior, i.e. the ability of correctly sized neural networks to learn about the structure of a single image, sufficiently well for denoising tasks. This self-supervised learned network can then be used in a plug-and-play architecture to solve inverse problem with a variational approach, i.e. as if the learned denoiser were a Total Variation (TV) minimiser. Their denoiser can be improved in a transfer learning approach to benefit from a more complex network trained on similar images than those at hand, and fined-tuned on the image to be reconstructed\n\nThe authors go on to apply their plug-and-play architecture to solve the MRI reconstruction problem. They provide experiments and comparisons.", "summary_of_the_review": "The authors make a big deal of using TL, 0-shot and plug-and-play to achieve good results on MRI reconstruction. All of these elements are known and have been applied before to this problem. They do achieve better results than Senouf et al [5], thanks to a better TL regimen.\n\nThe paper is correct overall. The early stopping condition is debatable and not studied theoretically.\n\nOverall the paper is pretty good, but I think it does not clear the bar for acceptance at ICLR due to the lack of technical novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the problem of accelerated magnetic resonance imaging where the goal is to reconstruct an image from undersampled measurements. The paper proposes a zero-shot self-supervised learning approach for accelerated deep learning based magnetic resonance imaging. The approach partitions the measurements from a single scan into two disjoint sets, one set is used for self-supervised learning, and one set is used to perform validation, specifically to select a regularization parameter. The set that is used for self-supervised learning is then again split into two different sets, and a network is trained to predict the frequencies from one set based on the frequencies in the other set. This enables accelerated MRI without any training data. 
The paper evaluates on the FastMRI dataset, a standard dataset for deep learning based MRI research, and the paper compares to a trained baseline and an un-trained baseline (DIP). The paper finds their self-supervised method to perform very well compared to both and shows images that indicate excellent performance. It would have been even better to compare the method on the test set of the FastMRI competition to have a proper benchmark comparison.

Here is how the discussion went:
- Reviewer pt6r is supportive of acceptance, but notes a few potential irregularities, such as the method pre-trained on brain and tested on knees performing better than the method pre-trained on knees and tested on knees, and not providing a comparison of the computational cost. The authors added a table to the appendix revealing that the computational costs are very high, much higher than for DIP even. The reviewer was content with the response and raised the score.  

- Reviewer mBMk argues that the contribution is too incremental compared to prior work, in particular relative to the results of [Yaman et al., 2020], and also argues that the idea of partitioning the measurements is not new. The authors argue in response that their approach of partitioning the measurements is new, and the reviewer was inclined to raise the score slightly, but still thinks that the novelty on the technical ML side remains limited, and doesn't want to back the submission too much, and did not raise the score at the end in the system.

- Reviewer 19v3 has the concern that the all elements used (transfer learning, plug-and-play, etc) are well known techniques and have been applied before to MRI, and therefore thinks that the paper does not clear the bar for acceptance. The paper points out that while those ideas might be applied for the first time to MRI, they have been used before in other image reconstruction problems, in particular denoising. 

I've read the paper in detail too, and am somewhat on the fence: I think it's very valuable to see that a clever application of self-supervised learning works so well for MRI. I agree with the reviewers that the technical novelty is relatively small, but on the other hand this is the first time that I see self-supervised learning being applied that successfully to MRI. I don't share the concern about novelty --- yes, the paper's approach builds on prior work, but it's not clear from the literature how well such a well tuned self-supervised learning approach would work. 
What I would have liked to see in addition to the experimental results is a proper evaluation on the FastMRI dataset: An advantage of the FastMRI dataset is that it provides a benchmark and if researchers evaluate on that benchmark (on the testset/validation set) we can compare different methods well. The paper under review doesn't do that, it only evaluates on 30 test slices, and thus it's hard to benchmark the method. Also, the paper would benefit from more ablation studies.

In conclusion, I would be happy to discuss this paper at the conference, and think that other researchers in the intersection of deep learning and inverse problems would be too, and therefore recommend acceptance.
Decision: Accept (Poster)