Erase to Enhance: Data-Efficient Machine Unlearning in MRI Reconstruction

Authors: yuyang  xue
jingshuai  liu
steven  mcdonagh
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 18:37:41

Abstract: Machine unlearning is a promising paradigm for removing unwanted data samples from a trained model, towards ensuring compliance with privacy regulations and limiting harmful biases. Although unlearning has been shown in, e.g., classification and recommendation systems, its potential in medical image-to-image translation, specifically in image reconstruction, has not been thoroughly investigated. This paper shows that machine unlearning is possible in MRI tasks and has the potential to benefit for bias removal. We set up a protocol to study how much shared knowledge exists between datasets of different organs, allowing us to effectively quantify the effect of unlearning. Our study reveals that combining training data can lead to hallucinations and reduced image quality in the reconstructed data. We use unlearning to remove hallucinations as a proxy exemplar of undesired data removal. Indeed, we show that machine unlearning is possible without full retraining. Furthermore, our observations indicate that maintaining high performance is feasible even when using only a subset of retain data. We have made our code publicly accessible.

Main paper: 
 introduction: 
 As the dependence on data-driven learning grows, model generalisation remains a key challenge. Common strategies include the aggregation of multiple data sets to expand the available data distributions, as a means of improving generalisation (Teney et al., 2021) . However, such data agglomeration raises concerns about privacy, perpetuation of biases, and, in the case of image reconstruction, may generate hallucinations. In particular, successful magnetic resonance imaging (MRI) reconstruction of distinct organ types, under a single trained model, requires generalisation abilities. We corroborate recent work and find that model generalisation can be strengthened using data aggregation (Sect. 4.1). However, such aggregation leads to hallucinations that manifest themselves as false structures and artifacts in the reconstructed images (Darestani et al., 2021) . Crucially, such resulting effects harm the clinical diagnostic value (Zbontar et al., 2018) . Figure 1 shows an example of how learning from combined brain and knee data training can result in unwanted hallucinations in structures such as the corpus callosum, which could potentially lead to misdiagnoses.While machine unlearning has been shown to remove unwanted data and classes in other tasks (e.g., classification), research on the integration of machine unlearning into image reconstruction remains, at best, nascent (Nguyen et al., 2022) . This paper sets up a protocol to analyse what models can learn about datasets from different distributions and uses this protocol to study machine unlearning. The protocol consists of training MRI reconstruction models with data combined from different anatomies, as obtained with multiple coils, which we have found can produce unwanted artefacts (see Figure 1 ). We find the "knowledge gap" between the two datasets to effectively quantify this anatomy shift. We then propose suitable unlearning models that can remove such a shift. Thus, we use this setup as a proxy exemplar for unwanted data removal 1 . We highlight the task difficulty: existing well-defined settings of unlearning focus on classification and whole class removal. Instead, our task not only deals with a generative image-to-image task but also requires the effective removal of misgenerated structure whilst accurately preserving the anatomical features of a patient's body. To the best of our knowledge, we are the first to establish unlearning algorithms for a reconstruction task c.f. previously considered classification tasks. Our contributions can be summarised as follows: (1) We propose a formal problem formulation and definition of machine unlearning within the MRI reconstruction domain. (2) We adapt machine unlearning approaches to MRI reconstruction and verify that successful machine unlearning allows for effective artifact removal and improves reconstruction quality. (3) In contrast to existing approaches, which necessitate access to an entire dataset, we demonstrate that unlearning algorithms can operate effectively with only a restricted fraction of data and achieve data-efficient unlearning for MRI reconstruction. 
related works: 
 MRI Reconstruction reconstructs medical images from acquired Fourier domain samples, known as the k-space. Deep learning reconstruction enables high-quality image reconstruction from undersampled k-space data. This is achieved by training networks to capture complex patterns and features within MRI data (Schlemper et al., 2017) , learning mappings between the under-and fully sampled k-space (Akçakaya et al., 2019) , or directly in the image domain (Wang et al., 2016) . However, recent research shows that such models produce artifacts, overlook small structural changes, and have a decrease in performance under distribution shifts (Antun et al., 2020; Darestani et al., 2021) . Inspired by these challenges, we set up a proxy exemplar of data removal. Machine Unlearning aims to address the need for data privacy and regulatory compliance (Liu and Tsaftaris, 2020; Xu et al., 2023) . It is the process of removing the influence of a set of data from a trained model, effectively "forgetting" this set without retraining the model from scratch (Ginart et al., 2019; Su et al., 2022) . Numerous studies have emerged using data-based (Bhadra et al., 2021; Tarun et al., 2023; Graves et al., 2021) or model-based algorithms (Neel et al., 2021; Chourasia and Shah, 2023; Jia et al., 2023) . However, these approaches are mainly suited to classification tasks. Parallel to this work, an image-toimage machine learning framework for generative models was developed Li et al. (2024) . The idea is simple and borrows from a class of unlearning algorithms for classification tasks (see noisy labelling approaches discussed in Section 4): they introduce random noise during image generation (of the diffusion model) to degrade the model performance on the forget data. Similar to many unlearning methods, it relies on having access to the full original database, which can be unrealistic in medical practice. Machine Unlearning in Medical Imaging Hartley et al. (2023) showed that networks could pose a risk of retaining and potentially revealing the distinctive characteristics of patients. Liu and Tsaftaris (2020) used the Kolmogorov-Smirnov distance to detect whether a model has used/forgotten a query dataset. Su et al. (2022) explored patient-wise forgetting and revealed that forgetting medical image data from a patient is harder than other vision tasks. These works highlight several challenges in medical imaging settings: models can easily learn unwanted relationships, and there is considerable similarity between medical data. While they don't present algorithms for image reconstruction, their findings help enforce why a suitable proxy exemplar is required,2 why unlearning in image-to-image tasks is hard and why a delicate balance of unlearning and performance is required. Furthermore, as fully-sampled k-space data are extremely large and rare, unlearning methods that can relax the need for complete access to all training data should be explored.  
methodology: 
 Our main goal is to effectively investigate machine unlearning for MRI reconstruction. We set up a proxy exemplar where the task is to reduce artefacts and hallucinations without reducing diagnostic precision. Utilising unlearning algorithms, we systematically erase the influence of a forget dataset from the model by evaluating image performance. We evaluate the integrity of reconstructed images and found that the performance of the models remains uncompromised after unlearning. Lastly, we demonstrate the practical implementation of our methodology on MRI data that limits the size of the training set, highlighting its potential for real-world application. The overview of the pipeline is shown in Figure 2 .We address the scenario where a reconstruction model G, parameterised by θ, is initially trained on a combined dataset that comprises both a retain set D r and a forget set D f , optimised via the loss function J (θ). We posit the existence of an ideal oracle model Ĝ, exclusively trained on D r and thus uncontaminated by D f . The objective of the unlearning algorithm A, applied to G, is twofold: (1) to preserve or enhance the performance of the model in the retain set distribution D r , ideally achieving the upper performance bound of Ĝ and (2) based on the premise of not destroying structural integrity, to eliminate the impact of D f as much as possible, that is, to reduce the performance in the forget distribution. We call the unlearned model G U . In practical applications, accessibility to the training dataset is usually restricted due to privacy or large storage requirements, the oracle model Ĝ cannot be retrained, forcing the unlearning algorithm A to operate on G with the D f alone and with limited access to a subset of retain data D ′ r . We selected two test sets to measure the effect of unlearning, namely D t r of the retain distribution and D t f of the forget distribution.We present a proof-of-concept protocol for machine unlearning for MRI reconstruction. We treat the brain as retain data D r and the knee as forget data D f . An oracle model Ĝ trained exclusively on multi-coil brain MRI data D r establishes the gold standard for evaluating performance for unlearning algorithms. To simulate a real-world scenario, we then train the original model G on a combined dataset comprising data from D r and D f , each of which has a different anatomy. Subsequent application of unlearning algorithms removes the influence of knee data incorporated during the original training phase.There is no precedent in evaluating the quality of forgetting in MRI reconstruction; therefore, we employ image quality assessment metrics as indicators of successful unlearning. An increase in performance on D r , accompanied by a decrease on D f , suggests that the model is redirecting its attention towards D r , the desired in-distribution data. Compared to pre-unlearning performance, this shift in the model manifests itself through improved image quality both quantitatively and qualitatively. To evaluate the efficacy of the unlearning process, we performed comparative analyses using several baselines and unlearning methods. The goal is to discern which algorithm achieves a balance of efficient unlearning while maintaining, or ideally enhancing, the model performance on D t r . Since the unlearning process should be acute and accurate, rather than retraining the entire model, we restrict the unlearning steps to 10% of the training epochs. This proof-of-concept underscores the potential of machine unlearning to address bias removal in AI models, and serves as a basis for future research in deploying machine unlearning within medical imaging domains.We introduce machine unlearning into MRI reconstruction, adapting three major techniques, originally developed for classification tasks, to suit an image-to-image reconstruction context. All algorithmic details are found in the Appendix.Fine-tuning (FT) is the most widely deployed unlearning strategy, often used to address distribution shifts and adapt models to new distributions. In machine unlearning, the fine-tuning process is only implemented on D r , using a loss function J (θ, x, y), asEQUATIONwhere J (θ, x, y) denotes the reconstruction loss, normally L 1 between the undersampled input x and the ground truth y, and θ refers to the model parameters. We use the same notation for the methods in the following. However, as fine-tuning does not impose any penalty on the forget data, the unlearning tends to be less efficient than other methods. Gradient Ascent (GA) maximises the training loss w.r.t. forget samples, effectively decreasing model accuracy for knowledge removal (Halimi et al., 2022; Warnecke et al., 2021) . However, without any restrictions, the model tends to forget all the knowledge obtained before unlearning. We hence add ℓ 1 -regularisation (GA-ℓ 1 ) to constrain the weights to prevent divergence, which encourages sparsity in the model weights during training (Jia et al., 2023) . Sparsity can lead to simpler and more interpretable models that may perform better when it comes to generalisation on unseen data by reducing overfitting. The regularised GA loss is defined byL GA ℓ 1 = -J (θ, x f , y) + γ∥θ∥ 1 , x f ∈ D f .(2)Noisy Labelling (NL) minimises the training loss by reassigning the label of forget samples by adding Gaussian noise with a hyperparameter λ indicating the amount of noise, thus removing the mapping of the input to the OOD images (Graves et al., 2021; Gandikota et al., 2023) . In MRI reconstruction, a false mapping to the target would lead to a considerable drop in global performance. We change the forget set labels as follows:L N L = J (θ, x f , [y + λN (0, I)]), x f ∈ D f .(3) 
dataset: 
 We use FastMRI dataset (Knoll et al., 2020) to reconstruct multiple coil brain and knee images. We randomly selected T2-weighted brain volumes, yielding 5,898 slices in 1,000 volumes as the full retain dataset D r . The forget set D f has 536 randomly selected knee image slices. We adhered to a retain-to-forget set ratio of 10:1. For validation, a separate subset was created by randomly choosing 1,198 slices. Two test sets are 100 volumes of brain and knee data, respectively. For data processing, an equispaced Cartesian sampling pattern was applied to simulate unacquired k-space lines. The acceleration rate and the centre fraction rate were set to 8-fold and 0.04, respectively.We selected the E2E-VarNet model (Sriram et al., 2020) , which processes masked multi-coil k-space data as input, as the main network for reconstruction. The Sensitivity Map Estimation (SME) module inside computes sensitivity maps from the input k-space, together with a sequence of UNet cascades that iteratively refine outputs. We follow the training protocol of the original implementation, with 12 cascades, resulting in 29.9M total parameters. Models were trained using the Adam optimiser with a learning rate of 0.001 over 50 epochs without any data augmentation. We used the PyTorch Lightning framework on a NVIDIA A100 Tensor Core GPU. Image quality was evaluated using peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) (Wang et al., 2004) as metrics.We analysed the experiments in three dimensions (Jia et al., 2023) : accuracy on brain test data D t r as brain test accuracy (BTA), aiming to demonstrate the capacity to retain the learned target knowledge, accuracy on knee test data D t f , denoted by knee test accuracy (KTA) and serving as a metric to gauge the efficacy of unlearning algorithms, and additionally accuracy on forget data D f as unlearning accuracy (UA) which is used to confirm the unlearning quality when combining it with KTA. We also added run-time efficiency (RTE) as another metric for the evaluation of the unlearning algorithm. Performances are highlighted in Table 1 and Figure 3 shows a radar chart of comparison of all algorithms. Oracle vs. Original To show the domain gap existing between brain and knee data and hence quantify the influence of the forget set D f on the reconstruction model, we compared the original model G, trained on both D r and D f , and the oracle model Ĝ, trained solely on brain data D r to represent an upper bound for unlearning methods. We can see from Table 1 that these two models show differences in terms of evaluation metrics for each organ, with the oracle Ĝ better on brain data and worse on knee data. This is expected given the domain shift, but the gap of about 5dB on knee data implies the distribution dissimilarities between the two organs. The findings highlight the need for machine unlearning to counteract domain-related issues and maintain reliable image reconstruction quality.Fine-tuning does not unlearn Fine-tuning aims to unlearn the forget set by shifting the learned data distribution towards the target domain. Two levels of fine-tuning are FT and NL-FT achieve the best, closest to oracle with the highest RTE.selected. One is to fine-tune G on the whole D r , termed Full FT, which requires us to have access to large-scale training data which is impractical. The other is to fine-tune the model on a small subset of target data, D ′ r , termed FT. Both Full FT and FT marginally enhance the performance of the model on the target brain dataset. However, the ability to reconstruct knee images remains unaffected or, in the case of FT, even improved. This indicates that fine-tuning is insufficient in discarding knowledge of the forget set. These findings have inspired the integration of unlearning algorithms into the fine-tuning process.Combining unlearning and fine-tuning We used GA-ℓ 1 and NL as unlearning algorithms. GA-ℓ 1 fine-tuned with D ′ r is referred to as GA-ℓ 1 -FT, and the other is designated as NL-FT. We noted from Table 1 that NL and GA-ℓ 1 lower the metrics for the knee test data, suggesting their effectiveness in the unlearning of domain knowledge. However, these methods also substantially deteriorated the reconstruction fidelity of brain images, indicating that unlearning algorithms, if applied solely with D f , can inadvertently compromise vital information in the target domain. Methods GA-ℓ 1 -FT and NL-FT combine fine-tuning to maintain the learned knowledge in the target domain during their unlearning steps, using the knowledge from D ′ r to steer the unlearning direction toward the retain data distribution, as illustrated in Appendix Figure 4 . According to the results in Table 1 , both methods can effectively achieve model unlearning in a reasonable range, with NL-FT, which could be a promising unlearning method, showing a superior ability to maintain the reconstruction integrity compared to GA-ℓ 1 -FT. More details are provided in Appendix Table 2 .Do we need all retain data? We evaluate whether having full access to the retain dataset results in a more effective unlearning compared to having limited access practically to a portion of the retain dataset. For these experiments we used a smaller retain dataset D ′ r , at percentages 1%, 5%, 10%, 20%, 50% of the full retain dataset D r . We evaluated performance considering two primary factors: run-time efficiency and PSNR of the brain test set D t r . We expect that the more retain (in-distribution) data we used for fine-tuning, the better the results should be. Unexpectedly, a decrease in PSNR was observed when the model was fine-tuned using the complete D r . Figure 5 in the Appendix depicts the results on NL-FT and GA-ℓ 1 -FT using a second-degree polynomial fitting. It shows that, to some extent, as the amount of retained set data increases, the quality of image reconstruction does not improve, but rather deteriorates. More information is in Appendix Table 3 . 
conclusion: 
 Our study is the first to investigate the application of machine unlearning to MRI reconstruction, marking significant progress in medical imaging research with respect to artifacts reduction without comprehensive retraining. Our findings underscore the potential of unlearning approaches to improve image quality and preserve data integrity by fine-tuning in a data-efficient way. Further investigation is necessary to understand the decrease in using larger retain datasets and to determine the optimal subset size for successful unlearning. This work lays the groundwork for future research to extend these methods to more complex datasets and enhance their efficiency in clinical settings, facilitating accuracy for clinical decision-making and the ethical standards demanded by society.  
 
 Appendix: We found that unlearning a "single patient's k-space data" is impossible to effectively quantify, it would not guarantee creating a significant performance gap to be observed. 


Decision: The reviewers find the work interesting and the results promising. I concur with the reviewers that this is an interesting approach to a problem that has not been heavily explored yet and that the paper would be of interest to the MIDL community.

Reviews: review: paper_summary: paper_summary: The authors employ unlearning as a means to eliminate hallucinations, serving as a representative example of removing undesired data. Their findings demonstrate the feasibility of machine unlearning without the necessity for complete retraining, and they suggest that it is possible to maintain high performance levels even when utilizing only a subset of retained data.
main_review: main_review: Same that asked in rebuttal: The paper provides a thorough and comprehensive examination of unlearning techniques applied to MRI reconstruction. It represents a potential pioneering contribution in the field of unlearning for MRI reconstruction. However, there are certain details omitted in the methodology description in section 3.3 that, if included, would enhance comprehension and reproducibility.
strength_weakness: strength_weakness: "high performance levels" is not strictely defined
Although their mentioned it as future work, there is not much discussion on why, when the amount of retained set data increases the quality of image reconstructions does not improve
May be this due to some effect of the l1 norm in 2?
How are the model parameters?
questions: questions: The paper provides a thorough and comprehensive examination of unlearning techniques applied to MRI reconstruction. It represents a potential pioneering contribution in the field of unlearning for MRI reconstruction. However, there are certain details omitted in the methodology description in section 3.3 that, if included, would enhance comprehension and reproducibility.
paper_summary: paper_summary: The authors tackle the problem of unlearning, ie removing biases in the model from a subset of a model’s training data, in MRI.

To do so, the authors use the task of image reconstruction from k-space. Several unlearning methods are tested, as well as varying the amount of unlearning data. The authors report performance relative to an “unbiased” model.
main_review: main_review: The authors have thoroughly responded to issues raised in the initial review. The motivation behind the problem as posed and the datasets used should be obvious enough for a new reader. The performances of the proposed methods lay the groundwork for future research.
strength_weakness: strength_weakness: Unfortunately, the task of unlearning in MRI **as presented** in this work feels artificial. Several potential applications are hinted at throughout the paper, such as removing artefacts ("hallucinations"), removing bias, or preventing patient data recovery. However, none of these applications are actually explored and the unlearning is presented as a goal in itself, which as far as the reviewer is aware, does not reflect a real life problem. The experiment considered, where a researcher would enhance a brain images dataset with knee images, does not represent a meaningful scenario. The considered evaluation metrics do not tell us if hallucinations are removed or if knee images can no longer be recovered, for example.

Moreover, the presented results are unconvincing and none seem to outperform the others. Most considered methods either provide no impact wrt their "oracle" network, or catastrophically degrade the results on the target "retain" dataset. Therefore, even though the problem is novel, there is seemingly no grounds for a potential future avenue of research.
questions: questions: Most importantly, the authors should put forward the motivations of exploring unlearning, and ideally review the aim of the article so that applications of unlearning are better put forward. Actual applications of unlearning could be explored in the experiments and results.  Please address the comments in the section above.
paper_summary: paper_summary: The manuscript „Erase to Enhance: Data-Efficient Machine Unlearning in MRI Reconstruction “ describes a modern and very innovative approach of image reconstruction in magnetic resonance imaging (MRI). 
Objects of this approach are twofold: 1) An ideally identical performing reconstruction model for the wanted data (retain set D_r, here brain data) and 2) an efficient untraining method such that the model G^U eliminates as much as possible falsely structured introduced by through presence of the unwanted data (forget set D_f, here knee data) during the training. In addition to the direct benefits of unlearning a positive side effect is training data reduction, i.e. limiting the dataset size of large k-space data for training, is pointed out. Overall, the manuscript is in most parts well written and scientifically sound to show a good understanding of the topic. The main reasoning why untraining should be done in contrast to other methods such as fine-tuning is not very strong backed by the findings in this work.
main_review: main_review: The authors addressed my raised points and answered my questions. With that crucial points were clarified thanks. Overall, I will remain with my initial rating for weak accept to be presented at MIDL 2024.
strength_weakness: strength_weakness: Some part could benefit from clearer writing, e.g. in 4.2 as in Sriram 2020 your NN ended with k-space data, didn't it? Please point out, that you use FFT and sum-of-squares (or other) method to compute an image from k-space and combine multi coil data. Also, to emphasize that PSNR and SSIM are applied in image (and not k-) space. Please also explain, how you handle complex-valued k-space data.
questions: questions: -	Was benchmarking again conventional reconstruction methods (not a trained model) as gold standard done? 
-	Couldn’t be the following alternative approach more explainable and straight forward:
Perform standard/low level image reconstruction, do image classification, exclude unwanted anatomies and then simply train on wanted and ‘good’ data the full reconstruction model?
-	Table 1: To me the Fine-tuning approach seems to result in the best overall metrics. Could you please comment on this? Why would I need unlearning then? In the paragraph below you refer to a Sub FT set. Is this identical to FT in Table 1? If so, I still can not completely follow you interpretation of the results which show – just by considering the overall quantitative metrics – that Fine-Tuning works best.

Comments: title: Rebuttal
Rebuttal to Reviewer tpqe (2)
Rebuttal to Reviewer tpqe (1)
Rebuttal to Reviewer fLbn (2)
Rebuttal to Reviewer fLbn (1)
Rebuttal to Reviewer aeoX
comment: We sincerely appreciate reviewer aeoX's thoughtful review and the time you invested in evaluating our manuscript. We are glad to hear that our rebuttal has satisfactorily addressed concerns you raised, and we are grateful for your concurrence.
Thank you for the rebuttal, adding the definition in section 3.2 and add the details of the methodology in Annex I. The authors have effectively addressed the raised concerns, and I concur with the majority of the responses.
We sincerely appreciate reviewer fLbn's thoughtful review and the time you invested in evaluating our manuscript. We are glad to hear that our rebuttal has satisfactorily addressed your concerns, and we are grateful for your agreement with our responses!
Thank you for the thorough and well presented rebuttal. The authors have done a good job of addressing the concerns raised and I agree with most of the responses. I have no further comments.
Continued from `Rebuttal to Reviewer tpqe (1)`.

---

**Q5**: *Couldn’t be the following alternative approach more explainable and straight forward: Perform standard/low level image reconstruction, do image classification, exclude unwanted anatomies and then simply train on wanted and ‘good’ data the full reconstruction model?*

**A5**: While the suggestion is indeed a valid strategy under certain circumstances, we would like to underscore that it addresses a different problem setting from the one our paper targets. Our work is rooted in the concept of machine unlearning, which is particularly relevant to cases where a machine learning model has already been deployed and trained on a dataset that includes both desired and undesired data. The key challenge we address is the ability to remove the influence of unwanted data from this pre-trained model without retraining it from scratch, a scenario that is becoming increasingly common in practice due to constraints related to time, computational resources, and accessibility of the original training data, etc.

---

**Q6**: *Table 1: To me the Fine-tuning approach seems to result in the best overall metrics. Could you please comment on this? Why would I need unlearning then? In the paragraph below you refer to a Sub FT set. Is this identical to FT in Table 1? If so, I still can not completely follow you interpretation of the results which show – just by considering the overall quantitative metrics – that Fine-Tuning works best.*

**A6**: Thank you for pointing out this. As other reviewer also pointed out this, we here apologise for the misunderstanding.  The inconsistency of `Sub_FT` and `FT` will be amended in the manuscript to ensure clarity and uniformity. The term `Sub-FT` mentioned in _Section 4.3_ is intended to correspond to `FT` as in _Table 1_. Similarly, `Full-FT` in both the text and the table refers to the same fine-tuning method applied across the full dataset. We will rectify this oversight to align the narrative of _Section 4.3_ and _Table 1_.
The authors thank reviewer tpqe for the time and effort for reviewing our work and the constructive suggestions. Here is the rebuttal to the questions.

--- 

**Q1**: *In 4.2 as in Sriram 2020 your NN ended with k-space data, didn't it? Please point out, that you use FFT and sum-of-squares (or other) method to compute an image from k-space and combine multi coil data. Also, to emphasize that PSNR and SSIM are applied in image (and not k-) space. Please also explain, how you handle complex-valued k-space data.*

**A1**: Our model takes as input the masked $k$-space data, denoted as $\mathbf{k}_0$, which is originally in complex form and is converted into a two-channel real-valued representation. Using an end-to-end variational network (E2E-Varnet) backbone, the network refines the input to generate a reconstructed $k$-space representation, $\mathbf{k}$. Following the refinement process, we then convert the final k-space output back into image space. This conversion is performed by applying an inverse fast Fourier transform (IFFT), which transforms the k-space data into its corresponding spatial domain. To reconstruct the final image representation from the multicoil MRI data, we employ a root-sum-of-squares (RSS) combination method across the coil images for each pixel, yielding a single coherent image that reflects the combined signal.

We wish to emphasise that all evaluation metrics, specifically `PSNR` and Structural Similarity Index Measure (`SSIM`), are computed on these reconstructed image-space data.

---

**Q2**: *p.5 “…we then train the original model G on a combined dataset…” suggesting to rename it to “…train the mixed model G on…”*

**A2**: While we appreciate the intention behind the suggestion to rename "the original model $G$" to "the mixed model $G$," we would like to explain our rationale for the terminology chosen within our text. The terms "original model," "unlearned model," and "oracle model" are consistently used in our manuscript as per the naming conventions established in notable previous works [1, 2] and guidelines from authoritative challenges such as the Google Unlearning Challenge [3]. 

To further improve readability, we will add a footnote or parenthetical note at the first instance of "original model" on _page 5_, explaining that this term refers to the "mixed" state of $G$ when trained on a combined dataset of both in-distribution and out-of-distribution data. 

---

**Q3**: *Suboptimal definitions of variables in Eq. 1 and 2. please use subscripts to indicate whether x is from the retain or forget set. In the end the mathematical expression should be clearly interpretable such that a user knows, how losses differ and what information they take into account.*

**A3**: Agreed. We update the manuscript and thank the reviewer for the suggestion.

---

**Q4**: *Was benchmarking again conventional reconstruction methods (not a trained model) as gold standard done?*

**A4**: Thanks for the comment. We later tested both multicoil brain and knee data in an acceleration rate of `8x` using `GRAPPA` to reconstruction and coil sensitivities were estimated using `ESPIRiT`. Since the sampling pattern was pseudo-equispaced, multiple `GRAPPA` kernels were used to calculate the `GRAPPA` images. The final `SSIM` is `0.844` for brain images and `0.693` for knee images in average, which is similar but inferior to the oracle model $\hat{G}$. We omitted the conventional reconstruction method as we were doing *unlearning*, and comparing with a learning-based model is more meaningful in our context.

---

To be continued on `Rebuttal to Reviewer tpqe (2)`.

---

**Reference**:

[1]. Liu, Y., Ma, Z., Liu, X., Liu, J., Jiang, Z., Ma, J., Yu, P., Ren, K., 2021. Learn to Forget: Machine Unlearning via Neuron Masking.

[2]. Kurmanji, M., Triantafillou, P., Hayes, J., Triantafillou, E., 2023. Towards Unbounded Machine Unlearning. Presented at the Thirty-seventh Conference on Neural Information Processing Systems.

[3]. Google, 2023, NeurIPS 2023 Machine Unlearning Challenge. Available at: https://unlearning-challenge.github.io/assets/data/Machine_Unlearning_Metric.pdf (Accessed: 15 March 2024).
Continued from `Rebuttal to Reviewer fLbn (1)`.

---

**Q3**: *Moreover, the presented results are unconvincing and none seem to outperform the others. Most considered methods either provide no impact wrt their "oracle" network, or catastrophically degrade the results on the target "retain" dataset. Therefore, even though the problem is novel, there is seemingly no grounds for a potential future avenue of research.*

**A3**: We wish to clarify the multi-dimensional approach we used to evaluate the effectiveness of our unlearning method. Specifically, our evaluation framework considers: 1) accuracy on in-distribution data (brain data), 2) the degree to which model performance on out-of-distribution data (knee data) approximates oracle network performance, and 3) the run-time efficiency of the model. This allows for a nuanced understanding of the advantages and trade-offs intrinsic to unlearning. Upon reviewing _Table 1_ and _Figure 3_, we have observed that the Noisy Label Fine-Tuning (`NL-FT`) method demonstrates a balanced profile across these metrics. While `NL-FT` ranks third in Brain Test Accuracy (`BTA`), it secures the first position in Knee Test Accuracy (`KTA`) and Unlearning Accuracy (`UA`), while maintaining relatively high Run-Time Efficacy (`RTE`). This profile highlights a significant step forward in unlearning to achieve a balance across multiple critical dimensions. The improvements in unlearning offer a compelling argument for continued exploration in this field. The potential to refine these models further and adapt the unlearning techniques to more dynamically reallocate a model's representational capacity presents an exciting and viable future research avenue. Our work serves to lay the groundwork for such advancements in the understanding and application of unlearning.

---

**Q4**: *Run-time efficiency (RTE) is introduced as a metric in section 4.3 but is never explained. Figure 3 displays it for some considered method but does not bring anything to a reader not familiar with the metric. Figure 3, with our without RTE, seems redundant with Table 1 and could be excluded from the article.*

**A4**: In _Section 4.3_, we introduced run-time efficiency (`RTE`) as a metric to quantify the operational efficiency of our unlearning model. We will amend the paper to include a clear and comprehensive explanation of `RTE`. Run-time efficiency will be defined as *the time taken by the model to carry out an unlearning process, divided by the total time used for retraining the whole model*.

---

**Q5**: *4.3 refers to Sub-FT and Full-FT yet table 1 considers FT and Full-FT.*

**A5**: Thank you for pointing out the discrepancy in the terminology used in _Section 4.3_ and _Table 1_. We acknowledge this inconsistency and will amend this in the manuscript to ensure clarity and uniformity. The term `Sub-FT` mentioned in _Section 4.3_ is indeed intended to correspond to `FT` as presented in _Table 1_. Similarly, `Full-FT` in both the text and the table refers to the same fine-tuning method applied across the full dataset. We will rectify this oversight to align the narrative of _Section 4.3_, with narrative of _Table 1_, making it clear that `Sub-FT` and `FT` are synonymous.

---

**Q6**: *The results in Figure 4 seem very sparse and could do with more samples.*

**A6**: _Figure 4_ shows an ablation study of how many retain data are utilised to keep the unlearned model in-distribution, while the x-axis is the related unlearning time with respect to the data portion and the y-axis shows the corresponding brain test accuracy in `PSNR`. The sparse image shows that with the increase of data samples, the time required will increase significantly, as we choose some representative portion of the data. We will add more data portions as a test of unlearning efficiency.
The authors thank reviewer fLbn for the time and effort for reviewing our work and the constructive advice. Here is our reply to the comments.

---

**Q1**: *Unfortunately, the task of unlearning in MRI as presented in this work feels artificial… does not reflect a real life problem… does not represent a meaningful scenario... the authors should put forward the motivations of exploring unlearning, and ideally review the aim of the article so that applications of unlearning are better put forward.*

**A1**: Thank you for raising the question of the motivation. We reemphasize that our work is grounded in emerging challenges within the field of MRI analysis. A growing body of research has identified the phenomenon of hallucinations in MRI reconstructions — artifacts that can severely distort clinical interpretations [1, 2, 3]. These hallucinations are due to the instability of AI-based reconstruction methods, highlighting a critical issue that can directly impact patient care.

Currently, we lack both controllable methods capable of creating realistic artifacts and further, oracle detectors capable of reliably detecting and locating artifact pre-existence. We acknowledge that our nascent experimental setup therefore leverages artificially induced artifacts that none-the-less, enable us to illustrate and explore method abilities to unlearn synthetic visual artifacts that we believe provide a useful proxy for meaningful scenarios.

The reported scenario is meaningful as it approximates genuine potential sources of data bias (in this case by mixing data sources). Further, the simple synthetic nature of our setup affords future work with an easy and intuitive path to experimental reproducibility.

---

**Q2**: *The considered evaluation metrics do not tell us if hallucinations are removed or if knee images can no longer be recovered, for example.*

**A2**: Indeed, metrics such as mean squared error (`MSE`) and peak signal-to-noise ratio (`PSNR`) predominantly quantify image fidelity in comparison to Groundtruth. However, these metrics also indirectly reflect the absence of hallucinations; hallucinations would typically contribute to increased error or a lower `PSNR` value by introducing discrepancies between the reconstructed image and the Groundtruth (shown in Fig. 1, error calculated by `MSE`). However, we recognize the limitation of using `PSNR` alone to conclusively demonstrate the removal of hallucinations or the prevention of knee image recovery in a strict sense.

In light of this, future work will look to integrate additional qualitative analyses, potentially including expert radiologist evaluations or advanced hallucination-specific detection methodologies, to supplement our quantitative metrics. Expert review is time consuming and absence of datasets that allow us to train unbiased detectors render the automated detection also difficult.

---

To be continued on `Rebuttal to Reviewer fLbn (2)`.


**Reference**:

[1]: Antun, V., Renna, F., Poon, C., Adcock, B., Hansen, A.C., 2020. On instabilities of deep learning in image reconstruction and the potential costs of AI. Proc. Natl. Acad. Sci. 117, 30088–30095. 

[2]:  Bhadra, S., Kelkar, V.A., Brooks, F.J., Anastasio, M.A., 2021. On Hallucinations in Tomographic Image Reconstruction. IEEE Trans. Med. Imaging 40, 3249–3260.

[3]:  Darestani, M.Z., Chaudhari, A.S., Heckel, R., 2021. Measuring Robustness in Deep Learning Based Compressive Sensing, in: Proceedings of the 38th International Conference on Machine Learning. Presented at the International Conference on Machine Learning, PMLR, pp. 2433–2444.
The authors thank reviewer aeoX for the effort for reviewing our work and the constructive suggestions. Here is the rebuttal to the questions.

---

**Q1**: *"high performance levels" is not strictly defined*.  

**A1**: Let us first clarify what we mean by "high performance levels", which is the ability of a model to achieve high-quality reconstruction for the domain it has retained, referred to in-distribution data; in the context of our study, this refers to brain imaging data. A model should strive to maintain comparable reconstruction quality for data from outside the domain, known as out-of-distribution data, such as knee imaging data. We further aim to characterise "high performance" in the context of unlearning by setting a benchmark against what we refer to as the oracle model denoted by  $\hat{G}$. Our definition of successful unlearning is captured by a model performance to approach the oracle model’s performance on in-distribution data as well as on out-of-distribution data. This has been defined in _Section 3.2_ of our manuscript, where we provide a formal definition and elaborate on the assessment metrics. Moreover, another factor on which to measure unlearning performance is computational run-time, noted as run-time efficiency (`RTE`) in our manuscript. In the revised manuscript, we will re-define the goals to avoid any potential ambiguity. 

---

**Q2**: *Although their mentioned it as future work, there is not much discussion on why, when the amount of retained set data increases the quality of image reconstructions does not improve. May be this due to some effect of the l1 norm in 2? How are the model parameters?*

**A2**: We observed that an increase in the amount of retained set data can improve the reconstruction performance when starting from 1%, as illustrated in _Figure 4_, and then decreases the performance from a certain percentage, which we speculate this is owing to the distribution shift between the retained set and the large-scale training set. Fitting the model to the comparatively small amount of retained data can degrade the reconstruction performance thought it may enhance the unlearning efficacy. Both `GA-L1-FT` and `NL-FT` has the same trend, while `GA-L1-FT` is constrained with $\ell$1-norm, while `NL-FT` is not. As for the model capacity, in the future work we would conduct more experiments on different scale of reconstruction model to get to know better the relationship of model size and the amount of retained set data in unlearning.

---

**Q3**: *However, some details are omitted during the methodology description in 3.3.*

**A3**: All details about the methodology are in the Appendix. We hope this solution offers the ideal balance  between explicit clarity and tight spatial constraints; we will happily take suggestions  to make further amendments as required.