Implicit neural obfuscation for privacy preserving medical image sharing

Authors: mattias p heinrich
lasse  hansen
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 14:58:12

Abstract: Despite its undeniable success, deep learning for medical imaging with large public datasets leads to an often overlooked risk of leaking sensitive patient information. A person's X-ray, even with proper anonymisation applied, can readily serve as fingerprint and would enable a highly accurate re-identification of the same individual in a large pool of scans. Common practices for reducing privacy risks involve a synthetic deterioration of image quality, e.g. by adding noise or downsampling images, before sharing them publicly. Yet, this also adversely affects the quality of downstream image recognition models trained on such datasets. We propose a novel strategy for finding a better compromise of model quality and privacy preservation by means of implicit neural obfuscation. Our method jointly overfits a neural network to a small batch of patients' X-ray scans and applies a substantial compression - the number of network parameters representing the images is more than 6x smaller than the original images. In addition, we introduce a k-anonymity mixing that injects partial information from other patients for each reconstruction. That way identifiable information is efficiently obfuscated, while we manage to maintain the quality of relevant image parts for the intended downstream task. Experimental validation on the public RANZCR CLiP dataset demonstrates improved segmentation quality and up to 3 times reduced privacy risks compared to a more basic image obfuscation baselines. In contrast to other recent work that learn specific anonymous representations, which no longer resemble visually meaningful scans, our approach remains interpretable and is not tied to a certain downstream network. Source code and a demo dataset are available at https://github.com/mattiaspaul/neuralObfuscation.

Main paper: 
 introduction / motivation: 
 The trend towards larger models, in particular vision transformers, for image recognition have exemplified the need for training with millions of images at the same time. While the advent of grand challenges in medical imaging has led to an ever increasing amount of public CTs, MRIs and X-rays -their amount is still orders of magnitudes smaller than natural image databases (e.g. LVD-142M or SA-1B). Yet, hundreds of millions of digitised scans (Schöckel et al., 2020) are acquired and stored in local clinical picture archives each year. The vast majority of them is never shared (anonymously) with the research community, one likely strong reason being privacy concerns and tighter regulations (Mostert et al., 2016) . Despite its benefits of restricting direct access to personal information the current process of image anonymisation or pseudonymisation is far from perfect (Kaissis et al., 2020) . (Packhäuser et al., 2022) revealed a severe risk of re-identification even if rigorous anonymisation of images is performed, which may enable an attacker to find a person with probabilities as high as 90% within a large public dataset given another X-ray of them. In fact millions of scans together with medical reports have already been leaked due to poor IT security at some hospitals 1 that could be linked to anonymised data and increase the risk of re-identification attacks even further. Our objective is hence to devise a safer mechanism that enables anonymous image data release with substantially reduced re-identification risk, but at the same time this data should retain its diagnostic value for a given intended downstream task, e.g. semantic segmentation. 
related work: 
 Much research has been devoted to de-identifying individuals in natural images or video sequences. Since visual re-identification risks pose a severe challenge to comply with current data privacy regulation obfuscation strategies have been devised to modify images to make persons harder to identify. The DP-Net (Fan, 2018) explores blurring, black/white boxes as well as adversarially learned degradations (cf. also (Wu et al., 2018) ) to maintain the targeted downstream task performance while reducing privacy leakage. (Zhu et al., 2020) and (Dall'Asen et al., 2022) propose to create synthetic image replacements (DeepFakes for de-identification) to preserve privacy in medical videos while preserving diagnostic features for downstream tasks, i.e. preserving keypoints. Advanced methods for video-based person re-identification have been developed in (McLaughlin et al., 2016) . (Kim et al., 2021a) and (Packhäuser et al., 2023b) proposed to learn certain geometric deformations that make the re-identification of brain MRI or chest X-rays with retrieval learning much harder. Latent diffusion models are explored in (Packhäuser et al., 2023a) to create replica datasets that demonstrate only moderate performance drops for training models for downstream abnormality classification, while enhancing privacy preservation. (Kim et al., 2021b) propose a Privacy-Net that jointly learns to map input MRI brain scans into an intermediate privacy-preserving representation, train a semantic parcellation U-Net and also minimises the re-identification risk. While showing excellent results for the given tasks, this procedure requires access to paired patients for each annotation (which is often not fulfilled) and leaves the intermediate representations not interpretable for humans. Mixup-privacy (Kim et al., 2023a) is another strategy aimed at avoiding full knowledge transfer between client and server. Both can therefore be more closely associated with recent differential privacy approaches in federated learning (Rieke et al., 2020 ) that could also be supplemented by encryption with mathematical security guarantees (Kaissis et al., 2021) . k-anonymity, which mixes information from several identities in a single output datapoint, can be seen as a particularly promising strategy to strike a good balance between privacy preservation, downstream task performance and interpretability of the obfuscation. (Meden et al., 2018) compare several approaches for k-anonymity including k-Same-Pixel (Newton et al., 2005) and a new proposed k-Same-Net along with basic pixelation strategies for face photos. They demonstrate good performance for learning to generate synthetic images that share attributes from multiple persons but are specific labels (age, gender, facial expression).Contribution: Our method advances the state-of-the-art in effective medical image obfuscation strategies with regards to the following three main points:• robust generative model, by adapting recent work on neural implicit representation and compression for video sequences to the obfuscation of a subset of an X-ray collection,• novel strategy for k-anonymity that only moderately affects visual image quality while substantially reducing re-identification risks, and• alleviation of the strong requirements of prior work that are based on simultaneous availability of multiple scans per patients at each data providerAlong with these technical contributions, we advance the field of privacy concerning medical deep learning with comprehensive experiments that include the evaluation of privacy risks along downstream task performance (semantic segmentation of catheters in X-rays) for baselines compared to our proposed model. Furthermore, we provide reproducible code for public Kaggle challenge data for others to replicate and built upon our work.  
methods: 
 Our study comprises three aspects: image obfuscation, semantic X-ray segmentation and siamese network re-identification. The concept is implemented within the following scenario.Several data providers want to contribute anonymised X-ray scans along with detailed expert annotations of clinically relevant objects. Here, we use pixel level segmentations of foreign material, in particular central venous catheters (CVC), which are commonly used to detect critical malpositioning (Roldan and Paniagua, 2015) . We assume that part of the combined dataset comprises images with the same patient pseudonym that can be used to train a siamese retrieval network, which will be used to assess the re-identification risk. But crucially neither every image has to be annotated with CVC labels nor does every patient have to be present multiple times. Hence, we do not assume the possibility of jointly training an image obfuscation strategy to de-identify patients along with the segmentation task but rather require the obfuscation to work as a stand-alone step. In addition and in contrast to (Kim et al., 2021b) and (Packhäuser et al., 2023b) , we define the obfuscation strategy to be a white-box model that is accessible to the potential attacker, since having to keep such methods hidden to the public while sharing them across multiple clinics would pose another severe risk/challenge. Our main contribution lies in the development of a novel strategy for creating partially k-anonymous scans using neural implicit compression for open data sharing that preserve relevant feature to train semantic segmentation networks. Yet, the employed semantic segmentation and siamese re-identification methods are described as well for completeness.Implicit neural obfuscation: We base our work on the recent NeRV approach for neural representations for video compression (Chen et al., 2021) . Implicit Neural Representations (INRs) are rapidly gaining attention for effective image representations that amongst others enabled performance leaps for 3D reconstruction (Mildenhall et al., 2021) , image compression (Strümpler et al., 2022) or alignment (Lin et al., 2021; Wolterink et al., 2022) .The key observation is that a low parameterisation of a fully-connected or convolutional network is sufficient to represent images based on an input of a positional encoding. Extending INRs to larger datasets (e.g. through amortised learning (Sitzmann et al., 2020)) is not trivial, yet several newer approaches either employ learnable encoders (Kim et al., 2023b) to predict a latent code embedding for each image or simply keep a dictionary of embedding vectors. (Chen et al., 2021) implements the latter and learns a compact decoder model to restore a video sequence. They clearly demonstrate that in contrast to traditional auto encoders, which have a shared encoder for the whole dataset, NeRV improves reconstruction quality by training a new model for each subset (in their work short video clip). For our approach, we adopt this concept and fit a NeRV to each chunk of 64 images in our data set. We specify the decoder to start from a 64-dimensional latent vector that is mapped with a fully-connected layer into a 16-channel 3×3 latent code and then upsampled with convolutions and pixel-shuffle operations to a target image size of 360 × 360 pixels. We firstly experimented with a mean-squared error reconstruction loss (used traditionally in auto-encoders to mimic a maximum likelihood model) yet this led to unsatisfactory results. Minimising the structural dissimilarity index (maximising SSIM) (Woods et al., 1998) , however, achieves high quality reconstructions with good convergence. The concept is presented in detail in Fig. 1 .Next, we introduce a k-anonymity mixer into the inference path of our NeRV-image reconstruction. A N × N matrix, which is the sum of an identity and Gaussian noise with a hyperparameter ρ controlling the standard deviation, is multiplied with the instance embeddings. That way the latent codes share information from other patients in the same mini-batch. Because the noise is injected at the lowest level of the convolutional decoder it also affects global contextual image content and will ideally mask a substantial amount of identifiable information. This step is only performed at inference, once a subset of images has been fully fitted to avoid the risk of learning to reintroduce personal fingerprints.Catheter segmentation: We opt to use semantic segmentation of catheters as downstream task, due to its clinical relevance paired with challenges for obfuscated images. Central venous catheters are extremely thin foreign objects that typically form an elliptic curve that end in the vena cava. We employ a straightforward 2D SegResNet model (using the MONAI implementation) (Myronenko, 2019) . A unit-weighted combination of soft Dice loss and binary cross entropy (after sigmoid activation) is used to train the network with pixellevel supervision. Note, that we always assume high-quality annotations are available and do not deteriorate labels as they pose a very limited risk for re-identification.Re-identification We implement a classic siamese re-identification network (Taigman et al., 2014) that comprises two identical ResNet34 streams, which produce D-dimensional feature encodings for each image within a mini-batch of size N . A cosine similarity is applied to produce a 2N ×2N score matrix which is fed into the objective function, noise-contrastive estimation loss (InfoNCE) (Oord et al., 2018) , which aims to maximise the similarity of the only positive example out of each 2N -1 candidates. 
experiments and results: 
 The data was obtained from the Kaggle RANZCR CLiP challenge2 . We follow a similar pre-processing as (Hansen et al., 2021) in that we first predict lung masks to each X-ray and automatically define a suitable bounding box for each scan. The images and labels are resampled to 384 × 384 pixels and the CVCs are dilated to approx. 5 pixels. The whole dataset comprises >10'000 images, but we make a subselection to datapoints that either contain a normal CVC annotation or a part of a patient that occurs at least twice to be able to evaluate the re-identification risk. This yields 1536 training and 512 test scans for CVC segmentation and 576 patients with 1152 scans -and 384 patients with 768 images respectively for training and testing for the re-identification risk evaluation (note: the sets do not have to be disjoint).For the image reconstruction/obfuscation, we leave the architectural design setup as is based on the public NeRV repository3 , yet we substantially decreased the capacity of the model to avoid overfitting. In initial experiments, we aimed for approx. 500k trainable parameters per batch of 128 images, which yields a compression of >33-fold when assuming the same quantisation of model weights and image pixels and image dimensions of 360 × 360 pixels. However, this resulted in an under-fitting of the reconstructed images with missing details. Hence, we opted for tripling the parameter count and storing 64 images per NeRV, which is still a considerable 6-fold compression and yields PSNRs of, on average, approx. 40dB. Such a high agreement with the original data will obviously make the re-identification of the same person easier and hence decrease the desired privacy preservation. It is therefore crucial to adjust a suitable noise parameter ρ ∈ {0, 0.04, 0.06, 0.08} for the proposed kanonymity mixing.As baseline obfuscation strategy, we employ pixellation. That means a range of compressed versions of the input images are obtained by downsampling the input images by factors of {1, 2, 4, 6} and resampling them afterwards to the original resolution. We expect both the segmentation quality and re-identification risk of models trained with these degraded images will be lowered.For the 2D SegResNet we chose 24 initial feature channels and 3.5 million parameters in total. The model is trained with a batch-size of 32 for 375 epochs (number of training images is 1536). We use Adam with an initial learning of 2•10 -3 that is reduced by half every 1500 iterations and restarted every 4500 iterations. The first 1000 iterations are stabilised with an additional heatmap loss. We employ the RandomPhotometricDistort and RandomErasing augmentation from Torchvision (v2) and add affine geometric transformations (with a standard deviation of 7•10 -2 and random horizontal flipping to both images and labels. At test time, we only include the horizontal flip, hence two predictions are averaged per input.For training a siamese re-identification network, we follow common practices of contrastive self-supervised learning and use an Imagenet-pretrained ResNet34 for each batch of 32 image pairs. The output feature size is fixed to 256 channels and the InfoNCE loss with cosine similarity and a temperature of 7•10 -2 is used as loss. Adam was used with initial learning rate of 10 -3 for 444 epochs with a single step of 0.2 at half-time. The same augmentations are used as before for the SegResNet. This time, we also include them for testing as we found otherwise all models could not cope with the large diversity of scanning parameters and/or geometric misalignment. We average 25 predictions for each pair of potential matches. Having 384 patients in the test set, we compute the risk of re-identification for a single other image of that same person in the training using any number of guesses from 1 to 15, meaning e.g. the random chance at top-5 would be about 1%.For both experiments (segmentation and re-identification) we evaluated whether the models trained with obfuscation perform better (here higher re-identification means better even though this is a worse outcome for an algorithm) with original test scans or the same modulation. We found that using obfuscated images at test time works in all instances best, likely due to the fact that the models have learned to adapt to these images. Crucially, all re-identification attacks reported were also retrained with the same obfuscation strategies to adapt to the knowledge of the defence mechanism. All models are trained on RTX A4000 cards with bfloat16 precision and torch.compile -within a typical run time of 1 hour. Further implementation details can be found in our public source code at: https: //github.com/mattiaspaul/neuralObfuscation.Apart from the strongest pixellation variant, all approaches perform reasonably well with average Dice scores of above 75% on the challenging downstream task. Remarkably the NeRV obfuscation with ρ = 0.08, which introduces strong visible artefacts still produces predominantly high quality segmentations as visualised in Fig. 2 and 4 . The highest overall performance of 86% is reached by using the original images followed by NeRV without k-anonymity and half-resolution scans with each 83%.Putting the segmentation scores into context with privacy preservation, it becomes evident that only the strongest pixellation with poor downstream performance comes close with a top-5 risk of 47% to the lowering of re-ID risks of the implicit neural obfuscation. When increasing ρ from 0.04 to 0.08 the top-5 risk decreases from 42% to 32%. The contrast is particular stark for top-1 re-identification with original images, >60% compared to our NeRV with ρ = 0.08 with <20% -a three-fold improvement. Choosing ρ depends on the intended use case and privacy-risk assessment.Since pre-trained task-and re-identification models can be quickly evaluated with different ρs for a given NeRV model this provides a first indication of suitable choices. A re-training of both networks is, however, required to validate this assessment. 
discussion and conclusion: 
 Our study demonstrates that privacy risks are imminent for anonymous medical image data sharing, but they could be addressed by a suitable neural obfuscation strategy with negligible performance drops of the models trained and evaluated with such data. The white-box model leads to interpretable outputs and does not impact the process of training downstream models, since normal images can be shared. It is computationally lightweight requiring on average one second to fit a NeRV per image (one minute for a batch of 64).The compression of > 6-fold also brings benefits for a more efficient data transfer. This is the first time neural implicit obfuscation is used for interpretable X-ray segmentation and the proposed introduction of k-anonymity yielded a large improvement in risk reduction.Limitations: There are limitations with regards to the employed comparative methods, since we restricted them to be viable in a scenario where not all labelled data has to be available with multiple scans per patient during training. In case this is possible, even stronger performance could be achieved by specifically optimising de-identification together with segmentation. We also wanted to avoid black-box obfuscation models that have to be kept secure for further anonymisation steps, e.g. at another clinical centre. This is not strictly necessary if all data comes from one hospital. Further initial experiments to extend the number of baseline comparisons to deformation based obfuscation and mixup-privacy as well as the extended evaluation on our NeRV based approach to digitally reconstructed radiographs (DRRs) and another downstream segmentation task can be found in our Github repository and Supplementary material.Future work: It is not yet clear, how such an approach could be extended to sharing volumetric scans. 3D CTs and MRI comprise substantially more anatomical detail and could thus lead to even greater privacy risks. There are also other promising strategies to learn implicit image embeddings, e.g. using generalisable INRs (Kim et al., 2023b) . While being more complex to train, using meta-learning, they could decouple larger parts of the neural networks between shared and instance based parameters and hence provide more control over the level of compression and obfuscation. It also remains to be seen, whether a dataset with NeRV-based k-anonymity still excels at other tasks of chest pathology detection.There are certain restrictions (see paragraph Limitation in main paper) that have limited our baseline comparisons for privacy-preserving data sharing experiments: the work of Packher et al. (Packhäuser et al., 2023b) e.g. assumes access to an already trained task model to perform obfuscation and some aspects of Kim et al. (Kim et al., 2023a) are set within a scenario where not all data from multiple providers is publicly shared. We designed two additional new experiments inspired by this state-of-the-art with certain adaptations to our setting 1) deformation-based obfuscation and 2) mix-up privacy; details of which can be found in the revised appendix. For 1) we create smooth, invertible local deformations to obscure the identity and do indeed see a 20% drop in top5 reidentification risk. However, when re-training the attack model with knowledge about deformations (online augmentations) the risk increases again by 10% making it 30% less safe than NeRV with ρ=0.06. For 2) the scenario of sharing mix-up versions of images and labels without requiring a dual client-server training setup is more challenging (but also possible) and leads to a great reduction of privacy risks (by a factor of 2 or 4 in our tests): we could, however, not avoid a substantial drop in segmentation accuracy to about 40% Dice score for 4-fold mix-up (lower than the strongest pixelation strategy). We attribute this to the fact that jointly training on multiple images with similarly looking thin foreign objects (catheters, tubes, electrodes) is less stable than brain segmentation. Future work could strengthen a combination of these orthogonal strategies. In addition, we included two additional proof-of-concept experiments on different datasets in the public repository (https://github.com/mattiaspaul/neuralObfuscation). They demonstrate the transfer of our method with same hyper-parameters onto a slightly different modality and a new downstream segmentation task. 1) We created DRRs (digitally reconstructed radiographs) of a public paired thorax CT dataset and evaluated the gains in re-identification risk (top5) reduction from 72.92% to 53.12% using NeRV (with ρ=0.08) 2) Due to the absence of fine-grained structures in large-scale X-ray databases (e.g. https://github.com/ngaggion/CheXmask-Database only provides masks for lungs and heart) we evaluated the possibility of segmenting the clavicles in the Montgomery County CXR dataset (Brioso et al., 2023) . This also led to satisfactory quantitative results of 81% Dice for qualitatively strongly obfuscated images. The repository also contains code for data preparation to further extend the experiments once more comprehensive data becomes available.  
 
 Appendix: https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/datagithub.com/haochen-rye/NeRV 


Decision: The paper describes a new approach to privacy-preserving data sharing. There has been an active discussion between the authors and the reviewers, and reviewer ratings vary. In particular, some of reviewer iTVz's concerns remain. I agree with the reviewer that the work appears in places preliminary, and additional experiments could strengthen the paper (such as tumor detection suggested by reviewer QorR ). However, I also believe the work would be of interest to the MIDL community and might raise interesting discussions on this work and privacy-preserving data-sharing approaches in general. As such, I believe it would have added value to the conference.

Reviews: review: paper_summary: paper_summary: In this work, the authors propose the use of implicit neural representation (INR) and k-anonymity mixer to enhance privacy preservation and decrease re-identification risks in a dataset of chest X-rays. To evaluate the approach, two experiments were performed: 1) catheter segmentation using a SegResNet, to assess the ability of the INR to preserve relevant features in the image despite compression and 2) re-identification using a siamese network to evaluate the re-identification risk. Compared to a conventional obfuscation approach such as pixellation, the approach showed better balance between segmentation quality and re-identification risk.
main_review: main_review: The authors responded clearly and concisely to all comments I raised in the review. Also, the authors followed my suggestion to implement and evaluate a second downstream task. I have no concerns with the last version of the manuscript, therefore, I strongly accept.
strength_weakness: strength_weakness: - The experiments conducted are sufficient to demonstrate that the approach has potential, but it would be interesting to also test other downstream tasks, such as tumor detection or segmentation of other relevant structures in the image
- The choice of using only pixellation as baseline method can be limiting given the available promising deep-learning based approaches described in the Related work section
- Only a single dataset was used, it would be interesting to see the performance of the same approach with the same hyperparameters in representing images of other anatomical districts/other imaging modalities
questions: questions: Two main questions:
1) How would you practically choose a suitable noise parameter $\rho$? Is it application dependent? I think this should be discussed in a sentence.
2) What is the impact of the NeRV neural network architecture on the segmentation and re-identifcation performance? What is also the impact of the activation functions?
paper_summary: paper_summary: This paper proposes a privacy-preserving modification to implicit neural representation (INR). In particular, they consider INRs of a batch of images and multiply them with a Gaussian-noise perturbed identity matrix to anonymize the implicit representations. They show that representations learn this way preserve useful information, which is demonstrated by better performance on downstream segmentation task, and are less likely to leak private information (shown by poor reidentification of the subjects). They compare their approach with obfuscation technique such as pixelation or downsampling.
main_review: main_review: I am satisfied with the authors' answers to my questions and added new experiments and comparisons. The approach is interesting and effective; hence, I am increasing my score and would like this paper to be accepted.
strength_weakness: strength_weakness: ### Guarantees
- The authors use an empirical measure of privacy by assessing reidentification risks. It would be nice to provide some mathematical quantification of privacy guarantees.
- What kind of "k-anonymity" is achieved could is unclear?

### Experiments
- The authors use only one empirical measure of privacy --- reidentification risks. Including other practical privacy measures to support the argument that leakage is reduced would be nice. 
The proposed approach is compared to a very simple downsampling baseline. It would be nice to see comparisons with more advanced obfuscation techniques or even with differential private training techniques.
questions: questions: - How would method deal with streaming inputs at the time of inference? How should the other k-1 examples be picked and would that affect privacy guarantees?
- Fig 3a, what does % with better score (y-axis) mean?
paper_summary: paper_summary: The paper presents a novel approach for privacy-preserving medical imaging based on learning implicit neural representations. In this approach, a neural network with few parameters overfits a small batch of images on the task of reconstruction. The learning neural representations, which correspond to a compressed version of images, are mixed to generate noisy reconstructions. In experiments, the paper that these noisy reconstructions offer a better protection of identity (i.e., it is harder for a Siamese network to determine to co-ownership of two reconstructed images) while also enabling a good segmentation.
main_review: main_review: I acknowledge the potential of the proposed method but feel that this current version of the paper falls short of expectations. Authors have performed necessary experiments but present them in a shallow way (in an Appendix). The visual results in Figure 2 cast doubt on the level of privacy offered by the proposed method.

Therefore, I give a final score of weak reject.
strength_weakness: strength_weakness: * The experimental validation of the method lacks depth: the method is only compared against an very weak baseline (pixellation) and on a single dataset/task/modality. 

* The end application of this method is not clearly presented in the paper. If the goal is simply to transfer images, then encryption offers stronger guaranties in terms of privacy. On the other hand, if the objective is training a centralized model, then the model is still susceptible to leaking patient identity information since the segmentation output (which is sent back to the client is not encoded). Finally, if the model is to be used on a federated learning setup, then the paper should compare against approaches of this type.

* I have some serious doubts about the re-identification results of the method. If I understand correctly, the Siamese re-identification network is trained on original images, hence it may suffer from domain shift when used on reconstructed images. As such, the re-identification score might largely underestimate the possibility of recovering patient identity from reconstructed images. Looking at the images in Fig 3, it seems quite easy to recognize the subject in reconstructed images (although longitudinal data should be shown to confirm this).

* The section on Related work is missing important papers on the topic (see comments below)
questions: questions: Clarify the end application setup of your method and explain how your experimental evaluation is fair (with respect to this setup). Moreover, assuming that the decoder is shared, I am not sure that the encoded images offer the level of protection claimed by the paper. Authors should further demonstrate this claim.

Comments: title: Second response to authors
Response to the authors' rebuttal
Summary comment to all reviewers
comment: We would like to highlight that all reviewers valued our contribution with respect to importance of the addressed privacy research challenge and acknowledged is as technically novel and interesting. There was constructive criticism about our initial focus on a single dataset, comparison method and downstream task, which we have alleviated to some degree by adding additional experiments and ablations. 

Reviewer iTVz, however, seems to be fundamentally opposed to our different research hypothesis that could be seen orthogonal to approaches of e.g. Kim et al. Their main reasoning is that they believe privacy-preserving image sharing should be performed using **visually** strongly obfuscated images (making references to our Fig. 4 which shows the same scan with different levels of obfuscation and prior work of Kim et al.), but carry the risk of being reverted by a strong attack. Whereas our approach introduces subtle variations by mixing small details across patients using a new neural field-based k-anonymity approach, which are harder to reverse for the siamese network as they stem from real other scans.  

Despite the strong opposition of one reviewer against such an alternate approach, we nevertheless feel that the MIDL community would benefit from a presentation of our work as it lies at the core of conferences to also enable a discussion of methods that are seen as controversial by some.
We really appreciate the reviewers' remarks within this discussion because it highlights an important aspect that might have been overlooked in their assessment. There is an **important difference for deep learning based re-identification versus human identifiable similarity**. On the one hand, a strongly obfuscated image, e.g. by means of warping, can be impossibly difficult to re-identify by a person but easy for a machine that is trained to ignore (internally reverse) such "within-scan" obfuscation. On the other hand, the power of adversarial learning (and attacks) has demonstrated that subtle differences, which are inconceivable to the human eye may lead to drastic changes in decision boundaries. **Our contribution** - that was acknowledged as technically novel and interesting by all reviewers - **introduces subtle variations by mixing small details across patients, which are harder to reverse** for the siamese network as they stem from real other scans. This leads to an improvement in privacy-preservation, while maintaining a relatively high downstream task performance. 

We make no claim that this method is the only possible solution and superior to the work of Packhäuser or Kim et al., but refute the statement that regulatory bodies or privacy advocates in hospitals would consider this a "low level of protection", solely based on visual human perception of obfuscation strength - whereas our evaluation demonstrates the opposite. We believe conferences are a great possibility to discuss - even sometimes differing hypotheses - and would highly appreciate this opportunity.
I thank authors once again for their responsiveness. 

Regarding the Appendix, I do see a long paragraph on page 12 however it seems to be a discussion of results which are not detailed in the paper. I expected authors to provide a table or a graph comparing their method to such baselines. I wish to point out that this section has no title (e.g., Appendix A) and mentions the following "..details of which can be found in the revised appendix.", making me believe that the Appendix is somewhere else.

With respect to the re-identification results, I understand that the Siamese network has a low(er) accuracy however that does not necessarily mean that information related to identity is removed. As I mentioned in my previous comment, the reconstructed image for rho=0.04 is almost indistinguishable from the original one (visually speaking). I seriously doubt that clinical centres or regulatory agencies enforcing data privacy laws would be convinced by this low level of protection.
We thank the reviewer for considering our answers and would like to respond to the additional comments. The PDF was updated by us, which directly contains the appendix (we will ask the OpenReview support whether the version that authors see can be different). Nevertheless, the main findings are the ones explained in our direct answer: both added approaches, deformation and mixup, as well as our proposed method assume the same knowledge of a potential attacker about the obfuscation method. That means in particular for the deformation strategy that the attacker knows which range of displacements and B-spline model is used, **but does not know the exact transformations**. 

We hence stand by our statement that deformations can be an orthogonal strategy to further improve privacy preservation. However, they are on their own - for our experimental setup - less safe. With regards to the mixup strategy (proposed by Kim et al. at IPMI 2023), we would like to re-iterate that this in fact provides very strong privacy, but cannot achieve comparable segmentation accuracy in our chosen scenario due to our focus on data sharing without client-server interaction and a substantially different domain/downstream objective.

We thank the reviewer for pointing to the visual similarity of different versions of the same patient/same scan in Fig. 4. Indeed a copy-detection would be able to retrieve the same scan for all considered obfuscation methods. The used setting for re-identification, however, requires the attacker to find another scan of that same patient within a database of openly shared images, which is much harder. **As suggested we evaluated a simple correlation analysis (measuring SSIM across compared scans), which revealed much lower re-identification scores** (approx. 10% top5 when each patient was present twice and 16% top15 accuracy when each patient was four times in the database). This demonstrates that the siamese retrieval network is more powerful and a reasonable choice to evaluate that indeed the obfuscation shown in Fig. 4 is strong enough to reduce re-identification risk for that patient.
I thank authors for considering all my comments and providing a detailed answer to my questions.

Regarding my concern about the weak comparison baselines, authors have added two privacy-preserving approaches, based on on mixup and learned deformation. I apologize to the authors if I missed something but I do not see the mentioned Appendix in their revised paper or any Supplemental material document in the OpenReview page.

For both added approaches, unlike the proposed method, the precise identity obfuscating transformation operation is not known to the attack because it depends on some information only known by the client. Is this how the approaches were used in the comparison or were the exact transformations shown to the attacker. 

To clarify my comment on the re-identification results, the reconstructed images in Figure 4 are visually similar to the original ones (especially for rho=0.04), and it seems easy to find identity using a simple correlation analysis (across a database of existing images). Could the author explain how identity is hidden here? Maybe I am wrong, but I assumed from the text that the image decoder is public hence encoded images can be reconstructed by a potential attacker.
We thank the reviewer for their very comprehensive remarks and appreciation of the novelty of the method and are confident that our response can address many of the remaining concerns. 

Regarding the fairness of the experimental evaluation we apologise for our unclear wording of the following sentence: "For both experiments (segmentation and re-identification) we evaluated whether the models trained with obfuscation perform better (here higher re-identification means better even though this is a worse outcome for an algorithm) with original test scans or the same modulation". We meant to **highlight that we indeed re-trained the re-identification networks with knowledge of the obfuscation strategy**. This is now clarified as "Crucially, all re-identification attacks reported were also retrained with the same obfuscation strategies to adapt to the knowledge of the defence mechanism." in the same paragraph. 

We would like to re-iterate the differences of our intended use case of our method for privacy-enhanced public data sharing, which is not directly comparable to the very comprehensive methods presented in the literature e.g. by Kim et al. for obfuscation in a (at least partially) federated learning scenario. To answer one question from the weakness paragraph: Yes, the objective is training a centralised model without encryption. Given that the anatomies and foreign objects depicted in chest radiographs are less privacy-revealing than e.g. structures in brain MRI we hence consider the leakage of patient identity information through segmentations as rather low.

We are very thankful for the two additional references (Kim et al. IPMI and BMVC) and **have incorporated both of them in our revised paper**. We **have also removed the statement about model vulnerability** with regards to the encrypted encoding in Kim et al. 2021. As suggested and detailed in the summary response, we included additional comparison experiments for deformation obfuscation and mixup-privacy. These experiments should not be interpreted as a re-implementation of the corresponding IPMI and BMVC papers, because the scope and intended use case is quite different. But they show that: 1) deformation obfuscation is another complementary solution that could further reduce privacy risk, but could be slightly diminished by re-training the attack with simulated warps; 2) mixup privacy works best in the scenario of a joint centralised model training and client obfuscation with appropriate hand-over of the information pertaining the mixup strategy. A straightforward sharing of mixed-up images and labels seems to be at least not easily possible for chest X-ray with ambiguous thin object segmentations. 

Further minor corrections are included in the revised PDF as suggested.
We thank the reviewer for their appreciation of our solution for an important problem and great suggestions for further improving the validation. As mentioned in the general answer, we extended the set of comparison techniques by deformation-based obfuscation and mix-up privacy. While the former did not show as strong privacy preservation when retraining the re-identification network the latter led to a performance deterioration for the downstream task. We have corrected the typos and will add a short description of alternative solutions that could ideally be combined with our proposed method in future work. 

Following the recent PriMIA paper (Kaissis et al. 2023, https://doi.org/10.1038/s42256-021-00337-8) one could add homomorphic encryption and differential privacy when considering a federated learning setup - for which mathematical guarantees are available. 
To the best of our knowledge no universally agreed guarantees (aside from empirical tests) exist for obfuscated data sharing without encryption. But combining implicit neural obfuscation with DP (which has provable guarantees) could yield a beneficial defence against gradient-based model inversion attacks. Since our work does not focus on model sharing the latter do not directly apply for our experiments.


Our answers to the two questions for the rebuttal are as follows. 
Streaming inputs at inference time could be handled in different ways: 1) by re-mixing a single new sample with images already obfuscated with our NeRV-solution (which would not affect privacy) or 2) if no further data sharing is intended (test-time) simply fitting the NeRV to a single image will also work based on our experiments.  
The y-axis in Fig. 3a) describes the cumulative nature of the Dice score plot. This means that e.g. at a threshold of 80% Dice (x-axis) the pixelation method with size 64 achieves a better score for just 25% of cases, whereas the original images (384) yield 85% of cases with Dice >80%.
We thank the reviewer for their positive remarks and have addressed the remaining concerns (see also general comment). As suggested we had applied our method with the same hyperparameters to a different dataset - digitally reconstructed radiographs from thorax CT for re-identificaiton - and a different downstream task - clavicle segmentation as illustrated in our GitHub repository, which yielded very similar results and will be mentioned in the revised appendix. Furthermore we extended the choice of baseline methods by deformation-based obfuscation and mixup-privacy with detailed explanation in the general comment.

Regarding the direct questions for the rebuttal: 1) Yes the selection of the noise parameter would be application specific. We will provide with the following suggestion for finding a good balance in practice: "Choosing $\rho$ depends on the intended use case and privacy-risk assessment. Since pre-trained task- and re-identification models can be quickly evaluated with different $\rho$s for a given NeRV model this provides a first indication of suitable choices. A re-training of both networks is, however, required to validate this assessment." 2) We mentioned in the second paragraph of Section 4 that a NeRV with only 500k trainable parameters per batch led to underperforming image reconstruction, hence a less-compact architecture was chosen - which ensured a PSNR of approx. 40 dB. Contrary to conventional coordinate MLPs that take xy-coordinates as input and require either sinusoidal activations (SIREN) or Fourier feature encodings, our approach already uses a learnable patient embedding and we did not see gains of modulating the activation function.
We would like to thank all reviewers for their very insightful comments and overall appreciation of our paper including the importance and practicality of the sharing of datasets (Wxho), interesting idea with well designed description (QorR and Wxho), its novelty (iTVz) and open source code (iTVz) along with our mentioning of limitations (QorR). The assessment of (iTVz) also points to an **important potential for clarification** regarding the re-training of re-identification attacks with knowledge of the NeRV-based obfuscation strategy. **This is certainly important and we had already done it but not mentioned clearly enough and hence a major concern of (iTVz) should be alleviated**. 

Furthermore, all reviewers suggest to perform additional experiments for improved validation across datasets and tasks and in comparison to state-of-the-art (SotA). There are certain restrictions (see limitation section in paper) that we wish to maintain for our privacy-preserving data sharing experiments: the work of Packhäuser et al. e.g. assumes access to an already trained task model to perform obfuscation and some aspects of Kim et al. are set within a scenario where not all data from multiple providers is publicly shared. Nevertheless, **we designed two additional new experiments** inspired by the SotA of Kim et al. and Packhäuser et al.: **1) deformation-based obfuscation and 2) mix-up privacy;** details of which can be found in the revised appendix. For 1) we create smooth, invertible local deformations to obscure the identity and do indeed see a 20% drop in top5 re-identification risk. However, when re-training the attack model with knowledge about deformations (online augmentations) the risk increases again by 10% making it 30% less safe than NeRV with \rho=0.06. For 2) the scenario of sharing mix-up versions of images and labels without requiring a dual client-server training setup is more challenging (but also possible) and leads to a great reduction of privacy risks (by a factor of 2 or 4 in our tests): we could, however, not avoid a substantial drop in segmentation accuracy to about 40% Dice score for 4-fold mix-up (lower than the strongest pixelation strategy). We attribute this to the fact that jointly training on multiple images with similarly looking thin foreign objects (catheters, tubes, electrodes) is less stable than brain segmentation. Future work could strengthen a combination of these orthogonal strategies. 

In addition, we would like to **point to two further experiments that were already included in our public GitHub repository** in time for review. They demonstrate the transfer of our method with same hyper-parameters onto a slightly different modality and a new downstream segmentation task. 1) We created DRRs (digitally reconstructed radiographs) of a public paired thorax CT dataset and evaluated the gains in re-identification risk (top5) reduction from 72.92% to 53.12% using NeRV (with \rho=0.08) 2) Due to the absence of fine-grained structures in large-scale X-ray databases (e.g. https://github.com/ngaggion/CheXmask-Database only provides masks for lungs and heart) we evaluated the possibility of segmenting the clavicles in the Montgomery County CXR dataset. This also led to satisfactory quantitative results of 81% Dice for qualitatively strongly obfuscated images. The repository also contains code for data preparation to further extend the experiments once more comprehensive data becomes available.