dataesr/scientific-paragraphs-categorization huggingface dataset:
The tool radii R and tool-tip lengths h were measured using a numerical caliper. The tool 3D pose estimation method closest to ours is  ( Agustinos and Voros, 2015 ). However, this method's implementation is not publicly available and would be difficult to reproduce, preventing a direct experimental comparison. Nonetheless, this method uses tool boundaries detection, which is not trained end-to-end and uses local filtering instead of DNN. Hence, it is likely not to be triggered in challenging surgery conditions, and its robustness and performance will depend on the color and material of the tool shaft and tip. Evaluation from animal data Two videos of laparosurgery performed on a pig were collected. A hook scissor and curved dissecting forceps were used in the first and second videos, respectively.
To obtain the ground-truth of the 3D pose, a chessboard was stuck on each of the tools in a position well visible on the laparoscopic videos, as shown in Fig. 16 (a). For each acquisition, a sub-image not containing the chessboard was manually extracted (shown in Fig. 16  (a) as the left-most, undimmed part) and used as an input image to ART-Net. An example of an estimated pose is shown in Fig. 16 (b ). It is compared from two different viewpoints against the ground-truth computed from the chessboard in Fig. 16 (c-d ). The ground-truth was computed as follows. A 3D reconstruction of the tool was performed using the Meshroom software ( AliceVision, 2018 ) from a collection of photographs taken with a high-resolution camera. The chessboard corners were then manually selected and refined to reach subpixel accuracy.
The tip points of the tools were also selected. The reconstructed cameras were used to triangulate the corner and tip points. The triangulated points were then refined through bundle adjustment using Matlab. The obtained 3D reconstruction was finally used to get the ground-truth of the tool 3D pose using EPnP  ( Lepetit et al. , 2009 )  from 3D-2D correspondences manually selected in the laparoscopic images. 50 images were extracted from each video. As the image domain associated with animal data differs from the one related to the training data, namely patient data, a subset of images for which the pose initialization provided acceptable reprojection errors were kept for ground-truth computation and evaluation of the tool 3D pose. It led to the use of 17 images for the first video and 15 for the second one.
4% and 17. 5% of the noisy dataset. A significant number of these failure cases are caused by the affine ambiguity discussed in Section 4. 6. The medians range from 4 • to 9 • and 4 mm to 6 mm for the angular and tip position, respectively. Evaluation on patient data We evaluated the estimated 3D pose quantitatively from patient data. 10 laparoscopic images were extracted from a video of laparoscopic myomectomy from CHU Estaing, Clermont-Ferrand, France. These images were selected to show the tool touching the uterus surface. A preoperative model was built from the patient MRI and registered to the laparoscopic images using the pipeline  ( Collins et al. , 2016 ). The registration result provides the ground-truth of the tool-tip depth. Specifically, the error on the estimated 3D position is expressed in that case as the shortest distance between the estimated 3D point and the registered uterus surface ( Fig.
A comparison of the proposed integrated framework against separated frameworks was conducted. The integrated ART-Net is shown to perform substantially better than separated frameworks solving only one task amongst geometric primitive extraction and is on par for detection and segmentation. These results are reported in table  8. The number of training images used in our experiments is very limited, namely 508 images. It would be interesting to prepare experiments with a larger dataset in future work and measure the segmentation and geometric primitive extraction accuracy for different amounts of training images. It would allow one to quantify the number of training images from which increasing the training dataset further only marginally improves the accuracy of ART-Net. The estimated 3D pose from the vanishing point of the obtained primitives is precise and robust.
positive. For segmentation and geometric primitive extraction, we used 508 and 127 images for training and testing, respectively. The distribution in terms of image disturbances of the 935 images composing EndoVis non-robotic and our dataset is as follows. No disturbance: 67 % (630 images); motion blur: 10 % (97 images); presence of trocar: 4 % (33 images); bleeding: 7 % (65 images); smog: 7 % (63 images); tool occlusion: 5 % (47 images). Table2shows a summary of the datasets where the numbers of annotations for detection, segmentation, and geometric primitive extraction are reported. 3. 5. Loss function, training strategy, and evaluation criteria3. 5. 1. Loss functionThe binary or categorical cross-entropy (CE) functions are widely used as loss function in both classification and semantic segmentation in CNN training.
he rapid evolution of artificial intelligence (AI) and machine learning (ML) in biomedical data analysis has recently yielded encouraging results, showcasing AI systems able to assist clinicians in a variety of scenarios, such as the early detection of cancers in medical imaging 1,2. Such systems are maturing past the proof-of-concept stage and are expected to reach widespread application in the coming years as witnessed by rising numbers of patent applications 3 and regulatory approvals 4. The common denominator of high-performance AI systems is the requirement for large and diverse datasets for training the ML models, often achieved by voluntary data sharing on behalf of the data owners and multi-institutional or multi-national dataset accumulation. It's common for patient data to be anonymized or pseudonymized at the originating institution, then transmitted to and stored at the site of analysis and model training (known as centralized data sharing) 5.
However, anonymization has proven to provide insufficient protection against re-identification attacks  6, 7. Therefore, large-scale collection, aggregation and transmission of patient data is critical from a legal and an ethical viewpoint 8. Furthermore, it is a fundamental patient right to be in control of the storage, transmission and usage of personal health data. Centralized data sharing practically eliminates this control, leading to a loss of sovereignty. Moreover, anonymized data, once transmitted, cannot easily be retrospectively corrected or augmented, for example by introducing additional clinical information that becomes available. Despite these concerns, the increasing demand for data-driven solutions is likely to increase health-related data collection, not only from medical imaging datasets, clinical records and hospital patient data, but also for example via wearable health sensors and mobile devices 9.
Evidence from prior work Recent work has shown the utility of PPML in biomedical science and medical imaging in particular. For instance, federated learning (FL) is a decentralized computation technique based on distributing machine learning models to the data owners (also referred to as computation nodes) for decentralized training instead of centrally aggregating datasets. It has been proposed as a method to facilitate multi-national collaboration while obviating data transfer. In the setting of the COVID-19 pandemic  10, 11  FL was used to allow the retention of data sovereignty and the enforcement of local governance policies over data repositories. In medical imaging, recent studies  5, 12  demonstrated that federated training of deep learning models on brain tumour segmentation or breast density classification performs on-par with local training and that it fosters the inclusion of data from more diverse sources, leading to improved generalization.
NATUre MAcHINe INTellIgeNce However, FL in itself is not a fully privacy-preserving technology. Previous studies  13, 14  demonstrate that inversion attacks can reconstruct images from model weights or gradient updates with impressive visual detail. Moreover, in the setting of inference-as-a-service  15  , exposure of the model to a non-trusted third party can enable model misuse or outright theft. Therefore, FL must be augmented by additional privacy-enhancing techniques to truly preserve privacy. For example, FL with secure aggregation (SecAgg) of weights or gradient updates or differential privacy (DP) can prevent dataset reconstruction attacks, and the utilization of secure multi-party computation (SMPC) protocols during model inference can protect the models in use. We provide an overview of these techniques in our previous work  16.
Aim and contributions The clinical application of PPML in medical imaging requires the development of frameworks for security and privacy, and their validation on non-trivial clinical tasks. Here we present PriMIA, a free, open-source framework for end-to-end privacy-preserving decentralized deep learning on medical images. Our framework incorporates differentially private federated model training with encrypted aggregation of model updates as well as encrypted remote inference. Our contribution provides the following innovations: • We demonstrate the training of a deep convolutional neural network (CNN) on the clinically challenging task of paediatric chest radiography classification using FL augmented with PriMIA's privacy-enhancing techniques over the public Internet. • Our framework is compatible with a wide range of medical imaging data formats, easily user-configurable and introduces functional improvements to FL training (weighted gradient descent/federated averaging, diverse data augmentation, local early stopping, federation-wide hyperparameter optimization, DP dataset statistics exchange), increasing flexibility, usability, security and performance.
• We examine the computational and classification performance of models trained with and without privacy-enhancing techniques against models trained centrally on the accumulated dataset, personalized models trained on subsets of the data and against expert radiologists on unseen real-life datasets to evaluate various scenarios typical in medical imaging research. • We assess the theoretical and empirical privacy and security guarantees of our framework and provide examples of applying a state-of-the-art gradient-based model inversion attack against the models under a number of training scenarios. • Finally, we showcase the utilization of the trained model in a secure inference-as-a-service scenario without the disclosure of either the data or the model in plain text and demonstrate the improvements in inference latency of our SMPC protocol.
Library functionality PriMIA was developed as an extension to the PySyft/PyGrid ecosystem of open-source PPML tools. PySyft (https://github. com/ OpenMined/PySyft) is a Python framework allowing the remote execution of machine learning tasks (for example, tensor manipulation) and for encrypted deep learning by interfacing with common machine frameworks such as PyTorch. PyGrid provides server/client functionality for the deployment of such workflows on servers and edge computing devices. A detailed description of the generic functionality provided by these frameworks can be found in our previous work  17. PriMIA builds upon this functionality towards medical-imaging-specific applications by being natively compatible with medical imaging data formats such as DICOM and able to operate on medical datasets of arbitrary modality and dimensionality (for example, computed tomography, radiography, ultrasound and magnetic resonance imaging We assumed an honest-but-curious threat model as defined previously  22  for the training phase.
classification performance We trained FL models without SecAgg or DP (DP-/SecAgg-), with SecAgg only (DP-/SecAgg+) and with both techniques (DP+/ SecAgg+). Furthermore, we trained a model on the entire dataset pooled on a single machine (centrally trained) and separate models on the individual data owners' subsets of the dataset (personalized). The centrally trained model represents the centralized data sharing scenario described in the introduction. The personalized models each represent a single institution training exclusively on their own data, a typical case in current medical imaging research workflows. FL aims to enable the training of models that are better than personalized training and-ideally-as good as the centrally trained model. We tested the classification performance of the models on the validation set and against the classification performance of two expert radiologists on test set 1 (145 images) and against clinical ground truth data on test set 2 (345 images).
We used accuracy, sensitivity/specificity (recall), receiver-operator-characteristic-area-under-the-curve (ROC-AUC) and the Matthews correlation coefficient (MCC)  27  for assessment. Details can be found in the Methods section. Model and expert classification performance on the datasets can be found in Table  1. The FL model trained with neither SecAgg nor DP performed best with no statistically significant difference to the centrally trained model. The addition of SecAgg to the model slightly, but non-significantly reduced performance. Both FL models and the centrally trained model significantly outperformed the human observers. The DP training procedure (ϵ = 6. 0, δ = 1. 9 × 10 -4 at an α-value (divergence order) of 4. 4) significantly reduced model performance, however the model still performed statistically on par with human observers and retained stable performance on the out-of-sample data of test sets 1 and 2.
Training and inference performance benchmarking To assess the performance ramifications of PriMIA's privacy-enhancing techniques, we benchmarked the training and inference performance in a variety of scenarios, shown in Fig. 3. Training timings were measured as average time per batch at a constant batch size to decouple them from dataset size. Compared to training locally, FL incurs a performance penalty due to network communications, which is further increased by the addition of SecAgg and DP, yielding a threefold increase in training time when both SecAgg and DP are used. Large neural network architectures require proportionally longer to train due to network transfer requirements, providing justification for the use of the ResNet18 architecture in our study compared with larger ResNets.
The addition of more worker nodes led to a linear increase in times when utilizing SecAgg due to the communication overhead of the protocol. However, due to the small number of operations per round, the protocol scales well to multiple parties: linear regression analysis of the scaling yielded t(w) = 0. 57w + 2. 61 with t expressing time in seconds and w the number of workers (R 2 = 0. 98, p < 0. 001, N = 100 samples per number of workers tested). Training time was nearly constant without SecAgg. Training times per batch were constant for larger dataset sizes, signifying that training duration is dependent only on dataset size all other things being equal. Lastly, we benchmarked our encrypted inference implementation  26  based on the function secret sharing (FSS) protocol  28  , which offers increased efficiency in the evaluation of comparison operations, max-pooling and batch normalization layers compared to the widely used SecureNN  29.
Model inversion attack Prior work  13, 30  has demonstrated that model inversion attacks are able to reconstruct features or entire dataset records (in our case, chest radiographs), rendering them a threat to patient privacy in FL settings. To exemplify the susceptibility of models trained with and without the privacy-enhancing techniques offered by PriMIA, we utilized the improved deep leakage from gradients attack  31, 32  with small modifications detailed in the Methods section. We chose this method because it was the first technique shown to be highly effective against the ResNet18 architecture used in our case study. Figure  4  shows exemplary results from the chest radiography case study. We utilized the pixelwise mean squared error (MSE), signal-to-noise ratio (SNR) and Fréchet inception distance (FID) metrics for quantifying attack success.
Empirical evaluation yielded that the attack's success depends highly on the L2-norm of the gradient updates and the batch size used. To thus generate a best-case baseline of a highly successful attack, we attacked the centrally trained model with a batch size of one at the start of training, when the loss magnitude (and thus gradient norm) is highest. The attacks on the FL model with SecAgg used for our case study were not successful, most likely due to the high effective batch size of 600. Consistent with DP's privacy guarantees, the attacks were ineffective when DP training was used. Results showing that DP negates the attack even when the model is attacked locally or when SecAgg is not used are shown in Supplementary Section 5 and Supplementary Fig. 2. To further underline the high risk of privacy-centred attacks in the healthcare imaging setting and thus the importance of privacy-enhancing techniques for collaborative model training, we performed additional experiments on the publicly available MedNIST dataset and were able to recover images disclosing sensitive patient attributes when DP was not utilized.
Discussion We've presented PriMIA, an open-source framework for privacy-preserving FL and encrypted inference on medical images. We've demonstrated the decentralized collaborative training of an expert-level deep convolutional neural network in the challenging clinical task of paediatric chest radiography classification. Further, we've showcased end-to-end encrypted inference, which can be leveraged for secure diagnostic services without the disclosure of confidential data or exposure of the model. Our work serves as the first step towards the implementation of next-generation privacy-preserving methods in medical imaging workflows. It applies to both multi-institutional research and to enterprise model development settings, allowing the preservation of data governance and sovereignty over confidential patient health data.
Our framework can be used in inference-as-a-service scenarios in which diagnosrsquo support can be provided remotely with theoretical and empirical guarantees of privacy, confidentiality and asset protection. PriMIA represents a targeted evolution of our previous work  17  towards healthcare-sector-focused deployment. Although we focused on a classification task for the presented case study, PriMIA is highly adaptable to a variety of medical imaging analysis workflows employing different network architectures, datasets and more. We present an additional case study focused on semantic segmentation in computed tomography scans of the abdomen in Supplementary Section 7 and Supplementary Fig. 3 , to demonstrate this flexibility.
DP model training is able to offer objective privacy guarantees and resilience against model inversion attacks  30, 32. The utilization of DP diminished model performance, which was, however, still on par with human observers. At the same time, the DP guarantees achieved (ϵ = 6) by the selected model are only moderate. This phenomenon (privacy-utility trade-off) is a well-known observation in the still nascent area of deep learning with DP. For instance, previous work  23  reached an ϵ-value of approximately 8 on the CIFAR-10 dataset and another study reported  34  ϵ-values between 6. 9 and 8. 48. Both studies also report a diminished performance by the final model. We regard methods to improve the training of DP models as a promising direction for future research. over the entire confederation using the Tree-Structured Parzen Estimator algorithm  38.
DP training provides objective privacy guarantees in case of attacks against the model both by confederation members and during inference and is not limited to the gradient-based inversion attack we use in our example. SecAgg utilizing SMPC only discloses the aggregate model update to the parties, even in case up to n -1 out of n parties collude to reveal data. The DP secure aggregation of dataset statistics (means and standard deviations) we propose can protect FL participants from data leakage, especially when non-imaging data is included in model building (for example clinical records, in which the means of features such as age represent sensitive information). Finally, encrypted inference reveals no information about the data or the model to either party. Compared with fully homomorphic encryption protocols  40  relying on key-based cryptography, whose implementation for neural network training and inference is impeded by the computational complexity of the encryption process and the performance decrease due to function approximation for for example activation functions, communication overhead has traditionally been the limiting factor for SMPC.
In our recent work, we introduced AriaNN  26  , an SMPC protocol leveraging function secret sharing (FSS)  28  and building upon SPDZ  25. It represents an alternative to protocols like SecureNN  29  or Falcon  41  , and computes private comparisons with a single round of communication. This renders FSS substantially more communication-efficient than other SMPC protocols, especially when parties are geographically distant and communicate with high latency, for example when performing inference over the public web as showcased in our study. Through the present use-case, we confirm the results obtained in our previous work on other datasets: secure inference gains proportionally greater benefits from the FSS protocol in the high-latency setting. Thus, we propose its utilization over SecureNN in cases a reduction in latency is desired in an honest-but-curious setting.
Li et al. 46  utilize DP, however assume a fixed sensitivity and do not conduct privacy analysis. Their framework does not offer SecAgg or encrypted inference. Limitations. We consider the following limitations of our work. The computational requirements for deploying our system are substantial, and the latency resulting from encrypted inference is still very high compared to unencrypted inference, despite the proposed protocol improvements. The underlying remote execution environment currently offers experimental graphics processing unit (GPU) support, with full support planned for an upcoming version. The success of FL models is largely dependent on high data quality on the nodes. The auditing and curation of the data and its quality, methods to quantify the contribution of individual datasets to the model or to detect local overfitting are still under investigation  47.
Methods Dataset collection. For model training, we used the previously proposed paediatric pneumonia dataset  20. The dataset was reviewed by a specialist radiologist for image quality and representativeness and included 5,163 training images in the above-mentioned three categories, as well as a validation set of 624 images. For FL model development, the training set was randomly subsampled into three equally sized non-overlapping partitions. Class balance between nodes was not enforced. For model testing on unseen data, we retrospectively collected 497 chest radiographs of the same classes of an age-matched cohort from two university hospitals (test set 1: 145 images (43 bacterial, 68 normal, 34 viral), test set 2: 352 images (120 bacterial, 126 normal, 106 viral)). Ethics committee and data protection votes for data collection and exchange were granted by all institutions waiving the requirement for informed consent in this retrospective study (protocol number 111/20 S-KH).