dataesr/scientific-paragraphs-categorization huggingface dataset:
Data modeling for a federated-learning healthcare platform: Lessons learned The journey towards the implementation of a CDM for the GenoMed4All project has been a challenging experience that required multiple steps, engagement of different types of stakeholders, and the solution to technical and organizational issues. We summarize in Fig. 4  the main steps in this journey, starting from identifying the need for a  CDM and ending with the actual implementation in the data flows and processes of the project. In what follows, we relay the main lessons that we learned from our experience during this journey and extract generalizable insights for future projects. Lesson 1: The need for a common data model in a federated learning project One of the main results of the data modeling exercise in the GenoMed4All project has been to identify the precise motivation, cost, benefits, and challenges related to implementing a CDM in a federated learning platform for medical research.
A CDM contract between data providers and consumers enables homogeneity, consistency and standardization for both semantic and syntactic aspects, thus leading to improved data quality and reusability. Furthermore, it makes the data Fig. 3. Hierarchy of genomics data needs in the GenoMed4All project and level of applicability for the FHIR, OMOP and Phenopacket data models. The analysis and ML model training within the GenoMed4All is expected to cover different levels of genomics data refinement, spanning from raw sequencing data to linking genomics information with other clinical data. While all the analyzed standards are able to natively represent data at a medium level of processing, we find that OMOP and FHIR are more easily applied to highly processed genomics data (green bars) but do not support representing raw sequencing data (red bars), while the situation is reversed for Phenopackets.
Measures to mitigate risk, such as plans for dealing with data heterogeneity, must be implemented as a last step. more readily auditable and searchable, thus also improving its findability. In a federated setting, a CDM is also crucial to support scalability, by simplifying the recruitment of new SoR, and future-proofing of the platform. However, enforcing such a contract from the beginning can be met with some resistance by clinical data providers as it shifts some of the data management burden to themespecially in the cases where they lack the in-house expertise to perform complex dataset mappingsas well as by data scientists who may need to divert time and effort from the development and training of ML models to learning the technicalities of the CDM. Lesson 2: Implementing a CDM in a federated network is not only a technical challenge In our experience, the implementation of a CDM within a federated network of clinical data providers proved to be not only a technical challenge, but also a complex management issue requiring open communication between all stakeholders involved.
Breaking this cycle required a long and iterative work of educating each side of the discussion to the other side's specific needs, requirements, and sometimes even language. This lesson, which applies to any AI project in the medical field, is especially relevant for FL scenarios where the data scientists never have access to the full dataset. Lesson 4: Convergence of data standards from the -omics and clinical domains is needed Our comparative analysis has identified that, in the context of modeling -omics and clinical data for ML applications, there is a lack of an industry-standard common approach able to fully cover both phenotypic and -omics information. This may partly be due to the origin of common standards, which have typically been developed within a more restricted design space.
However, convergence of standards such as OMOP, FHIR and Phenopackets is sorely needed to enable the next generation of AI models to fulfill the goal of precision medicine. Lesson 5: The CDM strategy and FL platform are closely related: their integration must be anticipated The analysis and identification of the FL platform requirements (and the choice of the underlying technology) quickly revealed a close interrelationship with the CDM. Three key elements of this platform, previously described in this paper, have exposed interfaces with the CDM: the central server dataset manager, the cohort extractor, and the client data integration pipeline. This close inter-relationship highlighted a major architecture risk if not anticipated sufficiently upstream during the design phase. Lesson 6: Heterogeneity among stakeholder communities, and within data sources, represents a significant risk to the project We found that discussions on the data model represented the first moment where the three different stakeholder communities (clinical, AI, and engineering) were required to concretely discuss technical details and produce actionable outcomes in our project.
Care_Site (for institutions), Provider (for individuals). There are some description metadata fields which can store this kind of information. All the datasets used within the first phase of the project required significant manual work to reach an interoperable standardized format, directly impacting the project's execution as well as the final quality of the data. While this lesson applies to any ML project in healthcare, it is especially relevant for FL platforms due to their distributed design and explicit goal of including multiple heterogeneous data sources.
Choroid Plexuses (ChP) are structures located in the ventricles that produce the cerebrospinal fluid (CSF) in the central nervous system. They are also a key component of the blood-CSF barrier. Recent studies have described clinically relevant ChP volumetric changes in several neurological diseases including Alzheimer's, Parkinson's disease, and multiple sclerosis (MS). Therefore, a reliable and automated tool for ChP segmentation on images derived from magnetic resonance imaging (MRI) is a crucial need for large studies attempting to elucidate their role in neurological disorders. Here, we propose a novel automatic method for ChP segmentation in large imaging datasets. The approach is based on a 2-step 3D U-Net to keep preprocessing steps to a minimum for ease of use and to lower memory requirements.
The models are trained and validated on a first research cohort including people with MS and healthy subjects. A second validation is also performed on a cohort of pre-symptomatic MS patients having acquired MRIs in routine clinical practice. Our method reaches an average Dice coefficient of 0. 72 ± 0. 01 with the ground truth and a volume correlation of 0. 86 on the first cohort while outperforming FreeSurfer and FastSurfer-based ChP segmentations. On the dataset originating from clinical practice, the method reaches a Dice coefficient of 0. 67 ± 0. 01 (being close to the inter-rater agreement of 0. 64 ± 0. 02) and a volume correlation of 0. 84. These results demonstrate that this is a suitable and robust method for the segmentation of the ChP both on research and clinical datasets.
Training and validation were performed in healthy controls and people with multiple sclerosis, with the last validation step achieved on clinical data obtained in subjects with Radiologically Isolated Syndrome (RIS), a preclinical form of MS. This newly developed method will be compared to a standard 3D U-Net  (Çiçek et al. , 2016)  and to the widely used FreeSurfer and FastSurfer  (Henschel et al. , 2020)  software packages.
Datasets For this study, we used two different datasets: a research dataset and a clinical dataset. The research dataset (further denoted as dataset1) was constituted by gathering data from three prospective studies performed between May 2009 and September 2017 at the Paris Brain Institute (ICM). The dataset1 is composed of 141 participants: 44 healthy controls, 61 patients with relapsing-remitting multiple sclerosis (MS), and 36 patients with progressive MS. Images were acquired on two different Siemens 3 T MRI scanners (Trio and Prisma) with a 32-channel head coil (92 on Trio and 49 on Prisma). The sequence acquired is a 3D T1-weighted magnetization-prepared rapid gradient-echo imaging (MPRAGE) (repetition time = 2300 ms, echo time = 2. 98 ms). The studies were approved by the local ethics committees and written informed consent was obtained from all participants (EudraCT no.
2008004174-40 and ClinicalTrials. gov identifiers NCT02305264 and NCT01651520). Further details on the dataset1 can be found in  Ricigliano et al. (2021). The clinical dataset (further denoted as dataset2) was composed of 27 pre-symptomatic MS cases fulfilling the 2009 diagnostic criteria for Radiologically Isolated Syndrome (RIS)  (Okuda et al. , 2009) followed at the outpatient Neurology clinics of Pitié-Salpêtrière and Saint-Antoine Hospital in Paris, France, between September 2013 and January 2021 and for whom 3D-T1 weighted MRI scans were available. MRI was acquired as part of a clinical routine using non-harmonized protocols on different MRI machines and at different radiological departments (either within or outside the hospitals). Consent for data collection and analysis according to French legislation for non-interventional research was obtained from all subjects (APHP-20210727144630).
Choroid plexus manual segmentation Before the manual segmentation, all images were corrected for MRI field inhomogeneities using the N4 algorithm implemented in Advanced Normalization Tools  (Tustison et al. , 2010). On both datasets, the ChP in the two lateral ventricles were segmented by a trained neurologist and corrected by a senior neurologist with long-term expertise in MRI processing (procedure named annotator 1). All segmentations were done using ITK-SNAP  (Yushkevich et al. , 2006). The portion of choroid plexuses located within the lateral ventricles of the brain was the only one segmented by the two annotators, as done in previous studies  (Ricigliano et al. , 2022). Indeed, the other parts, located along the roof of the 3rd and 4th ventricles, are hardly and inconsistently visualized.
Manual annotation was performed on the axial non-enhanced 3D-T1w images, until the last visible portion within the temporal horn of the lateral ventricles on both sides, then verified and refined on the coronal and sagittal planes. Segmentations were used as ground truth to train the proposed model and evaluate its performance. Finally, for dataset2 only, ChP were segmented a second time by a trained neurologist (annotator 2), independently from the previous annotator. An example of manual segmentation can be found in Fig. 2.
Preprocessing First, images were reoriented to canonical voxel orientation, so that the left-right direction corresponds to the first dimension, the anteroposterior direction to the second dimension, and the infero-superior direction corresponds to the third dimension. Note that we did not perform any registration to a template. Images were then resampled to have an isotropic 1 mm 3 voxel size and were cropped or padded to match an image dimension of  (176, 240, 256) , corresponding to the dimensions of the majority of the training dataset (denoted as    "HighRes"). Voxel intensities were rescaled to the interval [-1, 1] excluding the lowest and highest 0. 5 percentiles.
Training and validation procedures The dataset1 was split into 72 subjects in the training set, 19 subjects in the validation set, and 50 subjects in the testing set. The testing set (from dataset1) was left untouched until the end and was only used to evaluate the performances. For both architectures, training was done with and without data augmentation. The training was performed during 200 epochs for the 2-step method, and 100 epochs for the 1-step method. The dataset2 was not used for training and was only used as a second test set. The python package Weights&Biases  (Biewald, 2020)  was used for experiment tracking, visualization, and memory monitoring.
Finally, VER and AVER, as we are looking for a reliable method for choroid plexus volume estimation, are all the more relevant and informative for the future task this method might be applied for. Results were reported as mean ± standard error of the mean (SEM). To assess the influence of each experiment on the outcome Dice coefficient, a linear mixed effect model was fitted using the package (v1. 1-31) in R version 4. 2. 2. For each linear mixed model, we tested the effect of data augmentation, model architecture, and their interaction. We, therefore, used a Bonferroni significance threshold of 0. 017 correcting for 3 tests. To assess statistical differences between the scores, paired Student T-tests with a Bonferroni correction for multiple comparisons were used. Note that statistical testing was only performed on the testing datasets.
Segmentation performances Fig. 2  displays examples of manual and automatic segmentations for different subjects of the testing sets from dataset1 and dataset2, corresponding respectively to low, average, and high Dice coefficients. Results on the validation set of dataset1 are reported in Table  4  and results on the testing set of dataset1 in Table  5. For all architectures, performances are comparable between the validation and testing sets, indicating that the validation set was not overfitted. The average Dice is around 0. 7 for all deep learning methods. For all metrics, the proposed deep learning methods provided considerably better performances compared to FreeSurfer and FastSurfer. Results on dataset2 are reported in Table  6. Performances were in general lower than those obtained on dataset1.
On dataset2, average Dice coefficients between all tested methods and annotator1 ranged from 0. 61 to 0. 67 and from 0. 56 to 0. 59 between all tested methods and annotator 2. On dataset1 however, Dice coefficients between predictions and manual segmentations ranged from 0. 69 to 0. 73. The 1-step and 2step methods also tended to provide higher segmentation metrics on the dataset2 segmented by the first annotator compared to the second annotator (cf. Table  7 , supplementary materials). These differences are significant for methods trained with data augmentation (p = 4. 23E-03 for 1-step, p = 4. 88E-04 for 2-step; corrected statistical threshold at 1. 25E-02). Finally, the automatic segmentation performance was of the same order of magnitude as the inter-rater variability (mean Dice = 0. 64 ± 0.
02 SEM for inter-rater agreement). A comparison between the different deep learning approaches (1step, 2-step, with and without data augmentation) is presented in Fig. 3. Data augmentation lowers the Dice coefficient by 0. 02 on dataset1 (p = 2. 24e-12) as estimated from the linear mixed-effect model. No other tests indicated a significant effect of the method, the data augmentation, or their interaction.
Discussion This study presents a simple and reliable method for the automatic segmentation of ChPs called "2-step 3D U-Net". The 2-step 3D U-Net provides excellent segmentation results that reach performances comparable to a regular 3D U-Net while being less memory hoarding. It advantageously requires minimal preprocessing, without any registration procedure. The 2-step 3D U-Net was demonstrated to be effective on heterogeneous clinical data, including healthy and non-healthy subjects, opening the perspective of broad applications aiming at measuring ChP volume in clinical and research datasets and for routine patient monitoring. ChP volumetric changes have indeed been associated with a variety of neurological disorders such as depression  (Althubaity, 2022) , stroke (Egorova et al. , 2019), Alzheimer's disease  (Choi et al.
, 2022) , Parkinson's disease  (Tadayon et al. , 2020) , and multiple sclerosis  (Müller, 2022; Ricigliano et al. , 2021). Recently, ChP enlargement was even found to discriminate multiple sclerosis from neuromyelitis optica spectrum disorder  (Müller, 2022)  and proposed as an early imaging biomarker in preclinical forms of multiple sclerosis  (Ricigliano et al. , 2022). There is therefore an increasing need for reliable segmentation methods of the ChPs to foster research in large datasets and further explore whether ChP volumetry could become a potential diagnostic or prognostic biomarker. To date, only a few automatic segmentation methods of the ChP have been proposed. Tadayon et al. (2020)  have introduced an automatic procedure based on Bayesian Gaussian Mixture Models (GMM) that was subsequently tested on multiple datasets.
Their method was trained using only 10 MRIs from healthy females. The algorithm was not tested on patients, known to present volumetric changes of the ChP. Their method reached a Dice coefficient of 0. 732 ± 0. 046 (mean ± standard error of the mean) in the healthy control population. No further testing of their algorithm was performed on other datasets, containing pathological subjects. Our 2-step 3D U-Net provided similar segmentation performances compared to a regular 3D U-Net, yielding a Dice coefficient of 0. 70 vs 0. 73. However, the non-significance of p-values does not imply the absence of differences between the methods. A larger test set would provide more statistical power to detect those potential differences. Nevertheless, the overall segmentation performances are close, with the best method alternating between the 1-step and the 2-step according to the task at hand, and the standard errors of the mean remain small,  allowing us to believe that the overall performances of the models are comparable.
It outperforms the segmentation provided by FreeSurfer by a vast margin. The poor performance of FreeSurfer could partly be due to the fact that their template was built using different guidelines. Indeed, it is difficult to define the ChP boundaries unambiguously, as illustrated by the inter-rater variability. However, visual inspections of FreeSurfer results revealed that there are clear areas of false positives. Thus, it is unlikely that its poor performance is only due to the template. Also, we did not consider SAMSEG  (Puonti et al. , 2016)  which be a way to improve the FreeSurfer-based ChP segmentation. This is left for future work. In addition, the models have been trained using both healthy subjects and MS patients and have been exposed to a wide range of ChP volumes. Interestingly, data augmentation reduced the loss of performance on dataset2 compared to the models trained without it (Dice = 0.
67 with data augmentation vs 0. 59 without data augmentation for the 2-step and 0. 64 vs 0. 56 respectively for the 1-step) and the 2-step method trained with data augmentation provided the best volumetric correlations on dataset2 (0. 84 with annotator 1 and 0. 62 with annotator 2), emphasizing its optimal reliability for further volumetric analysis of ChPs in large cohorts of subjects. Of note, the first step applied in our protocol was the detection of ChP-containing patches, thus overcoming potential bias linked to class imbalance. By considering spatial information in the image, preprocessing steps such as ventricle segmentation through FreeSurfer or registration to a standard space could be avoided. Another advantage of the 2-step method was the reduced memory consumption required during training, which allowed the use of larger batch sizes compared to the 1-step (4 vs 1).
Taken together, the 2-step 3D U-Net fulfills many prerequisites for large clinical applications. Our study has the following limitations. We acknowledge that the trained models learned slightly better segmentation patterns from annotator 1, which were available for both datasets, than the one from annotator 2, available only for dataset2, which may suggest a slightly reduced reproducibility of the method when heterogeneous real-life data are considered. Also, automatic volumetric analysis of ChPs may miss the presence of microcysts or calcifications, introducing a bias for a minority of subjects. Very few subjects in our datasets presented calcifications or cysts in the ChP so the performance of the methods in those specific cases could not be assessed. Training on a higher number of subjects well characterized for microcysts or calcifications will further benefit the model's generalizability.
Moreover, while the sequence used for segmentation, 3D-T1w MPRAGE MRI, is widely available in all imaging research protocols, it is, however, not the ideal sequence to visualize ChP. First T1w images enhanced with gadolinium allows following the ChP along the walls of the hippocampus. As contrastenhanced T1w acquisition was not available for all MS subjects and was never performed in healthy subjects, we could neither assess the methods' performance on this modality nor train the model to segment the ChP. Similarly, other T1w imaging modalities such as the spin-echo could not be studied due to unavailability in the dataset. Then, on fluidattenuated inversion recovery T2-weighted (FLAIR) MRI, thanks to the fluid attenuation, ChP contrast with surrounding CSF is highly enhanced. However, our datasets did not consistently contain 3D FLAIR images.
Nevertheless, 3D FLAIR is nearly becoming ubiquitous in imaging protocols and ChP segmentation using this sequence could be of great interest. Finally, no test-retest reproducibility assessment of the method was possible as no short-term rescanning of the subjects was available. In conclusion, our 2-step method allows excellent segmentation performances both on research and clinical datasets. Such a method could therefore be applied to a further study exploring the role of ChPs in various neurological diseases. Being an automated segmentation method, this protocol could be used to analyze a large amount of data, in cohorts where manual segmentation would be too time-consuming. This fast and easy-to-use tool would allow the extraction of a simple imaging biomarker with potential interest in many neurological diseases.
Acknowledgments The research leading to these results has received funding from the  French government  under management of  Agence Nationale de la Recherche  as part of the  "Investissements d'avenir" program , reference  ANR-19-P3IA-0001  ( PRAIRIE 3IA Institute)  and reference  ANR-10-IAIHU-06  ( Agence Nationale de la Recherche - 10-IA Institut Hospitalo-Universitaire-6 ), and from  ICM  under the  Big Brain Theory program  (project  IMAGIN-DEAL in MS ). This work was supported by the  Fondation pour la Recherche Médicale , grant number  FDM202006011247.
Tools for multimodal interactions ICON  (Dragicevic and Fekete, 2001 ) is a developer tool to configure multimodal systems. Users graphically assemble software components to create a given service, or use preexisting ones and modify them. It provides a separation between a service and the devices capabilities, but requires technical knowledge to create functional interaction techniques, and doesn't include usability evaluation features. The dynamic customization tool in  (Manca and Paternò, 2016)  enables users to distribute different parts of the interface across devices. They can also modify the distribution of the interface while testing the system. However, interaction techniques cannot be selected independently from the available devices. With MIBO IDE  (Peters et al. , 2016) , interaction techniques can be assembled from easy-to-understand components without programming skills.
Selection of services and associated interaction techniques and devices The third step consists in selecting for each service to configure an associated interaction technique and devices that provide the necessary modalities. For each service they want to configure, the MIBS administrator selects an interaction technique. Interaction techniques are described and are illustrated by a graphical representation of the associated software component chains. For example, in Figure  2  the user selected an interaction technique using a vocal command and a pointing gesture to turn on a pointed light bulb. Thus, the MIBS administrator doesn't need technical knowledge to understand how to test a configuration, but can obtain technical details by clicking on the different icons representing the component.