dataesr/scientific-paragraphs-categorization huggingface dataset:
The second major challenge on MS lesion segmentation was held in 2015 at the IEEE ISBI international conference  8. It was more focused on the study of longitudinal lesion evolution with specific evaluation metrics based on segmentation volume evolution (in addition to the regular segmentation overlap metrics used in 2008). This challenge evaluated 10 different methods on a dataset composed of five patients images each with an average of 4. 4 time points, each time point being manually delineated by two experts. As for the MICCAI 2008 challenge, two raters were not enough to account for disparities in the different raters manual segmentations and get a representative consensus. The process of evaluation was similar for the two challenges. A subset of the patient images was provided to the participants with the ground truth (GT) segmentation to the participants for them to train their respective methods.
g. required memory of computing time) is possible. There is therefore a need for computing platforms for supporting challenges including data storage, processing pipelines (i. e. segmentation algorithms work-flow used) integration and evaluation on stored datasets. Such platforms would provide a truly fair comparison between fully automatic methods. In addition, such remote computing platforms, able to host a large variety of algorithms, announce what the future cloud computing services will provide to assist clinicians (radiologists, neurologists, …) in using computer aided diagnosis solutions. This computing environment also opens the road to open-science platforms where people will find solutions to post their data, send or retrieve algorithmic solutions and provide an independent yet secure environment to compare, assess and combine various algorithms outcomes and solve clinical problems.
org). It aimed at proposing solutions to several of the previously mentioned defects first by gathering an unprecedented database of MS patients, coming from three different centers (representing four different scanners, one of which was intentionally hidden at the training phase from the challengers to test their algorithms' adaptation capabilities) but all following a common consensus protocol  9  , each patient being delineated by seven experts to evaluate not only automatic methods performance but also inter-expert variability of manual segmentation. We have performed the evaluation on a dedicated computing platform provided by France Life Imaging (https://www. francelifeimaging. fr/en), providing pipeline integration, database storage and automatic execution capabilities. Challenge participants were asked to train their algorithms on a reduced set (n = 15) and then integrate their pipeline on the platform, requiring no action from them in the latter parts of the evaluation process (n = 38).
Results Challenge data, computing platform and participating teams. The first major result of this study is the gathering of a database of 53 multiple sclerosis patients with "ground truth" of very high-quality. The database patient scans were following the OFSEP protocol recommendations in  9  , which is currently applied in France for the constitution of the national cohort in MS (for more details on the protocol, see Section 4. 1). Following this approach has allowed for an evaluation representative of the current imaging protocols standards and easily usable to characterize the best performing algorithm for future use. This standardization of imaging protocols announces how the dissemination of computer aided diagnosis and imaging biomarkers solutions will be implemented in the future.
Patients demographic data were the following: average age of 45. 3 years (±10. 3 years) with a male:female ratio of 0. 4. This database was then split into two sets: one training set of 15 patients from three scanners (thus intentionally missing one scanner from the database) given to participants, and one testing set of 38 patients, not seen by the participants, used for evaluation. Demographics of patients do not vary significantly over the different sitesin terms of age. Some variations exist in the male:female ratios in some centers. The training and testing sets have an average age difference of 5 years (training set patients are 5 years younger). A total of thirteen teams were evaluated, and the website (http://portal. fli-iam. irisa. fr/msseg-challenge/), databases and algorithms will remain open for future use.
59 while the lowest expert is on average at 0. 67 (significant difference, Wilcoxon signed rank test, p = 2. 9 × 10 -3 ). The average surface distance is a more balanced metric in terms of results with the second group of algorithms in each graph reaching the level of agreement that the experts do with the consensus. Segmentation on an unknown scanner leads to poorer performance. Scanner 3 in the testing database was unknown to the teams participating to the challenge. On this center, we have evaluated how automatic algorithms performed without knowing the image characteristics beforehand. The results of this comparison highlight for a large number of automatic algorithms a slight decrease in performance when encountering unknown images, even if they come from a common protocol. This evaluation per center and per evaluation metric is presented in Fig.
Discussion We have presented the first challenge based on an integrated computing platform, applied to multiple sclerosis lesions segmentation. The challenge computing platform was constituted of 1-a database to store the challenge images and results from the challengers, 2-a computing platform on which the evaluation was performed independently of the participants who were asked to post their processing pipelines, and 3-an automatic evaluation of the results against the "ground truth". This open-science computing platform has many advantages, including a fair comparison of the participants algorithms being run on the same platform and with the same set of parameters for all patients. In addition, the packaged algorithms may be re-used for other applications or if the validation database gets extended.
As future work, we plan at transitioning to use the BIDS format (http://bids. neuroimaging. io) to provide a standardized and more intuitive way of storing both the input data and output results. This would provide a great improvement in easing the pipeline design and integration in the platform. This platform was put together for the specific organization of a challenge on multiple sclerosis segmentation with a database of 53 patients each with an unprecedented number of seven manual segmentations from trained experts. A total of thirteen teams participated, illustrating the variety of algorithms both in terms of methodology and implementation. All results computed from the challenge were very insightful and revealed several points worth of discussion. First of all, despite that methods vary in their results, the point where a single automatic method is able to perform as well as the consensus of the experts has not yet been reached.
5 ), this composite method is not able to perform a good segmentation for these cases. Table  2. Number of lesions and lesion volume detected by each team and expert on the no consensus lesion case. We have also illustrated through this challenge the behavior of methods in several specific cases: testing for scanner dependency of the algorithms, where we compared the results for four different scanners, one of them being hidden from the training dataset. This study illustrated that all methods are still sensitive to scanners on which they were not trained for (either training in the machine learning sense or training in the sense of parameters tuning by a human being), obtaining lower scores for those images. On the contrary, when the training set included representative images of different scanners, all algorithms behaved equally independently of the center or scanner.
Evaluation metrics presented in this article are a selection per category (segmentation and detection) of the metrics described in Section 4. 4 and computed for the challenge. We chose them as being representative and most informative of the main qualities and defaults of the algorithms. Of course, as illustrated for more general segmentation evaluation purpose  13  , three metrics may not be enough for describing the behavior of each method in its entirety. As explained in Section 4, we have complemented these three measures with many other complementary measures, for which we encourage the interested reader to look at the supplementary materials (https:// doi. org/10. 5281/zenodo. 1307653).
Methods We present in the following the methodological details that allowed us to draw the results and conclusions previously outlined. This section is split in several subparts. Sections 4. 1 and 4. 2 present in more details the evaluation database used in the challenge and the way in which the manual delineations were carried out and averaged into a "ground truth". Section 4. 3 then outlines the computing platform used in the challenge which was necessary to guarantee a fair comparison of the algorithms. Finally, Section 4. 4 presents the evaluation metrics used in the challenge as well as the analyses plan derived from them in this article.
Reference Images Database. In this segmentation evaluation challenge, we relied on a database of images of 53 multiple sclerosis patients following the OFSEP protocol recommendations in  9  , which is currently applied in France for the constitution of the national cohort of MS patients. Following this approach allows for an evaluation representative of the current standards and easily usable for newly acquired images. For our challenge, images came from three different sites in France on four different MRI scanners from different manufacturers (Siemens, Philips and GE) including three 3T and on 1. 5T magnets. The repartition of the 53 patients is shown in Table  3. More demographic details on the ages and gender repartitions into the training and testing groups are provided in supplementary material (https://doi.
org/10. 5281/zenodo. 1307653). Overall, no significant difference of age can be seen between the different centers. While gender differences exist between some centers (in particular center 8), we believe this is of little importance with regard to lesion segmentation and detection quality compared to scanner to scanner differences. These patients were selected to have variable amounts of lesions both in volume and number, and from different centers to represent the variability that may be encountered across sites. For each patient, the following images were provided for each MS patient (see details of the sequence parameters in Table  4 ): a 3D FLAIR sequence, a 3D T1 weighted sequence pre and post-Gadolinium injection, an axial dual PD-T2 weighted sequence. These patients were then split (see Table  3 ) between a training and testing datasets.
The testing dataset was not made available to the challengers and was used to evaluate the different methods while the training dataset was provided, together with the ground truth segmentations, for challengers to train their algorithms. Images acquired on one center (3)  were not part of the training dataset, with the goal of evaluating how much algorithms were dependent on the training set and sensitive to acquisition settings. For each patient, the challenge data includes raw datasets, and preprocessed datasets where the following steps were performed: • Denoising of each modality using the non local means algorithm  14  • Rigid registration of each modality on the FLAIR image  15  • Brain extraction (skull stripping) using the volBrain platform  16  , from the T1-w image and applied to other modalities • Bias field correction of each modality using the N4 algorithm  17.
MS Lesions Ground Truth. Based on manual segmentation on our MS database, we first aimed at getting ground truth segmentations of multiple sclerosis lesions. This task is difficult and variability exists between experts depending on various factors, even when they follow common protocol, depending on many factors (image quality, training, modalities…). We chose to build for this challenge an unprecendented set of seven manual delineations for each patient. These delineations were performed manually on the 3D FLAIR image with control on the T2 weighted image. Each manual segmentation was performed by a trained junior expert, validated and corrected under the supervision of senior radiologists with a long experience in multiple sclerosis. More specifically, a first meeting between senior radiologists and workshop organizers of each site took place to determine the segmentation strategy and adopt a common tool (http://www.
This algorithm computes iteratively, using an Expectation-Maximization approach, a consensus segmentation based on penalties for individual deviations from agreement between manual experts segmentations. This algorithm has several advantages: it is robust to differences between manual expert segmentations, and it allows the computation of agreement scores with respect to the consensus segmentation considered then as ground truth. Computing Architecture for Automatic MS Lesions Segmentation Evaluation. One of the critical aspects in performing an independent challenge and benchmarking of medical image processing solutions is to provide a unified infrastructure able to: • anonymize and upload the training and testing data in a single place that all participants can access though the Web • integrate and execute the image processing algorithms through a web-based portal where all algorithms are executed on the test dataset in identical conditions • host the processed images and make them available to the participants • provide a cloud-based integrated solution with interoperable distributed resource management systems.
Most of the past and existing challenges in the field of image processing were able to provide part of these solutions but none of them was able to provide a computing solution able to perform all of these tasks seamlessly. We used the France Life Imaging (FLI) -Information Analysis and Management (IAM) (FLI-IAM in short) computing infrastructure for this challenge. FLI is a national infrastructure, which aims to coordinate and harmonize the network of resources on in-vivo imaging in France. Its IAM node represents the computing node of France Life Imaging (https://www. francelifeimaging. fr/en/about/noeuds/iam/). This architecture allows (see Fig. 7 ) the storage and management of preclinical and clinical in vivo imaging data and offers services of images processing and analysis. FLI-IAM is based on existing, technologically ready software solutions, coming from multiple research teams over France.
It proposes a web portal (blue box in Fig. 7 ) to unify the access to all resources and tools and provides multiple solutions for storage and computation on medical images. To operate this challenge, three components of the entire FLI-IAM portfolio have been used: • Web portal • Shanoir (SHAring NeurOImaging Resources) for the database  18  • VIP (Virtual Imaging Platform) for the computing platform  19. In addition to these three major components, additional tools/services have been developed to provide the required level of interoperability and to finally integrate all components into one unique workflow. The https://portal. fli-iam. irisa. fr/msseg-challengeweb portal has been used as a communication platform with all challengers. All information concerning the challenge has been distributed there, e.
g. the organizational aspects, dataset descriptions, evaluation details, etc. Challengers had to subscribe on the portal to participate in the challenge. https://shanoir-challenges. irisa. frShanoir (SHAring NeurOImaging Resources) served as central database for all datasets necessary for the challengers, all their processed results and challenger's scores. Shanoir is an open source neuro-informatics platform designed to share, archive, search and visualize neuroimaging data. It provides a user-friendly secure web access and offers an intuitive workflow to facilitate the collection and retrieval of neuroimaging data from multiple sources. Shanoir comes along many features such as anonymization of data, support for multi-center clinical studies on subjects or group of subjects. http://vip. creatis.
insa-lyon. frVIP (Virtual Imaging Platform) provided all necessary resources for the integration and the execution of all challenger processing pipelines. The pipelines were provided by challengers as https://www. docker. comDocker containers and were integrated into VIP using the Boutiques application repository. Boutiques relies on Linux containers to solve the problem of application installation in a lightweight manner and it uses a versatile JSON format to describe command line tools. VIP also ensured the execution of the challenger pipelines (and the subsequent segmentation performance analysis) on the computing resources available for the challenge. Figure  8  gives an overview of the integration level between database and computing platform and describes the workflow that was set up for the hosting of the challenge.
After the preparation of the challenge data, all source datasets were imported into Shanoir. The training data were shared with the challengers using the portal and its file download feature. The testing data was processed by VIP using a static export folder exported from the database -containing all necessary meta-data from the database to import results back into the database and attach them to the source dataset for each challenger. After the algorithms produced their results, the segmentation performance analysis (including all measures described in the next section) was run to compare challengers results with the consensus ground truth and calculate the scores. The continuously running DataTransferModule has been connected to the results folder of VIP to automatically import back the result datasets into Shanoir.
All result datasets and their corresponding scores have been made available in Shanoir for challengers. The challenge organization team could then easily access the scores summary within Shanoir. Challenge Evaluation Strategy and Metrics. Using the computing platform, challengers were asked to provide algorithms delineating lesions in the FLAIR reference space. This was asked to match the space in which the manual delineations were carried out and therefore avoid any unwanted discrepancies due to interpolation of the challengers' results. To evaluate these results, we have implemented a large set of evaluation measures for the challenge with the goal of evaluating the different aspects evaluated by clinicians when looking at MS patient images. For this reason, we have separated the evaluation into two major categories of evaluation metrics: • Segmentation evaluation: does the algorithm provide a precise delineation of each lesion? • Lesion detection evaluation: does the algorithm find all lesions in the image independently of its precise delineation? Each of these categories may contain several metrics that characterize differently the segmentation quality.
We describe in more details each of the chosen metrics for the challenge in the following sections. All evaluation algorithms used for this paper are available open-source as part of the Anima software (http://github. com/ Inria-Visages/Anima-Public). Although not presented in this article (but available as part of supplementary material (https://doi. org/10. 5281/zenodo. 1307653), we remind the strategy that was used at the challenge to rank, for each of these metrics, the different methods on all patients: • For each patient, compute the selected metric for each algorithm by comparing it to the ground truth previously computed • For each patient, rank the algorithms according to the selected metric (from 1: best performing to N: worst performing) • Compute for each algorithm its average ranking over all patients evaluated.
Acknowledgements This work was partly funded by  France Life Imaging  (grant  ANR-11-INBS-0006  from the  French "Investissements d' Avenir" program ) for funding and sponsoring the challenge. This work has also been partly supported by a grant (OFSEP) provided by the  French State  and handled by the  "Agence nationale de la recherche ", within the framework of the " Investissements d' Avenir" program , under the reference  ANR-10-COHO-002. We also thank the  French national cohort OFSEP (a French "Investissements d' Avenir" program ), and particularly the imaging group inside this cohort consortium for their constant support, fruitful discussions on the challenge and providing the MR images.
Methods: The data for this article were extracted from two distinct surveys. The first survey was conducted among HAE treating physicians and aimed to determine the availability and utilization of the various assays performed to help the diagnosis of C1-INH-HAE. The second survey was conducted with the various laboratories across Canada that performs the assays used in the diagnosis of HAE. The aim of this survey was to determine the availability and profile of the various assays used in the diagnosis of C1-INH-HAE in Canada, thereby ultimately bringing a rational basis for the biological testing. Results: C1-INH functional assay was widely available in Canada (93%), but was only offered by a small numbers of hospitals meaning that there could be longer delays in the analysis of these samples that may explain why the physicians expressed a lower level of confidence in this assay (59%).
Methods The data for this article were extracted from two distinct surveys carried out between May 2014 and November 2015. Missing or incomplete answers in both surveys were obtained through direct e-mails and phone calls. Both surveys were conducted by the Canadian Hereditary Angioedema Network (CHAEN), which is a non-profit organization that unites physicians committed to ensuring all HAE patients in Canada have access to excellent care that reflects current management and treatment guidelines, and works to promote research and education. The first survey was conducted among CHAEN members. The aim of this survey was to determine the availability and utilization of the various assays usually performed to help in the diagnosis of C1-INH-HAE across Canada. Respondents were asked about the four main assays used in the diagnosis of C1-INH-HAE: C1-INH function using chromogenic assay based on the residual C1 s peptidase activity (Dade Behring GmbH, Liederbach Germany) or the ELISA test based on complex formation (Quidel Corp, San Diego CA USA), antigenic C1-INH, antigenic C4 and antigenic C1q.
A real advantage given by NGS platform, when associated with copy number variation analysis, is to provide information about the size and localization of recombination fragments. Identification of family-specific variants is subsequently made possible by detecting the exact length and position of the deletion or insertion fragments after recombination between Alu repeated sequences. More than 500 SERPING1 variants have been reported in the on-line Human Gene Mutation database HMGD ®. Every new SERPING1 variant needs to be validated for its association with decreased C1-INH function and clinical phenotype. De novo mutations within SERPING1 gene are not uncommon, requiring further attention to family segregation. In proven C1-INH-HAE, genetic testing of APP and ACE deficiencies could potentially help in identifying patients with more frequent or more severe angioedema attacks  [36, 37].