dataesr/scientific-paragraphs-categorization huggingface dataset:
Introduction Since its advent in the late 1990 s, wireless capsule endoscopy has become an irreplaceable tool for the exploration of the small bowel (SB), and subsequently, for the entire gastrointestinal tract  [1]. Small bowel capsule endoscopy (SBCE) allows complete noninvasive exploration of the SB and its add-on value for the detection of mucosal lesions compared to conven-Multi-expert annotation of Crohn's disease images of the small bowel for automatic detection using a convolutional recurrent attention neural network Background and study aims Computer-aided diagnostic tools using deep neural networks are efficient for detection of lesions in endoscopy but require a huge number of images. The impact of the quality of annotation has not been tested yet. Here we describe a multi-expert annotated dataset of images extracted from capsules from Crohn's disease patients and the impact of the quality of annotations on the accuracy of a recurrent attention neural network.
Results The final dataset included 3498 images with 2124 non-pathological (60. 7 %), 1360 pathological (38. 9 %), and 14 (0. 4 %) inconclusive. Agreement of the experts was good for distinguishing pathological and non-pathological images with a kappa of 0. 79 (P < 0. 0001). The accuracy of our classifier and the available neural networks increased after the consensus annotation with a precision of 93. 7 %, sensitivity of 93 %, and specificity of 95 %.
Conclusions The accuracy of the neural network increased with improved annotations, suggesting that the number of images needed for the development of these systems could be diminished using a well-designed dataset. Supplementary material is available under https://doi. org/10. 1055/a-1468-3964 tional endoscopy and cross-sectional radiologic imaging has been clearly demonstrated in many pathologies. Over time, SBCE has proven to be a consistent tool for detection and evaluation of the severity of Crohn's disease (CD) lesions located in the SB  [2] [3] [4] [5]. Although the performance and acceptance of SBCE have been demonstrated  [6] , its use has not yet become standardized among the gastroenterology community for management of the patients with CD for a number of reasons. One of the main limitations is the time and expertise needed for a reading by gastroenterologists.
Various systems have been developed to limit the time required for gastroenterologists to analyze SBCE images. These systems can be classified into two main categories: algorithms based on extraction of well-identified features based on support vector machine and algorithms based on deep neural networks. The latter results in better performance in classification but requires more data for proper training. These algorithms currently are evaluated using different datasets containing a variable proportion of "pathological" and "non-pathological" images. Frequently, details concerning the modalities to attribute the "real truth" to each image remain unclear. Sometimes, no detail is available on the process used to classify the images  [11] [12] [13]. Most of the time, the images are selected by gastroenterology fellows supervised by a capsule expert  [14, 15]  or by only one expert in capsule endoscopy  [16].
When the annotation by two gastroenterologists is discordant, the final decision is accorded by a third one  [17]. Only in the CAD-CAP dataset, images were reviewed by a group of experts after their selection by gastroenterology fellows but the inter-observer concordance was not detailed  [15]. Moreover, datasets contain mainly images of ulcers and erosions and do not reflect the great variety of mucosal lesions visible in the SB of patients with CD, such as stenosis, edema, erythema. and mucosal break  [11] [12] [13] [14] 17]. The interobserver variability for annotation of capsule images in CD patients is unknown and the impact of possible errors of annotation on the performance of CAD tools has never been evaluated. The objectives of this study were first, to elaborate a dataset of carefully annotated images of SBCE extracted from patients with CD with a precise description of the process; second, to make it available for free to the scientific community to train their own CAD tools; and third, to evaluate the increased accu-racy of CAD algorithms based on deep learning at each step of creation of the dataset.
Dataset The CrohnIPI dataset is a multicentric dataset approved by the French Data Protection Authority and by the "groupe nantais d'éthique dans le domaine de la santé," the Ethics Committee of Nantes, France. Three French endoscopic units participated in the dataset. Four readers were involved in annotation of images: one gastroenterology fellow and three experts. The first reader, experienced in conventional endoscopy, had been trained to read capsule videos for the purpose of the study and was responsible for selection of images. The three experts had more than 10 years of experience in management of patients with inflammatory bowel disease (IBD) and had read more than 200 SBCE in patients with CD. Three annotation rounds were carried out to obtain a consensus annotation as close as possible to the "real truth.
Data-collection and definition of the lesions Third-generation SBCE videos (Pillcam SB3 system, Medtronic, Minnesota, United States) acquired between 2014 and 2018 from patients with CD or who had undergone exploration for suspicion of CD and registered in the three participating endoscopy units were retrospectively collected and de-identified. Only videos of patients with CD who had evidence of presence of SB lesions were selected. The successive still frames, in JPEG format, were extracted from the videos (without loss of quality), allowing their annotation. Clinical and demographic data at the time of the capsule were registered. Pathological lesions were defined as follows  [18] : ▪ Erythema: Area of reddish villi. ▪ Edema: Enlarged/swollen/engorged villi. ▪ Aphthoid ulceration: Diminutive loss of epithelial layering with a whitish center and a red halo, surrounded by normal mucosa.
Selection of images and first round of annotation Frames were selected and annotated by the initial reader (AM). All the frames selected by the reader, pathological or not, were considered as the frames of interest. Each frame of interest was extracted and included in the CrohnIPI dataset. When a frame contained more than one lesion, all the lesions were annotated. Images with a questionable lesion (type or presence) were labeled as "inconclusive. " Pathological images were selected regardless of the position of the lesion into the frame. The same proportion of non-pathological control frames, containing no lesion, was extracted from the same set of videos. Control frames were randomly and not successively selected independent of the presence of bubbles, residue, or turbid liquid to avoid any bias of learning.
Third round of annotation All the frames with a discordant annotation were reviewed. At this phase, only one lesion per frame was labeled, the most severe. The four readers met in three sessions to obtain a consensus annotation of the frames, considered as the "real truth" for each image. To prevent that one reader influenced the others, each of the readers had to give their opinion first, on 20 successive images, in the presence of an independent investigator not involved in reading the images. If necessary, a short adjacent video sequence could be retrieved, including 10 frames upstream and 10 frames downstream from the index frame. At the end of this process, discordant frames were classified as inconclusive and were excluded from the dataset for further analysis.
Statistical analysis Analyses were performed using statistical R Project software (version 3. 5. 3). For all statistical analyses, α = 0. 05 was considered as an acceptable threshold for type I error. Continuous variables are described using means ± standard deviations or median (25 th to 75 th percentile) and interquartile range. Categorical variables are described as raw counts and percentages. Whether after the first, second, or third round of annotations of lesions, there was no missing data and all readers annotated all the frames. Interobserver agreement for classification of images in the CrohnIPI dataset was evaluated using the inter-rater agreement Fleiss' kappa, with P value  [24]. Kappa value ranged from -1 to 1, with 0 value indicating statistical independence and 1 value indicating perfect agreement between observers.
A Fleiss' kappa between 0. 41 and 0. 60 can be construed as a moderate agreement, between 0. 61 and 0. 80 as a substantial agreement and above 0. 81 as a perfect agreement. Images from the CrohnIPI dataset were randomly distributed into three sets, with 70 % assigned to the learning phase, 10 % to the validation phase, and 20 % to the test phase. This last phase was for final evaluation of the algorithm performance. It is known that an unbalanced repartition of images over the classes can influence the algorithm's generalization capacity, and thus, its performance in real conditions. To avoid this phenomenon during the evaluation, a five-fold cross-validation was performed, involving five repeated measurements of the algorithm performance, each time using a different split in 70 %, 10 %, and 20 % of the images into the three groups and ensuring that each example had been used exactly once in the algorithm's test base for each measure.
Performance of the networks in the CrohnIPI dataset were evaluated by calculation of mean sensitivity, specificity, and precision for each of the five tests in the cross-validation. To evaluate the influence of multi-expert annotation, we trained and tested our neural networks on the first annotator dataset (AM) and then trained and tested on the multi-annotated dataset (AM, AB, CT, MF). A McNemar test was used to evaluate the differences in proportion of well-classified images by the neural network between the successive rounds of annotation.
CrohnIPI dataset Sixty-six videos containing at least one pathological frame obtained from 63 patients with CD were included in the dataset. Clinical and demographic characteristics of patients are detailed in Supplementary Table  1. Six patients were explored for a suspicion of CD, and at the end of the work-up, had a definitive diagnosis of CD based on the results of conventional ▶ Fig. 2  Flowchart of the study. The final dataset was obtained after the selection of non-pathological (NP) and pathological (P) images of interest extracted from 66 SBCE performed in patients with CD by an initial reader. All the images were reviewed and annotated by three experts. The discordant images were read again by all four gastroenterologists to obtain a consensual annotation. Inconclusive images (I) were excluded of the dataset.
014). In the most favorable situation (more and better data), it obtained a final precision of 93. 70 %, a sensitivity of 92. 09 % and a specificity of 94. 76 % for good detection of pathological/non-pathological images. Three additional and available pre-trained networks: ResNet 34 and VGGNet 16 and 19, were tested on the CrohnIPI dataset using the same modalities to verify that the increased performance of our network following the improved quality of the labeling was reproducible  [25, 26]. The results are shown in Supplementary Table  3. As for our recurrent attention neural network, the sensitivities, specificities, and accuracies increased progressively after each round of annotation, to reach a final accuracy of 94. 58 %, 94. 40 %, and 94. 35 % for ResNet 34, VGG 16, and VGG 19, respectively.
Discussion The CrohnIPI is a CD specific and dedicated dataset of pathological and non-pathological images carefully reviewed by several experts to obtain the consensus annotation as close as possible to the "real truth" for training, validation, and testing of CAD tools. After three rounds of annotations, the dataset contained 3498 well-coded images with a large variety of CD mucosal lesions and non-pathological images chosen independently of the quality of bowel preparation reflecting as close as possible real-life conditions. The network performance was good with an accuracy reaching 93 %. Moreover, we demonstrated, using four different pre-trained networks, that the performance increased when the algorithm was tested on the multi-expert-annotated dataset rather than on the first-reader-annotated dataset, highlighting the major importance of highquality annotation.
For the first time, the interobserver agreement between experts and gastroenterology fellows for classification of SB mucosal lesion often seen in patients with CD was evaluated and confirmed the difficulty in correctly classifying these lesions. The major strength of this study comes from the multi-reader annotation process described. The process that we used to create the CrohnIPI dataset has corroborated that readers agreed for classification of pathological and non-pathological images. However, the agreement concerning the different ▶ Fig. 3  Identified lesions retained to define pathological images. types of SB lesions frequently seen in patients with CD was weaker. This discrepancy was not reduced by the fact that all readers learned capsule endoscopy in the same endoscopy unit and by use of a standardized definition of each lesion.
Part of the disagreement could be due to the absence of common training before the annotation of images selected by the first reader. This could be tested later, during the next enrichment of the dataset with new images. The discrepancies concerning erythema and edema labeling also could be explained by the high variability and non-specificity of these lesions. This was highlighted by the difficulty to select typical images for the establishment of a consensus on the nomenclature and description of typical SB mucosal lesions of CD  [14]. The difficulty in classifying these lesions had a negative impact on the performance of our neural network and highlights the need for a more rigorously developed dataset to properly train deep learning systems. Other strengths of this study come from the use of native images on which no transformation and no filter were applied, permitting us to hedge on colorimetric bias and, unlike with other datasets, to use all typical lesions of CD without limitation to ulcerations.
Moreover, non-pathological images were extracted from the same videos as pathological images and the images were not selected with regard to the quality of preparation, presence or not of bubbles, residue, or lightness, so as to simulate real-life clinical practice as much as possible. It is noteworthy that, with a lower number of images, our neural network reached similar performance to other networks trained and tested on larger datasets. This is not specific to our network, as some available pre-trained networks, such as ResNet and VGGNet, have had similar performance greater than 90 %. This could be explained by the quality of the labelling of images and also by the ratio of pathologic to non-pathologic images. Our dataset contains as many normal as pathological images, while in larger datasets, most of the images are non-pathologic, even if the task assigned to the network is to detect images containing at least one lesion.
The performance of our neural network was insufficient, however, for classification of all types of lesions. The perform-ance was acceptable concerning ulcerations and stenosis but not for edema and erythema. This is clearly due to the non-specificity of these lesions and the great variability of their presentation. Other reasons could be the limited number of images for these classes available in the dataset and the imbalance between the different types of lesions. To improve the performance of the neural network, a larger number of images may be necessary or a different training strategy (e. g. sampling or weighting hard examples). Large dataset solutions have been explored in other studies for detection of erosions and ulcerations with datasets of more than 15 000 images, reaching accuracy rates close to 90 %  [11, 12].
By contrast, no data on edema and erythema have been published yet. Other limitations of the study come from the unique source of the images i. e. the Pillcam SB3, preventing generalization of this CAD tool to other devices and, despite the multicenter design of the study, by the learning of capsule endoscopy in the ▶ Table  3  Neural network performance evaluated on CrohnIPI dataset after successive rounds of annotation. same endoscopy unit by the experts, which could have introduced a bias compared to real-life multicenter studies. Future development of algorithms needs to be tested on different sources of images to be used in clinical practice. The promising performance of our recurrent attention convolutional neural network paves the way for development of new automatic diagnosis tools for clinical practice.
Even if classical architectures allow us to achieve significant detection performance, our attentional network aimed to improve the explainability of the model decisions  [20]. Indeed, it provides attention localizations, which could be used by experts to assess the network prediction. This will be explored in future research. All these tools need to be trained and tested on multiple and various images. The CrohnIPI dataset was built to be shared for free with the scientific community to facilitate and accelerate the development of such tools, which also will be accessible to gastroenterologists in the future. The CrohnIPI dataset will be enriched over time by including pathological and non-pathological images representing all types of lesion. A new process of annotation will be tested by selecting the frames of interest by the neural network itself with a posteriori validation by a group of experts in capsule endoscopy and IBD.
Conclusions In conclusion, we developed a dataset of images of SBCD lesions with a process allowing us to approximate the "real truth" as much as possible. We demonstrated that the performance of our deep neural network increased in parallel with the quality of annotations, highlighting the need for the best possible annotated dataset. The objective was to enlarge the base for the institutional research teams that need images for to develop new CAD systems. The CrohnIPI dataset can be downloaded, on-demand, at http://crohnipi. ls2n. fr/. The dataset is for research use only and protected under the creative commons license CC BY-NC-ND (https://creativecommons. org/licenses/by-nc-nd/2. 0/). All the data produced in part or in totality with the CrohnIPI dataset need to reference the original publication.
Swept-source intraocular lens tomography All included patients underwent preoperative sweptsource biometry acquisition using a IOL-700 Zeiss (Léna, Germany). The device uses a 1055 nm wavelength, with 44 mm A-scan depth and 6 mm width. Its measurement speed is 2000 A-scans per second, with a 22 μm resolution in the tissue. The IOL then provided the horizontal B-scan, which is displayed in the report as a 0°-180° B-scan. Lens pictures were imported from the Zeiss Forum software biometry report in Portable Network Graphics format (figure  1 ). Fixation was considered to be poor if foveolar depression was not visible on the OCT report, and dilatation was considered to be poor if this iris presence might alter the posterior cortex peripheral signal.
Image processing Linear densitometry was performed on the horizontal lens images obtained using the SS-OCT of the IOL measuring device. Original SS-OCT B-scans of the lens were exported to open-source ImageJ software (National Institutes of Health). The images were analysed according to the Scheimpflug imaging method previously described by Grewal et al  5  and transposed by Panthier et al 9 for OCT imaging. The lens area was selected using the magic stick tool, which delineated lens boundaries and defined the region of interest (ROI) centred on the lens nucleus (figure  1A ). The anterior cortex, nucleus and posterior cortex were considered following clinical slit-lamp observation, as illustrated in online supplemental figure 1. The ROI density was measured in pixel intensity units, on a scale from 0 to 255 (figure  1B ).
Open access The algorithm is based on an adapted random forest regressor machine learning approach (figure  2 ). It does not directly use the all-lens density, but rather the density values and variation between them from the anterior to the posterior side of the cataract. The wavelet transform allows for the capturing of density patterns and variations along the lens, which are used by the algorithm to create a number of classifying decision trees for various subsamples of the dataset. Then, the algorithm uses averaging of comparable database samples to predict poBCVA for the test samples, providing more discriminating information compared with mean cataract density.
Low or high BVCA data under-representation in the algorithm database may cause a bias of recognition and statistical overestimation or underestimation, respectively. The algorithm analyses patterns of density variation, compares them to the database and may lack comparable data in limited visual acuity. Moreover, the alternative comparative method (75% database, 25% analysis) could alter this statistical bias. Retrospective medical record analysis and the high number of operators may cause selection bias. However, all patients had standardised poBVCA evaluation using the same refraction systems, and biometry was acquired using the same machine. Single-operator screening could reduce this bias in further evaluation. The IOL Master 700 performs six scans with different sagittal axes of the eye, but only the horizontal scan was included in analysis.
07 mm. The difference is still very low. These low-threshold reconstructions of Fig. 7  are actually very powerful tools to adjust the exact constraint's threshold of the reconstruction, i. e. the one that gives the final particle's reconstructions of Figs. 2(c ), 3(c), 4(c) and Fig. 5(c ) respectively. In particular, they are very useful to identify the real sizes of particle's branches. In summary, the ER-algorithm gives relatively good reconstructions of the initial object, although a precise size estimation can be more difficult in absence of this complementary low-threshold reconstruction. A very important criterium is the calculus time. Each reconstruction presented in this section requires only three iterations. Using a matlab code on a standard portable computer, the reconstruction from an interferometric image of 680x680 pixels requires between 30 s and 2 min (depending on the computer).
Ethics This is an ancillary study of a prospective cohort study of patients with COVID-19 infection hospitalized in ICU at Amiens University Hospital (NCT04354558). This study was approved by the Amiens University Hospital IRB (Comite de Protection des Personnes Nord-Ouest II CHU-Place V. Pauchet, 80054 AMIENS Cedex 1, CNIL Number: PI2020_843_0026). In accordance with French law on clinical research for non-interventional studies, informed consent was waived but oral and written information as provided whenever possible to the patients and systematically to their families specifying that they could oppose the use of their data  [14].
Data Data from electronical data, medical reports and biological values were collected prospectively. SARS-Cov2 infection was confirmed by a positive Reverse transcription polymerase chain reaction on nasopharyngeal swab or bronchoalveolar lavage on admission to our ICU. The severity of illness at the time of TTE exam was evaluated by the simplified acute physiology score (SAPS) II  [15]  and the Sequential Organ Failure Assessment (SOFA) score  [16]. Vasopressor use was evaluated by the SOFA cardiovascular (SOFA cv) score  [16]. Severity of COVID-19 pneumonia was defined according to the World Health Organization (WHO) case definition  [17]. The severe group included patients with respiratory distress syndrome (respiratory rate ≥ 30 bpm) and/or oxygen saturation ≤ 93% at rest and/or ratio of arterial partial pressure of oxygen to fractional concentration of oxygen in inspired air < 300 mm Hg and/or > 50% lesion progression over 24-48 h by pulmonary imaging.