dataesr/scientific-paragraphs-categorization huggingface dataset:
ACKNOWLEDGEMENTS We would like to thank  Prof Zhigang He  for critical reading of this manuscript and  Dr. Chen Wang  for technical assistance ( Boston Childrens Hosptial-Harvard Medical School ). We thank  Charlotte Corrao  and  Mohamed-Elmehdi Boughanmi  for virus preparation. This work was supported by a grant from  ANR  to HN ( C7H-ANR16C49 ) and  SB  ( ANR-18-CE16-0007 ), from  European Research Council  ( ERC-St17-759089 ) and  NRJ Foundation  to HN. This work was supported by grants from the  French National Research Agency  in the framework of the  "investissements d'Avenir" program  ( ANR-15-IDEX-02  NeuroCoG (HN, SB and CT)). JS is supported by  UNADEV/AVISAN  grant (Appel à projets 2017 "Maladies de la vision, origine et traitement") and by  Fondation pour la Recherche Médicale (FRM) postdoctoral fellowship  ( SPF201909009106 ).
Statistical Analysis Quantitative data were analyzed using SPSS software, version 24 (IBM Corp). Means and SDs for each of the items in the AGREE-REX Usability Survey were calculated. Cronbach α and correlations-ifitem-deleted were calculated to assess the internal consistency of the items. Intraclass correlations were calculated for 2 to 5 appraisers using the Spearman-Brown reliability adjustment to assess the reliability of the overall AGREE-REX score. 29, 32, 33 A 2-tailed P <. 05 was considered as statistically significant. Differentiating itself from the AGREE II tool, the AGREE-REX tool evaluates the quality of CPG recommendations, defined as the extent to which they are credible and implementable. Thus, to explore construct validity, correlations between the overall AGREE-REX score and the implementability score and the clinical credibility score were calculated, with the expectation that positive correlations would emerge.
As an exploratory measure of discriminant validation, the correlations between the overall AGREE-REX score and AGREE II domain scores, assuming the mean scores across 4 raters and correcting for the attenuation in the correlation due to measurement error, were also calculated. The correlations of the former were expected to be larger than those of the latter. No standard for CPG recommendation quality currently exists; thus measures of criterion validity were not appropriate. 23, 32, 33 rticipants provided written feedback, and themes that emerged were noted. Formal thematic analysis was not undertaken. Using the quantitative data and the written feedback from participants, the research team used an iterative process to refine the Draft AGREE-REX tool. This refinement was achieved through an in-person meeting, a feedback session with stakeholders at the 2017 Global Evidence Summit,  34  and multiple teleconference meetings with the AGREE-REX team (2017-2019).
Results Of the 692 individuals who responded to the advertisement and were emailed a formal invitation, 322 (47. 0%) completed the study. Of the 322 respondents, 202 (62. 7%) were female, 252 (78. 2%) had some experience with the AGREE II tool, 188 (58%) indicated that English was their first language, and 170 (53. 8%) identified themselves as CPG developers (  The AGREE-REX instructions are clear 5. 8 (1. 06) The AGREE-REX instructions are helpful 5. 9 (1. 00) The AGREE-REX instructions are complete 5. 8 (1. 11) The AGREE-REX was easy to use 5. 4 (1. 32) I felt confident when applying the AGREE-REX to a guideline 5. 1 (1. 43) The AGREE-REX is complete; there are no missing items 5. 7 (1. 18) The use of multiple evaluation statements for each of the 11 items is appropriate 5. 5 (1. 52) The use of a 7-point response scale is appropriate 5.
Final Refinements Based on the study results and feedback from participants, changes were made to the tool. Table  5  lists the final items and criteria. eAppendix 1 in the Supplement compares the draft with the final version 1 of the tool and eAppendix 2 provides the entire AGREE-REX User's Guide. The original 11 items were edited to 9 items (2 items combined and 1 item deleted) and clustered into 3 conceptual categories: clinical applicability, values, and implementability. The original 3 response scales were modified to 2. The mandatory quality assessment scale asked appraisers to rate on the 7-point scale the overall quality of the item by considering whether the item criteria were addressed in the CPG and influenced the recommendations-for example, the extent to which data on the values and preferences of the various stakeholders were obtained and reported and extent to which these data were explicitly considered in formation of the recommendation.
Limitations This study has limitations. The measurement properties and usability surveys were performed with the penultimate draft version of the tool. Financial considerations prohibited the repetition of the studies to confirm that the changes made to the AGREE-REX tool were associated with improvements in measurement properties and usability. Nonetheless, we believe that decisions for modifications made were informed by evidence. Capturing information from in-the-field experiences on an ongoing basis will be essential in continuing to develop the evidence base to support use of the AGREE-REX tool. Additional supporting materials (eg, training tools) are being developed to improve interrater reliability of the tool. Another limitation is the criteria used to select the CPGs (<50 pages, English language only) and that the tool was applied to the whole set of recommendations in each report.
Results: The performance of networks trained on the RETOUCH dataset is higher than human performance. The analysis further generalized the performance of the best network obtained by fine-tuning it and achieved a mean Dice similarity coefficient (DSC) of 0. 85. Out of the three types of fluids, intraretinal fluid (IRF) is more recognized, and the highest DSC value of 0. 922 is achieved using Spectralis dataset. Additionally, the highest average DSC score is 0. 84, which is achieved by PaDeeplabv3+ model using Cirrus dataset.
This modality is also further improved to enhance the image quality and time required for a single scan to be taken by incorporating the spectraldomain optical coherence tomography (SD-OCT). Various suppliers of SD-OCT are currently available, with each of them offering different features relating to the signal-to-noise (SNR) ratio as well as resolutions. The image quality produced by OCT scans declines due to the eye movements by the patient during the scanning process. Vendors are required to decide on the trade-off between the signal-to-noise ratio (SNR), photography time, and volume resolution. An example of the OCT volume dataset released for the OPTIMA challenge by four main vendors (Cirrus, Nidek, Spectralis, and Topcon) is shown in Fig. 1. The Spectralis device has the highest image quality due to the scan averaging process that reduces the noise.
Related work As the healthcare systems worldwide are vastly improving, the increase in life expectancy leads to the growth of the elderly population. Automatic age-related macular degeneration (AMD) segmentation using spectral-domain optical coherence tomography (SD-OCT) has become an active area of investigation since it provides a better view for doctors to examine the status of the retina. SD-OCT was introduced commercially in 2006 and has since grown rapidly in the field of image analysis as compared to other medical imaging modalities. Datasets were released for the segmentation of either SRF, IRF, PED, or a combination of these fluids. However, not all the AMD fluids were available in the dataset until the RETOUCH challenge  [4]  provided a dataset to benchmark the detection of AMD fluids.
Schlegl et al. [34]  initiated the first proposal to segment IRF and SRF fluids using voxel classification in patch-based networks in both weak and fully supervised settings. Subsequently, this study was further validated in a larger dataset, including cases of AMD, RVO, and DME, and showed excellent performance for IRC quantification  [20]. Other deep learning methods yielded a dice coefficient of 0. 729 based on the investigation performed using a set of 934 B-scans with different pathology types acquired with a single SD-OCT vendor as opposed to the assessment based on SD-OCT volumes  [17]. Hence, the accurate segmentation of retinal layers would greatly enhance the precise segmentation of AMD fluid, as demonstrated using ReLayNet  [16]. However, the developed method was evaluated in a relatively small dataset consisting of 110 scans from 10 patients obtained by a single OCT volume.
Another deep learning algorithm used for the segmentation of IRC in SD-OCT volumes was obtained with the Spectralis device  [18]. The algorithm trained 221 SD-OCT volumes and used an extended Unet model. The OPTIMA dataset was used to generalise the model by re-training the model. Table  1  shows a summary of studies that have applied deep learning models for retinal fluids segmentation. In the RETOUCH challenge dataset, the RMIT group  [22]  used a modified U-net model accompanied by an adversarial network. Similarly, the UCF group  [23]  proposed a deep CNN ResNet based architecture for pixel classification. In the training phase, data augmentation was used to increase the size of samples up to 45 times using a novel method known as myopic warping that causes the retina to look curvier. Images were fitted to the CNN model, comprising 43 convolutional layers with 32 layers in the encoder.
False-positive pixels were ruled out by applying a random forest classifier. The OPTIMA challenge released 30 SD-OCT volumes, with five groups contributing to the dataset. However, only one group employed a deep learning model. Venhuizen et al. [15]  utilised three CNNs that were trained independently at different ranges to detect the IRF lesions. The network trained three sizes of patches separately and fused the three scales by applying the AND operation. On the other hand, Girish et al. [19]  used a modified U-net model in their study. An improvement achieved by Girish et al. [21]  was the use of depthwise separable convolution operations to the encoder-decoder model. Additionally, Gopinath et al. [14]  proposed the CNN model and their work was validated using three datasets, including the OPTIMA dataset.
While most of the studies employed the U-net model with some modifications, several other models were tested in this study using two datasets, RETOUCH and OPTIMA, to identify the best model. Additionally, the benefits of utilising patches such as in other medical imaging modalities were not well elucidated in these studies. Thus, this study aims to analyse the retinal diseases (IRF, SRF, and PED) using four different deep learning architectures and provide a comprehensive understanding of its advantages and limitations.
Dataset This study used two publicly accessible datasets, RETOUCH challenge dataset  [4]  and OPTIMA challenge dataset  [35]. RETOUCH challenge released the largest dataset to detect and segment three retinal fluids (IRF, SRF, and PED). Another dataset used for generalising network performances is the OPTIMA challenge dataset that aimed to detect the intraretinal cystoid fluid (IRC). Both datasets include training and testing sets. However, labels were not provided for the testing set in the RETOUCH challenge dataset (held by the challenge organizers), only the training sets were used in this study. The number of volumes used in this work is 100 volumes (70 volumes from RETOUCH dataset and 30 volumes from OPTIMA dataset). Table  2  provides the main characteristics of each dataset.
Pre-processing This section describes the steps used for the pre-processing of the datasets for both the RETOUCH and OPTIMA challenges. The SD-OCT volumes were processed as follows: 1. B-scan extraction : For the adequate comparison of all network performances, 2D B-scans were extracted from each volume and each B-scan was concatenated three times and stacked together to form an RGB image format. In this process, the volume size dimension was altered (e. g. from 512 x 1024 x 1 x 128 to 512 x 1024 x 3 x 128) for the Cirrus scanner. Additionally, all the B-scans and patches were in a 3-channel format. The primary reason for this concatenation was the existence of the VGG network in Seg-net that only accepted a three-channel input. Subsequently, the patches were extracted from the B-scans in various size dimensions of 32 x 32, 64 x 64, 96 x 96, 128 x 128, 256 x 256, and 512 x 512 to examine the effects of patch size.
Only the best patch size performance was reported in this study. Since the volumes varied in size, all the B-scans and their ground truths provided by the RETOUCH challenge were resized to 572 x 572 x 3 using the nearest-neighbour interpolation. For the OPTIMA dataset, two ground-truths were provided by two graders. Thus, the simultaneous truth and performance level estimation (STAPLE)  [36]  algorithm was used to generate a single reference image, and the ground truth was equally resized to the images in the RETOUCH dataset as shown in Fig. 3. 2. B-scans de-noising : All the B-scans were denoised using blockmatching 3D filtering (BM3D)  [37]  with a sigma value of 25 to trim the speckles in the scans. The denoised patches were then extracted from denoised B-scans. Fig. 4  shows an example of denoised images.
The network architecture CNN architecture has been applied exceedingly well to several medical imaging applications for various tasks. A higher performance has been achieved using deep convolutional neural networks (DCNNs) as compared to hand-crafted feature systems. The integration of DCNNs into local image transformation allows DC-NNs to acquire abstract data representation in a local image. Although this integration was coveted for classification tasks, semantic segmentation tasks were prevented, and this led to the loss of spatial information. At present, the main challenge for medical imaging segmentation tasks is the high ratio difference between the different classes (specifically between the background and disease pixels). Thus, encoder-decoder networks such as U-net, Segnet, and Deeplabv3+ were designed to overcome these limitations.
In this study, the 8s model and VGG16 model were used as the framework. Additionally, FCN model was trained with patches and is referred to as patched FCN (PaFCN). 2. U-Net  [31]  : This network was introduced for biomedical imaging applications, and it overcame the shortage of data size by creating patches. This network consisted of four max pooling operations and four up-sample operations. Contextual information was obtained from the left side (encoder), while information on localisation was supplied from the right side (decoder). In this study, the vanilla U-net as well as the extended form of U-net were trained with two extra max pooling operations to allow the training of images at different scales. Subsequently, shortcut connections transferred the high-resolution information from the left side to the low-resolution information on the right side through the addition of pixels for resolution compensation.
Results Various models with different inputs (all raw images, images containing diseases, and all denoised images), and the effect of using overlapping patches as input to the models were investigated in this study. Additionally, different weight balancing techniques on the best performing network were also investigated. Besides, the optimal number of volumes required to feed the trained network was experimented, which aids in reducing the analysis time instead of re-training the network. Lastly, the trained networks were trained using RETOUCH datasets as opposed to the OPTIMA testing datasets, and the benefits of fine-tuning to boost the performance were described.
Comparison of network input The B-scans were extracted from each volume for all the experiments, in which a data size of 16 volumes from each scanner was used to train the neural models. Data were tested from Cirrus (8 volumes), Spectralis (8 volumes), and Topcon (6 volumes). After training the networks, a median filter was applied to reduce the false positive pixels, followed by morphological operations to increase the performance by 1% to 2%. However, the result was not significant (p = 0. 17) despite these processing steps. Each network was trained thrice with different hyperparameters and the score of the three networks was ensembled using the majority voting rule. The network ensembling procedure reduced the generalisation error and yielded a slight improvement in all the models (p = 0.
Exp 5 PaDeeplabv3 + is fine-tuned, such as the trained network on Cirrus was fine-tuned over Spectralis and Topcon datasets with a various number of volumes. The same process is applied for other datasets to decide how many volumes are enough to supply the network, aiming to save time while maintaining almost the same accuracy as training from scratch. Exp 6 Testing the OPTIMA dataset using networks in experiment 3. Nidek dataset was tested directly from all networks trained in all scanners using the RETOUCH dataset, and no fine-tuning is employed yet. Exp 7 Fine-tuned networks in experiment 3 using the OPTIMA training dataset and testing using OPTIMA testing dataset. Exp 8 Nidek dataset from OPTIMA challenge is trained from scratch and tested using the provided testing data. than Seg-net.
The performance for each network is displayed in Table  5. The number of images that displayed retinal diseases in the training data from Cirrus (16 volumes) was 1083 B-scans, while Spec-tralis and Topcon scanners had 467 and 983 images, respectively. Hence, the total number of diseased images was 2533 images acquired from 48 volumes. The total number of images used in Experiment 1 was 4752 images derived from 48 volumes, thus representing less 2219 images used when testing with the whole volume size (background and diseased images). Therefore, it is evident that training the network with fewer images was a faster process. Some networks performed better using only the diseased data, but the results were not significant. Most networks performed better in Experiment 2 for IRF and PED detection using the Cirrus dataset.
The identification of SRF was improved in Experiment 1 and Experiment 2, except with the use of the Spectralis scanner data. Patched Deeplabv3+ and patched extended U-net models also performed significantly better than the other networks (p = 0. 047), whereby the patched networks per-formed better than the non-patched networks as shown in Figs. 5 , 6  , and 7. In Fig. 8  , the patched deeplabv3+ and patched extended U-net models were evaluated using the RETOUCH testing dataset. All networks were tested using the Wilcoxon T-test to measure the statistical significance of the architectures.
Network optimization In Experiment 4, the effects of different balancing techniques were verified to determine the best algorithm. The effects of the weight balancing algorithms for the two highest-performing networks (PaDeeplabv3+ and PaEx-Unet) are displayed in Table  7. PED  has a better segmentation in uniform class balancing as compared to inverse frequency balancing for both networks, while IRF has a higher recognition in inverse frequency balancing as compared to uniform class balancing. It was shown that the median frequency balancing method outperformed all the other balancing techniques and hence, all the experiments in this study utilised the median balancing algorithm. In Experiment 5, the two best-performing networks from Experiment 3 were selected. Both these models were tested using the OPTIMA dataset.
For example, the PaDeeplabv3+ network was trained using the Cirrus dataset but evaluated using the Spectralis and Topcon dataset. For data obtained from different vendors, the networks were re-trained to achieve a specific task. The training of deep models from scratch takes a long time and hence, obtaining a generalised network is crucial. The PaDeeplabv3+ network was selected to evaluate the fine-tuning stage. The model was trained with various volume numbers, as shown in Table  8. It should be noted that three volumes were not sufficient for the identification of fluids, in which the doubling of the volume number led to better performance. It was also observed that the use of nine volumes significantly improved the network findings (p = 0. 038), while feeding the network with 12 volumes only slightly improved the performance with no significance (p = 0.
23) observed in the results. Hence, the addition of more volumes did not significantly improve the performance. Moreover, some networks performed better without the need for fine-tuning at the cost of spending more time to train the network. For example, the fine-tuned Cirrus network using the Spectralis datasets and many others did not exceed    the networkâs highest performance. Lastly, fine-tuning achieved a relatively good performance as compared to networks that were trained from scratch. However, the time taken to perform any fine-tuning in model training was reduced to half. The figures displaying the results using the RETOUCH dataset was depicted in yellow (IRF), dark blue (SRF), and light blue (PED).
Towards a generalised performance The data for Experiment 6 ( Table  9  ) represents the performance of networks that were trained using the RETOUCH dataset and tested using the OPTIMA dataset. The highest performing networks were selected from Experiment 1, 2, and 3 to generalise the performance of these networks. The RETOUCH datasets consist of four classes (background, IRS, SRF, and PED), while the OPTIMA datasets have only two classes (background and cystoid). Hence, the resulting images from the networks were sent for evaluation after converting the SRF and PED expected pixels values to the background pixel values. PaDeeplabv3+ and PaEx-Unet outperformed other networks in both Experiment 6 and Experiment 7, as shown in Fig. 9. Fine-tuning the neural models improved the overall performance.
Models trained with the Nidek dataset also enhanced the generalisation of the networks, as shown in Table  10. In Experiment 8, networks were trained from scratch using the same setup as in Experiment 1, whereby the PaDeeplabv3+ model outperformed all other networks ( Table  11  ). The best performing neural networks used in this study outperformed all the models proposed by the teams who joined the OPTIMA challenge and achieved a similar or slightly better performance. The number of volumes released from the Nidek scanner was low (only three volumes), while training the network from scratch led to better performance when compared to the fine-tuned networks. Retinal fluids vary in size and small lesions are difficult to be detected by the neural system. Nevertheless, PaDeeplabv3 and PaEx-Unet per-formed better than the other networks and this observation may be due to the input of these networks being trained at multiple scales.
Discussion In this study, three retinal fluids were segmented using four different deep encoder-decoder neural networks with optical coherence tomography volumes. This study focused on the importance of fitting patches to the network models. The difficulty in using these datasets was reported in the RETOUCH challenge  [4]  , whereby the average DSC performance based on data testing was 0. 73. It is evident that if the graders were to have the same opinion for each disease, the performance of the network would have been better. The manual testing based on the human eye is not sufficient. Hence, the networks proposed in this study were shown to perform slightly better than the grader level and some of the teams who participated in the training of the data. It is also worth mentioning that no extra data was used to train the networks in this study.