dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgements We would like to acknowledge the  University of Colorado Boulder 's next-generation sequencing core headed by  Dr. Amber S. Scott  at the  University of Colorado , and  Melanie Magnan  at  INSERM  for support with deep sequencing for our study. We would like to thank  Dr. Erick Denamur ,  Dr. Ivan Matic ,  Dr. Philippe Nghe , and  Maureen Micaletto  for useful discussions and for proofreading our manuscript. A. C. and R. T. G. were funded by the  US Department of Energy  grant  DE-SC0018368  awarded to R. T. G. A. C. , B. G. , Z. D. , R. F. , and O. T. were funded by grants  European Research Council ,  FP7  grant  310944 , the  French Agence Nationale pour la Recherche ANR GeWiEp  ( ANR-18-CE35-0005-01 ), the  Fondation pour la Recherche Médicale  ( EQU201903007848 ). A. C.
Data availability All sequencing data has been deposited at the Europen Nucletide Archive (ENA) and are accessible through the project accession number PRJEB59215. The sequencing data can also be accessed at DRYAD database: https://datadryad. org/stash/share/pMgkxJauRphu1eU49Xyx LOznG6zakpHaWBgLqzCcL-Y. The processed data is accessible with the Dryand link: https://datadryad. org/stash/share/ZfUDt7kbGXPGw4_ tl_roBQ2okJde3oKFftqgAwYlF0o. Source data is provided with the paper. Source data are provided with this paper.
By contrast, transformer architectures use the self-attention mechanism to propagate information over time, and each region of the sequence pays attention to all other regions. In this way, all the dependencies-even those at a very long range-are modeled without loss of information. Furthermore, transformers are parallelizable, as they can process multiple inputs simultaneously. Given these characteristics, the advantages of using transformers led us to choose this architecture instead of LSTM or the RNN. The proposed method-based on Vision Transformers (ViTs)  [17] -addressed the limitations of conventional RNNs in terms of gradient flow by surpassing classical LSTM and the identity RNN (IRNN) in terms of performance on several benchmark image datasets. Indeed, the ViT feature extraction method is based on linear embedding of the patches in which the input image is divided.
The results highlighted that ViTs, when adequately pretrained with large data, exhibit similar or superior robustness when compared with ResNet models across various perturbations. Additionally, this study revealed that transformers demonstrate resilience when specific layers are removed, indicating their flexibility. Although activations from later layers exhibit strong correlations, it is evident that each layer contributes significantly to the classification process. The authors of  [19]  introduced a comprehensive framework (called the Classification Transformer (C-Tran)) for multi-label image classification that leverages transformers to capture the intricate relationships between the visual features and labels. The proposed approach involves training a transformer encoder using masked labels and the visual features from a CNN to predict a set of target labels by achieving state-of-the-art performance on challenging datasets such as COCO and Visual Genome.
The remainder of this paper is organized as follows. Section 2 provides the characteristics of the dermatoscopic melanoma image dataset used and processed in this study. Section 3 describes the implemented methodology to set up the DL predictive model, detailing each step of the processing pipeline. Section 4 illustrates the obtained experimental results concerning each specific step and the performance of the built-up model, both in the training and validation phase and in the test phase. Section 5 delves into the obtained results, offering comprehensive analysis from both the technical and clinical perspectives. Finally, Section 6 provides a concise summary of the study's key findings and draws meaningful conclusions.
Dataset The dataset considered for this study is based on a collection of dermoscopic images of skin lesions obtained from the ISIC Archive (ISIC Archive website https://challenge. isicarchive. com/data/#2017, accessed on 1 June 2023). Specifically, the dataset served as the basis for the ISIC 2017 Challenge (Skin Lesion Analysis toward Melanoma Detection)  [22] , where researchers and clinicians worldwide competed to develop innovative algorithms and tools for the automated diagnosis of skin lesions, including melanoma. Metadata, including the patient's age, sex, and lesion location, accompany each image. The images were labeled by expert dermatologists who identified the presence or absence of melanoma based on histopathology examination or clinical examination. This procedure ensured the accuracy and reliability of the labels.
The availability of such a comprehensive and annotated dataset has facilitated research efforts in this field, leading to the development of more accurate and efficient diagnostic tools for melanoma and other skin cancers. The ISIC 2017 Challenge consisted of three tasks: lesion segmentation, dermoscopic feature detection, and lesion classification. In this article, we focused on the lesion classification task, aiming at the classification of skin lesion images into three classes: malignant melanoma (MM), nevocellular nevus (NCN), and seborrheic keratosis (SK). Melanoma is defined as a malignant skin tumor derived from melanocytes (melanocytic). Seborrheic keratosis (SK) is defined as a benign skin tumor derived from keratinocytes (non-melanocytic). A nevus is defined as a benign skin tumor derived from melanocytes (melanocytic).
The training dataset contains 2000 color dermoscopic skin images in JPEG format, comprising 374 images from melanoma cases, 254 from seborrheic keratosis cases, and the remainder being benign nevi (1372). The validation and test sets were evaluated in new databases with 150 images (30 for MM, 42 for SK, and 78 for NCN) and 600 images (117 for MM, 90 for SK, and 393 for NCN), respectively  [22]. The images were of many sizes (from 1022 × 767 to 6748 × 4499 pixels) and photographed with different angles, artifacts, and lighting conditions. The images were acquired from various dermatology clinics and hospitals across different continents, ensuring diversity in the dataset. Figure  1  represents examples from the three classes: MM, NCN, and SK. Inhomogeneous dark spots with irregular shapes usually characterized the MM class, and similar features were shared with the NCN and SK classes, making the classification task particularly challenging.
Experimental Results The experiments were conducted in the Google Colab environment while deploying the models available on Hugging Face. The best selected model was ViT-Large (307 million parameters), with an input resolution of 384 × 384 and a patch size of 32 × 32. The proposed model, which was pretrained on ImageNet  [27] , was fine-tuned on the ISIC 2017 dataset for 40 epochs using the hyperparameters listed in Table  1. The ViT-based model was trained using several configurations, which are listed in Table  2. In particular, the performance was evaluated by varying the learning rate of the Adam optimizer and changing the number of hidden layers. The scheduler, which was used for the training phase, applied a linear learning rate variation with a warm-up. When the set point for the learning rate (maximum value to be reached) was chosen, the scheduler started from a lower value until the set point was reached and decreased with a gentler slope in the last few epochs.
Discussion In the literature, various methods have been proposed to improve the decision-making process in the diagnosis of skin lesions, particularly in cases of melanoma. The authors of  [28]  proposed a comprehensive comparative study of U-Net and attention-based skin lesion image segmentation methods. The results indicated that the hybrid TransUNet outperformed the other benchmarking methods in segmentation tasks, achieving an accuracy of 0. 921 and a Dice coefficient of 0. 898. The authors of  [29]  proposed the use of fully Transformer networks (FTNs) for skin lesion analysis, a hierarchical transformer that uses a spatial pyramid transformer (SPT) to capture long-range contextual information from skin lesion images. Their conducted experiments on the public International Skin Imaging Collaboration (ISIC) skin lesion segmentation and classification datasets demonstrated the effectiveness and efficiency of FTNs.
In  [30] , the authors presented a novel deep learning-based framework named Attention Deeplabv3+ for skin lesion segmentation. Their proposed method uses attention mechanisms in two stages to capture the relationships between channels and to emphasize the relevant field of view more. Their experimental results on public skin cancer datasets showed high state-of-the-art performance. The authors of  [31]  proposed a novel self-attention-based network for diagnosing melanocytic lesions from digital whole slide images. The method outperformed other stateof-the-art methods and achieved results comparable to 187 practicing U. S. pathologists. In  [32] , the authors proposed a framework that employs methods for data augmentation and a Medical Vision Transformer-based classification model for classifying skin cancer.
The results achieved on the HAM10000 datasets showed that the proposed model outperformed the state-of-the-art techniques for skin cancer classification, proving that early detection can increase survival rates by up to 70%, leading to improved outcomes for patients with this deadly disease. However, the potential of ViTs has not yet been exploited to its fullest in the classification of melanoma skin cancers. Melanoma detection is a critical task in improving cancer diagnoses and prognoses. To this aim, this work proposed a ViT-based architecture for efficient MM recognition compared with the NCN and SK classes. The proposed model (training and testing procedures) was built up using publicly available skin cancer data from the ISIC challenge. This choice enabled a fair comparison with other competitors who have employed similar methods to address the same clinical issue by using the same dataset.
The performance in classifying skin lesion images of the introduced architecture based on ViTs outperformed the current state-of-the-art models. This improvement can be attributed to the model's ability to capture and model long-range spatial relationships within the images effectively. Table  3  shows the performance of our proposed ViT-based approach and a comparison with the other state-of-the-art solutions that used the ISIC 2017 dataset. It should be noted that in the specific application scenario approached by our study, there was no possibility of changing (increasing) the number of classes (i. e. , the MM, NCN, and SK classes). Indeed, the number of classes was established in the ISIC 2017 skin lesion classification challenge to which the dataset refers. The optimal configuration is able to accurately identify images from all three categories, showing remarkable classification capabilities on the given dataset.
Conclusions Melanoma detection is crucial to improving cancer diagnoses and prognoses. This work presented a ViT-based model to efficiently recognize malignant melanoma, nevocellular nevus, and seborrheic keratosis cases. The model was evaluated on a known dataset from the literature called the ISIC challenge to ensure comparison with the state of the art. The results show how the proposed architecture outperformed the state-ofthe-art models mainly based on convolutional networks. Ablation studies conducted on the presented model with its hyperparameters highlighted the robustness of the model for classification purposes. Future plans could include the use of attention maps to provide explainability to the model and understand which regions of the images are relevant for classification purposes.
[30]  The spatiotemporal drug release kinetics were evaluated with continuous flow of aqueous solution through the hepatic vein to mimic the tumor-specific drug depletion condition. Doxorubicin, one of the most frequently applied anticancer drugs in TACE, was tested. [3, 31, 32] The red color and strong fluorescence of doxorubicin combined with the excellent optical transparency of the DLM model enables a direct observation of the drug distribution using bright-field and fluorescent microscopies. Quantitative evaluations of the spatiotemporal drug diffusion dynamics in the DLM model were performed through deep learning-based vascular segmentation, skeleton extraction and classification, as well as drug statistical analyses (Figure  1b ). Accurate segmentation of drug-containing vessels is particularly important but challenging due to the weak contrast between drug-containing and nondrug-containing vessel pixels, fuzzy vessel boundary caused by drug diffusion, and limited sample size of the training dataset.
To tackle this problem, we designed DAU-GAN that employs the following specific strategies to enhance the segmentation performance (Figure  1c ). In the deep learning pipeline, we applied the general framework of GAN containing a generator (G) and a discriminator (D) that were trained alternatively to make the predicted output indistinguishable from the ground truth. The G adopts an encoding-decoding U-Net structure, which is especially complemented with a DA module comprising a channel attention block and a spatial attention block on each Skip Connection path to endow excellent generation capability. Importantly, GAN-based segmentation can improve model generalization with small amount of image samples, and the dual attentions emphasize the spatial and channel features to have a better feature representation for solving the low contrast of vessel images.
2 and Q-value < 0. 05". Both enriched and depleted proteins increase with the decellularization process (Figure  7f ) with 234 common DEPs detected across the three samples (Figure  7g ). Among the 580 enriched and 591 depleted DEPs for DLM model as compared to fresh liver (i. e. , S2 vs FL), 50 identified proteins are associated with extracellular structures based on eukaryotic orthologous group database (Figure  S17 , Supporting Information), showing enrichment of 96% of the ECM proteins in the DLM (Figure  S18  and Table  S7 , Supporting Information). Furthermore, we found that the collagen proteins, especially the fibrillar collagens such as Types I, II, III, VI collagens, were dramatically enriched during the decellularization processing (Figure  7h ; Table  S8 , Supporting Information).
The average diameter along the skeleton path can then be calculated for vessel branches. Extravascular Drug Diffusion Characterization and IVIVC: Extravascular drug diffusion behavior was evaluated based on fluorescent microscopy images of DLM model obtained at predetermined time points. In the DLM model, the overall drug diffusion region was derived by selecting the area from the fluorescent images with grayscale from 1 to 255 (dark: 0; white: 255) using ImageJ software (NIH, Bethesda, MD). For each drug formulation group, four vascular locations along with one main Level P 1 's branch (L1: 800-1000 μm; L2: 600-800 μm; L3: 400-600 μm; and L4: 200-400 μm) were selected to analyze extravascular drug diffusion capacity. For the DEB group, IVIVC was established between the extravascular drug diffusion behavior of L2 location of DLM model with the in-human results reported by Namur et al.
For MS2 fragmentation, the ion screening conditions were set with the charge from 2+ to 6+, and the top 20 parent ions showing a peak intensity exceeding 10 000. The fragment ions were detected in Orbitrap mass analyzer (Thermo Fisher Scientific, San Jose, CA). The automatic gain controls for MS1 and MS2 were set to 3 × 10 6 and 1 × 10 5 , respectively. Proteins detected from the tandem mass spectra were identified and quantified using IQuant software (BGI, Shenzhen, China) that integrates Mascot Percolator algorithm. The acquired data were searched against the reference proteome of Rattus Norvegicus based on the UniProt Database (https://www. uniprot. org/). Totally 1 096 186 spectrums were generated, 26 260 peptides and 5358 proteins were identified with a false discovery rate ≤ 1%. DEPs were determined by a threshold of fold change > 1.
2 and Q-value < 0. 05 in single replicate. Statistical Analyses: Data analyses were performed using GraphPad Prism 8 (GraphPad Software Inc. , San Diego, CA, USA). For evaluation of deep learning-based vessel segmentation and skeleton extraction, twosided paired t-test was performed to assess the difference of DSC and ASSD. For evaluation of ECM compositions and drug-DCM interactions, two-sided unpaired t-test was performed to compare the liver samples. pvalue of < 0. 05 was considered statistically significant, i. e. , *P < 0. 05, **P < 0. 01, and ***P < 0. 001.
Acknowledgements X. L. and X. W. contributed equally to this work. This work was supported by the  Key-Area Research and Development Program of Guangdong Province  ( 2019B010941001 , Q. G. ),  National Natural Science Foundation of China  ( 81971764 , Q. G. ; 62071210, X. T. ),  Guangdong Research Fund  ( 2019ZT08Y191 , Q. G. ), the  Innovation Committee of Shenzhen Municipality  ( JCYJ20200109141638004 , Q. G. ;  JCYJ20190809120205578 ,  JCYJ20200925153847004 ,  KCXFZ2020122117340001 , and  RCYX20210609103056042 , X. T. ),  Guangdong Provincial Key Laboratory of Advanced Biomaterials  ( 2022B1212010003 , Q. G. ),  Scientifiques de la Fondation ARC  (Grant No. PJA 20171206526 , X. W. ),  Shenzhen Key Laboratory of Smart Healthcare Engineering  ( ZDSYS20200811144003009 , Q. G. ), and the Startup Funding from the  Southern University of Science and Technology (SUSTech , Q.
Study intervention procedure Each PGT Lycra ® sleeve was measured by a well-trained physiotherapist (trained by a specialist from the manufacturer) according to manufacturer's instructions. Size segments were measured at standard points of reference for circumferential measurements (Supplementary Figure  S1 ). It was a tailor-made sleeve which covered the arm from the axilla to half of the palm and the thumb, without covering other fingers  (20). Lycra ® sleeves were manufactured for each UCP child according to a randomization list within 1 week. A randomization sequence 1:1 ratio was performed by an independent statistician using computer-generated random numbers structured in blocks generated by PROC Plan SAS software version 9. 4 (SAS Institute Inc. , Cary, NC). All participants and research staff were blinded regarding the randomization allocation during the whole duration of the trial, as were the parents, physiotherapists, and physicians involved in the child's care.
Study variables The primary objective of this study was to evaluate the safety of a PGT Lycra ® sleeve worn over 6 months on the upper arm of children with UCP. Safety outcomes included the number and intensity of Adverse Events of Special Interest (AESIs). AESIs were defined as adverse events imputable to compressive therapy and Lycra ® wearing, localized at the arm. AEIs were classified into two subgroups: (i) cutaneous events linked to Lycra ® wearing and (ii) compression events linked to compressive therapy. According to the device classification panel from the FDA, the PROPENSIX PGT Lycra ® sleeve is a class I medical device. In this context and according to French regulations, it was not mandatory to address a Medical Device Reporting process and a Data Safety Monitoring Board report to conduct this study.
Frequency of AESIs were computed as a percentage of occurred days of AESIs from the total wearing time. Systolic blood pressure (SBP) and diastolic blood pressure (DBP) were measured as a safety outcome using an oscillometric device KYPIA COLSON (COLSON ® , Paris, France) with a pediatric bladder following the recommendations of the British Hypertension Society  (31)  and recommendations for children populations (32). SBP and DBP were measured on the hemiplegic arm of UCP children without the Lycra ® sleeve pressure garment. SBP and DBP were measured in a sitting position after a 10 min rest in a quiet room after sleeve removal at least 10 min before measurement. The duration of 10 min of sleeve removal before BP measurement was considered enough time to not affect BP data  (33). Level of adherence was assessed using a conventional paperbased method  (34) (35) (36).
The diary logbook collected daily amount of Lycra ® sleeve pressure garment wearing period in number of hours and the reason of non-adherence if the sleeve was worn for under 3 h per day. Level of adherence was expressed in percentage of number of days when the sleeve was worn for at least 3 h per day compared to length of duration in days (start and end date of wearing period). Frequency of reasons of nonadherence were computed as percentage of occurred days of AESIs from the total wearing time. Anthropometrics and neurologic examination acquisition data were previously described by Gerard et al. (20). Weight was measured in underwear, with shoes removed, using an electronic scale (SECA ® 861, SECA ® , Birmingham, UK) to the nearest 0. 1 kg. Height was measured with shoes removed using a telescopic height measuring instrument (SECA ® 225) to the nearest 0.
1 cm. Functional profile of the child was classified according to Gross Motor Function Classification Scale (GMFCS): type I-V (37) and Manual Abilities Classification System (MACS) type I-V  (38). CP etiology was classified according to previous studies  (39, 40)  according to in utero or perinatal events  (41). Physical examination (standard clinical exam and neurologic exam) was performed by the same investigator throughout the study period. Data were entered in electronic Case Report Forms (eCRF). The eCRF was developed using Ennov Clinical© software. Data were checked by the data management team of the data monitoring department of the trial sponsor using the predefined rules of quality assurance.
Statistical analysis Data are presented as frequency (percentage) for categorical variables and mean ± standard deviation (SD) for continuous variables. Normality of distribution was checked graphically with the Shapiro-Wilk test. Comparisons between groups (intervention vs. placebo) were compared using Khi2 test for categorical variables and Student's t test for continuous variables. ANOVA for repeated measures was used for longitudinal data analysis. All statistical tests were done at the two-tailed α level of 0. 05. Data were analyzed with SAS software version 9. 4 (SAS Institute Inc. , Cary, NC).
Results Using the data from the first 10 children with CP Lycra ® sleeve pressure garment measurements, a second set of 10 sleeves were manufactured for compression laboratory testing. Data of compressive properties of these sleeves measured in the laboratory were obtained for four active PGT Lycra ® sleeves and six placebo PGT Lycra ® sleeves. The mean and standard deviation of compression was 17 ± 1 mmHg for active vs. 7 ± 1 mmHg for placebo. Figure  1  presents the CONSORT flow diagram of this ancillary study. Of the 58 included UCP children, 49 were analyzed (25 intervention vs. 24 placebo), which allowed a well-balanced and sufficiently powerful statistical analysis between groups. Clinical characteristics of UCP children at baseline are presented in Table  1. There was no difference in anthropometrics, etiology, and neurological contexts between groups.
Acknowledgements We thank the children and their parents for participating in PROPENSIX study. We thank  Muriel BEUVRY  and  Anne GAUTREAU  ( CIC-1403-CHU-Inserm de Lille, France ) for help in typing this manuscript. We thank  Anne-Sophie BLAIN  ( CIC-1403-CHU-Inserm de Lille, France ),  Bérangère VANTORRE  and  Sandra MILLIEN  ( Centre Marc Sautelet, Villeneuve d'Asq, France ) for your help in sleeve measurements and logistics;  Magalie VEDRENNE  for PGT Lycra ® sleeve manufacturing (Medical Z ® , F-37208, Saint-Avertin, France);  Edouard MILLOIS  and  Morgane COEFFET  ( Lille University Hospital, Lille, France ) for regulatory and study quality assurance management; and  Florence DUFLOT  for datamanagement. We thank  Lille University Hospital  which was the study sponsor of this study.