dataesr/scientific-paragraphs-categorization huggingface dataset:
We will therefore need to continue validating and comparing them using new data. In terms of computing, TF-IDF or FastText-based LGBM models are lighter and run faster than transformers, such as XLM and CamemBERT, but these might emerge as more accurate in the long term, as transformer-based deep learning models are currently considered state of the art for NLP  [26]. They also have the advantage of being multilingual, potentially allowing for the transposability of our models in other languages than French. Overall, while still perfectible, our AI models showed promising performance to support the automatic coding of ADR reports, though a more thorough interpretation of our results might be needed. In our study, reported precision (= positive predictive value), sensitivity ( = recall) and specificity were estimated using a predictive threshold aimed to maximise the F-measure.
In this regard, ROC and PR curves are more informative, as they allow a better understanding of the possible trade-offs between measures. For a use-case  [1] , our seriousness assessment models allow for an 80% recall/50% precision trade-off, as shown in Fig. 4B  and D. In our dataset, this translates to the manual assessment of roughly 32% of all cases to correctly capture four fifths of the 20% cases considered serious. For a use-case  [2] , experts may choose the 90% precision/70% recall trade-off of our models identifying ADRs, as shown in Fig. 5B  and D. This could allow for a nearly automatic "fill-free" validation of 70% of the pre-coded PTs in all reports. As these estimates are only based on the reading of our internal and external validation ROC and PR curves, these will also need to be confirmed in further real-life studies.
RegEx only achieved an AUC of 0. 69 and an F-Measure of 0. 50. Our study has several strengths compared with previous work on the subject. Our selection of data sources covered nearly a quarter of all ADR reports transmitted by patients to ANSM and CRPV between March 2017 and December 2020, providing a wide sample of reports over this period and across multiple teams, with a variety of suspected drugs and coded PTs. We used real-life expert-annotated data, in contrast with studies using case report abstracts or simulated data. We compared state-of-the art models for our classification tasks, and created in-house AI pipelines dedicated to processing Portable Document Format reports. In addition, we externally validated our models on a distinct CRPV to assess their generalisability in a different setting, which yielded consistent and robust results.
Our study also has some limitations. First, regarding selection, most reports focused on levothyroxine and Mirena ® , two drugs involved in sanitary crises widely reported by the media in 2017-18. Most ADRs described in our datasets are therefore those associated with levothyroxine and levonorgestrel. Although these cover a wide variety of PTs, susceptible to be associated with other drugs, this could bias the predictions of our model identifying ADRs in other situations. Second, models identifying ADRs in reports were not trained on PTs coded in fewer than ten reports in the development dataset. They are therefore not appropriate for the identification of all possible PTs. Nevertheless, our ambition was not to be exhaustive of all potential PTs but to identify those representing the most frequent PTs, which usually the issue with over-reporting.
Conclusions In this study, we successfully trained and validated the performances of AI models identifying ADRs and assessing seriousness using unstructured text data from nationwide patient reports. Gradient boosting and transformer-based approaches yielded close results for both internal and external validation, probably in relation to the respective strengths and weaknesses of the distinct methods used for data formatting and classification. Our system was considered accurate enough by ANSM for national deployment in France in January 2021, aiming to help PV experts cope with the COVID-19 vaccination campaign, during which authorities expected a massive reporting of ADRs. Further studies will be needed to validate the performance of our system in real-life settings and to continue identifying the best possible model, adding more evidence to the possible use of AI in the automatic pre-coding of PV reports.
Acknowledgements We gratefully thank the  Ministère de l'Enseignement Supérieur de la Recherche et de l'Innovation (MESRI)  for the PhD funding of  Aline Delamare  and the  Univ. Bordeaux  for the temporary teaching and research assistant position of Guillaume Naulet. We thank  ANR  for its nancial support (Project  ANR-20-CE06-0008 ). Pierre Waffo Teguo , from the  Institut des Sciences de la Vigne et du Vin , is gratefully acknowledged for his help with specic rotation measurements. We thank  Estelle Morvan  from the  IECB Biophysical and Structural Chemistry Platform (BPCS) ,  Univ. Bordeaux ,  CNRS UAR3033 , Inserm  US001 , for her assistance with the NMR experiments.
A UV-VIS calibration curve of CUR (1-7 µg/mL) in EtOH/DCM (60/40%, v/v) was obtained at λ = 427 nm, presenting a slope of y = (0. 143 ± 0. 002)x; R 2 = 0. 999. Method was tested in the range of 0. 5-0. 7AU. Chromatographic analyses were performed using an Azura HPLC (Knauer, Berlin, Germany) equipped with an Azura Pump P6. 1L, a KnauerEurospher II C18 column (250 × 4 mm) and a UV-VIS Azura detector DAD 2. 1L, controlled by ClarityChrom ® software version 6. 1. 0. The detection wavelength was fixed at λ = 425 nm. The mobile phase consisted in a 13 min-gradient from 50:50 to 0:100 (water (+0. 1% TFA)/ACN (+0. 1% TFA)) at a flow rate of 1 mL. min -1. The injection volume was 20 µL. All samples were diluted in acetonitrile (ACN) and filtrated (PTFE, 0. 45 µm). An HPLC calibration curve for CUR (0.
After incubation, photos were taken using a DMi1 inverted microscope (Leica Microsystems, Wetzlar, Germany). Obtained RGB images were converted to grayscale images using Fiji software (ImageJ, NIH, USA). MTT was then removed and 200 µL of isopropanol was added to each well. Plates were gently stirred and stored at room temperature in the dark for 40 min. The percentage of viable cells was calculated as in no stress conditions through absorbance readings at 570 nm as described in Section 2. 8. 1 with a microplate reader (Xenius XM, Safas, Monaco).
Phase-contrast images of the gap were taken immediately after adding the treatment using a DMi1 inverted microscope (Leica Microsystems, Wetzlar, Germany). Cells were incubated for 24 h and photos were taken again in the same place. For each photo, the areas were measured using the manual area measurement tool in the Fiji software (ImageJ, NIH, USA). For each sample, 5 images corresponding to different places along the gap were analyzed. Experiments were repeated twice. Pixel size at the specimen level was 0. 235 µm. For controls, cell medium (10% FBS DMEM or DBM, for BJ fibroblasts and HEKn, respectively), was used without CUR, Blank-NLCs or CUR-NLCs. Reduction in the gap area was determined by comparing the areas in the photos at 0 h with the ones at 24 h with the following equation: % Reduction of gap area = 100 -[(Area at 24 h / Area at 0 h) × 100] 2.
Statistical Analysis The software OriginPro 2018 (Northampton, MA, USA, version 9. 5) was used. All results are presented as mean ± standard deviation (SD). For the MTT test, the analysis of statistical significance was performed using one-way analysis of variance (ANOVA) followed by Tukey's multiple comparison HSD (honestly significant difference) post hoc test. For penetration studies, analysis of the statistical significance was performed using two sample Student's t-test.
Acknowledgments:  Rosa Calderon-Jacinto  thanks Doctoral School n • 417, Sciences and Engineering for the PhD grant and traveling funding. The authors are grateful to Gattefossé for the kind donation of Precirol ® ATO 5 and Labrafac ® lipophile WL 1349 and to Lubrizol for the kind donation of Carbopol ® 980. Authors thank  Ana-Andreea Arteni  for TEM images from the platform of Cryo-EM of I2BC (supported by  IBiSA  and by  French Infrastructure for Integrated Structural Biology (FRISBI)  ( ANR-10-INBS-05 ). Authors are grateful to LPPI ( EA2528 ) for the use of dynamic light scattering and  Zetasizer Nano ZS equipment. Sapienza  grant  RM12117A81AEF242  is acknowledged by  Pietro Matricardi.
However, we could compare the amount of Amy53 detected in the perfused brain and compared the values obtained from the nonperfused brain at 30 min (Figure  5C ). At this time, similar AmyP53 amounts were measured in both conditions, showing that the therapeutic peptide is indeed able to reach brain tissue after intranasal administration. Finally, we studied the pharmacokinetics of AmyP53 in blood (Figure  5D ). In this case, the peptide peaked immediately after intravenous injection, whereas there was a lag following intranasal administration. In both cases, the amount of AmyP53 recovered in blood was lower than the amount detected in brain. A major outcome of these pharmacokinetics studies is that AmyP53 is able to reach the brain after either an intranasal or an intravenous treatment. In the latter case, the data strongly suggest that the peptide is able to cross the blood-brain barrier.
Kinetics of AmyP53 Transport through an In Vitro Model of the Blood-Brain Barrier The transport of intact AmyP53 through the blood-brain barrier was studied with three reconstituted in vitro cellular models cultured in a two-compartment chamber  [76]  (Figure  6A ). These three models consist of (i) pure bEnd. 3 cells (mouse endothelial brain cells), (ii) bEnd. 3/CTX (rat brain astrocytes) co-cultures, and (iii) bEnd. 3/C6 (rat brain glial cells) co-cultures  [77]. The impermeability of each of these cellular barriers was assessed by transendothelial resistance measurements. Experimental data have shown that an electrical resistance > 100 Ω. cm 2 is sufficient to prevent the paracellular passage of ions  [78] , radioactive iodine  [79] , and sodium fluorescein (a compound with a Mw of 376 Da, i.
blood-brain barrier (bEnd. 3/C6 model). Incubated in competition with the β-amyloid 1-42 peptide, which forms amyloid pores and induces an influx of calcium ions into the SH-SY5Y cells, the transported AmyP53 peptide remained fully active: it blocked the formation of amyloid pores (Figure  7 ). In this experiment, calcium fluxes were measured with the Fluo-4AM probe, a fluorescent indicator sensitive to calcium concentration. Fluorescence microscopy analysis then makes it possible to follow in real time the evolution of the calcium concentrations of individual cells, to determine the calcium concentration of each of these cells and to sum the values resulting from the analysis of hundreds of cells (150 for the histograms shown in Figure  7 ). Overall, these data showed that intact and biologically active AmyP53 can be transported through the blood-brain barrier, consistent with pharmacokinetics data obtained in rodents.
Finally, according to the therapeutic peptide database THPdb  [123] , whi FDA-approved therapeutic peptides and proteins, there are currently 11% o  Finally, according to the therapeutic peptide database THPdb  [123] , which lists all FDA-approved therapeutic peptides and proteins, there are currently 11% of peptides with 1-30 amino acids and 2% for neurological diseases. Interestingly, a large ratio of therapeutic peptides (46%) are directed against receptors, but none against gangliosides, which remains a distinctive innovation of AmyP53.
AmyP53 LC-MS Measurements The concentrations of intact AmyP53 in biological samples were determined by liquid chromatography-mass spectrometry (Acquity iClass-Vion LC-MS; Waters, Guyancourt, France). A total of 1-10 µL of each extract was injected into a C18 reverse phase column (BEH 2. 1 × 50 mm, 1. 7 µm; elution gradient H2O/CH3CN 0. 1% formic acid). The injection needle was rinsed with 20% isopropanol (to avoid memory effects). Positive-mode electrospray ionization was performed at 2. 5 kV/40 V, and AmyP53 [M+3H]3+ major ions were selectively monitored by a Tof-MRM method using 3 fragmentation transitions (456. 58 → 527. 32; 456. 58 → 579. 30; 456. 58 → 841. 43). CID fragmentations were operated at 20 eV for the three transitions. Data processing was performed with the UNIFI software (Waters, Milford, MA, USA).
Acknowledgments: We thank  Sandra Robelet  and  Caroline Pin-Barre  from  Syncrosome  for animal experimental procedures and  Maya Belghazi  and  Claude Villard  from  PINT  for chemical stability analysis. We also thank  Kevin Sadelli  from  AmyPore  for bibliographic studies and  Romain Guegan  from  Dominique Dutscher  for his support and expert technical advice.
Despite this commitment from 2013, an audit found that data availability was reached for only 9/61 (15%) clinical trials on medicines sponsored by the pharmaceutical industry and first published between 1 July 2015 and 31 December 2015 in the top 10 journals of general and internal medicine  [10]. If such low rates of data-sharing were also to be observed for main trials, it would invalidate any efforts towards reproducibility for these important studies. However, the environment for data-sharing is changing fast. And indeed, data-sharing platforms like ViVli, YODA project, or Clinical Study Data Request are more widely used. In the fall of 2019, these platforms gathered a large number of trials sponsored by the pharmaceutical industry. These three platforms included about 8000 RCTs in November 2019  [11].
Eligible main trials Two reviewers (MS, JG) independently extracted all names of the new medicines, biosimilars and orphan medicines approved by the CHMP and entered the information on a standard data extraction form. Afterwards, a check was performed to verify that the CHMP opinion was adopted by the European Commission  [18]. Next, the reviewers identified the corresponding eligible EPARs on the EMA website  [19]  and independently extracted all main studies reported in these EPARs. Disagreements were resolved by discussion between the two reviewers or after referral to a third reviewer (CL or FN) until a consensus was reached.
Main study document accessibility For all randomly sampled studies, one reviewer (JG) searched for the EudraCT number and/or the Sponsor Protocol Number, and/or any other identification information in each EPAR and identified the official sponsor of the study. If this information was lacking, the same reviewer started a wildcard search using keywords (disease, drug) from the study in the European Union Clinical Trial Register  [21]. If this was not successful, the reviewer went on the websites Clini calTr ials. gov  [22] , International Clinical Trials Registry Portal (ICTRP), World Health Organization  [23]  and the International Standard Randomised Controlled Trial Number (ISRCTN) allocated by BioMedCentral  [24]. If information on sponsor and study number was still lacking, the reviewer contacted the EMA.
Once the sponsor and the study number were identified, the reviewer contacted the sponsor to collect all of the following main study documents: (i) IPD; (ii) data analysis plan; (iii) unpublished and/or published study protocols with any date-stamped amendments; (iv) all the following dates: date of the last visit of the last patient, date of database lock (if available) and date of study unblinding; and (v) unpublished and/or published (scientific article) study reports. To this end, the reviewer sent a standardised email (Additional file 2: Letter 1), presenting the research project with a link to the registered protocol on the Open Science Framework  [15]. In order to improve the return rate, up to 4 emails were sent, the original and 3 reminder emails (with a two-week interval between them).
Data extraction The identification of main studies and the following trial characteristics were extracted from the EPARs on a standard data extraction form by two independent researchers (JG and FN). For each study, the following information was collected: patient characteristics (e. g. percentage of women, mean age of participants, paediatric indication), study methods (e. g. type of endpoint, description for each primary endpoint) and intervention characteristics (e. g. drug). An exhaustive list of the trial characteristics extracted can be found in the web appendix (Additional file 4: Table  S2 ). Concerning the re-analysis, a first reviewer (JG) collected the information and collated data for the re-analysis. More specifically, the reviewer prepared a dossier with the following information for each study: (i) the protocol; (ii) all amendments to the protocol (with their dates); (iii) all the following dates: date of the last visit of the last patient, date of database lock (if available) and date of study unblinding; and (iv) the IPD.
In addition, this reviewer was instructed not to try to find these documents or the published report. For single-blind studies or open-label studies, analyses were performed according to the first version of the protocol, because outcome switching has been documented. For double-blind studies, all re-analyses were based on the latest version of the protocol issued before database lock and unblinding. If this information was not available, the date of the last visit of the last patient was used as a proxy. Although in therapeutic research, statistical analysis can be "routine", in some cases the re-analyses involve difficult methodological choices. An independent senior statistician (AR) was available to discuss any difficult aspect or choice in the analysis plan before the re-analysis, so as to choose the most consensual analyses (e.
Data analysis We performed a descriptive analysis of the characteristics of the main studies extracted included in the EPARs selected. This included counts, percentages and their associated 95% confidence intervals (CIs). Effect estimates in the different studies were expressed as standardised mean differences (SMDs) and their associated 95% CIs. For binary outcomes, odds ratios and their 95% CIs were calculated and converted into the standardised mean difference  [34]. To compare the results of our re-analyses with the original results, the following steps were implemented: (i) we compared the statistical significance in the form of the p-value. If different, the results were considered as not reproducible. If not different, (ii) we qualitatively compared effect sizes and their respective 95% CIs.
Changes to the registered protocol We set a 1-year deadline to obtain data. However, data demands were lengthy, and delays were in some cases produced from our side. Hence, study data that was sent after this date was included in the re-analysis process. Furthermore, although we said we would only use R software for data analysis, SAS software was used for two studies because of its more potent approach in mixed model analyses. For one study we were unable to calculate the odds ratio. Starting with the incidence rate ratio, we used Chinn conversion to receive obtain the SMD  [35]. This approach is justified in cases where events are rare and the incidence rate ratio can be treated as an odds ratio. Because of low data-sharing rates, one researcher checked (JG) whether data-sharing policies were posted on the companies' websites.
Study selection The searches and consensus finished on 27 February 2020 and yielded 317 main studies identified in 173 EPARs. Of these, 25 were excluded (duplicates and studies with no primary endpoint) resulting in 292 individual studies. Of these, 62 were randomly selected (Fig. 1 ) and the respective data was requested from forty sponsors. All sponsors were contacted, and data was requested, either by mail or directly through a data-sharing platform. After exchanges with staff, for six datasets on Vivli, and for three on YODA, requests were issued.
Data availability Among the 62 studies, we received IPD for 10 trials (16% [CI95 8 to 28%]) from six sponsors  [36] [37] [38] [39] [40] [41] [42] [43] [44] [45]. For these studies, the median number of days before data became available was 253 [interquartile range (IQR) 182-469]. For these studies, all but one of the sponsors were big pharmaceutical companies and all but one of these companies had a data-sharing policy on their website. IPD for four studies was provided via data-sharing platforms (one was provided by one sponsor on Vivli and three by a single sponsor on YODA). Three studies were shared via a remote desktop monitored by the company in possession of the data. Another three data sets from three different sponsors were sent directly to us. All IPD received was analysable and deidentified.
List of challenges Time required for data retrieval Requesting and receiving the data was time-consuming. Interactions with sponsors were, on some occasions, lengthy, especially if several were involved on the same data-sharing platform. For example, on Vivli, we submitted a data request concerning six studies from three sponsors. The sponsors raised various questions. In one study (NCT00927498), the ownership of the data created confusion. The trial data was purchased by Pfizer. However, Vivli informed us that Pfizer was not in possession of the data and referred us back to the original Principal Investigator of the study who no longer had any rights over the data. After clarification via the platform, access was denied by Pfizer for reasons of insufficient scientific merit of our approach.
As part of the data acquisition process, the legal department of our unit had to confirm the data agreement and this step on our side was also lengthy. For two datasets, we exceeded our 1-year limit for data retrieval by 2 weeks. For three additional studies that were requested on YODA, 134 additional days were calculated. In this case, the data was not available at the time of our initial request (18/05/2020) but YODA contacted us on (04/03/2021) to indicate that the data was now on the platform and could be requested. After a request, we received this data on (14/10/2021). These studies were included in our analysis since these long time-lapses were considered as minor deviations from our initial protocol.
Incomplete datasets, metadata and further clarifications Data dictionaries were available for 7/10 studies. In five studies, we had to contact the sponsor/platform   to request additional data, as the data necessary to re-analyse the outcomes was missing (in three esketamine trials and in two semaglutide trials). We received additional data after 28 days in the case of the esketamine trials and were able to re-analyse the primary outcome. In the two three-arm studies from the same sponsor, comparing semaglutide with sitagliptin and placebo-non-inferiority on a primary outcome (change in HbA1c) and superiority on a "key secondary outcome" (bodyweight)-data concerning bodyweight was not available and was deleted as part of the anonymisation process. This study used a hierarchical testing Table  2  Identification of spin one of the selected studies One study that examined the use of esketamine in treatment-resistant depression used a hierarchical testing approach: if the higher of two doses was not positive, according to the protocol the lower dose should not have been tested and reported.
Data analysis In 2/10 cases, while we reproduced the conclusions of the studies, we did not define the same analysis population with respectively 303 and 434 vs 297 and 433 patients analysed in the studies by Janssen SUSTAIN-1 and Merck Clarity. For the latter study, discrepancy for one patient was clarified with the sponsor. In this study, one patient was counted twice due to re-screening, and to the deanonymisation of the data which made identification impossible. This information was included in the analysis data reviewer's guide to which the authors had no access. Among those two studies, the esketamine study published by Janssen had a very complex design involving randomised and non-randomised patients. The absence of a clear randomisation list and of a data dictionary made the re-analysis very challenging.