dataesr/scientific-paragraphs-categorization huggingface dataset:
Transformer models can be trained on large datasets to learn language representations. Pre-trained models can then be used for zero-shot classification or they can be fine-tuned for a specific classification task. In zero-shot classification, pre-trained models are used without any specific additional training examples for new classes  (30). Their performance rely on their ability to generalize classification properties based on the language representation they captured during pre-training. On the opposite, the finetuning step consists of training further a pre-trained model on a new dataset and for new classification classes to improve the model's performance on this specific task  (31). Here, we explored the potential of different models of classification to automatically recognize comment texts describing weight gain, weight loss, or the absence of reference to a weight change.
In particular, we focused on BERT and BERT-related transformers, that have achieved state-of-theart results on a wide range of NLP tasks  (32). The novelty of BERT was to encode the context of a word from both the left and the right (bidirectional). BERT was designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. We also used RoBERTa -a robust re-implementation of BERT with some modifications to the key hyperparameters and minor embedding tweaks. It has been shown to outperform BERT on a variety of NLP tasks  (33). Roberta uses a byte-pair encoding tokenizer instead of a word-piece tokenizer to better handle rare words and out-of-vocabulary words. It trains with a larger mini-batch size and for a longer number of steps to better learn long-range dependencies in text and it removes the next sentence prediction (NSP) objective, found to be less effective than the masked language modeling (MLM) objective for pre-training BERT models.
If BERT and RoBERTa are highly efficient, it can be difficult to run these large models on edge devices or to train or use them with limited computational ressources. DistilBERT is a distilled version of BERT that is smaller and faster  (34). It is trained using a knowledge distillation procedure, which involves training a smaller model to mimic the predictions of a larger model and it has been shown to achieve comparable performance to BERT and RoBERTa. Distil-BERT is a good choice for applications where speed, efficiency, and ressources are important. Similarly, we explored SqueezeBERT, a lightweight and efficient transformer-based language model that is specifically designed for mobile and embedded devices  (35). SqueezeBERT is based on the BERT architecture, but it has a number of modifications that make it smaller and faster including using grouped convolutions instead of fully-connected layers to reduce the number of parameters in the model.
BERT and these BERT-derived models are pre-trained on a massive dataset of text and code. They can be used pre-trained and only fine-tuned on a specific downstream task including classification without having to be trained from scratch  (36). We also explored the performance of DistilBERT and DeBERTa in the context of zero-shot classification. DeBERTa uses disentangled attention for learning richer contextual representations of words and an enhanced mask decoder for more accurate predictions of masked words, providing this model state-of-the-art performance on a variety of natural language processing tasks  (37). Taken together, we aimed to explore the potential of large language models (LLMs) in detecting signals of changes in body weight as potential side effects of antidepressants in user-generated online content.
Our investigation included fine-tuning and zero-shot classification using various BERTbased models. Our main motivation was to explore the ability of these models in identifying signals within large datasets of user reviews. Additionally, we aimed to investigate their capacity to distinguish weight gain from weight loss -two closely related elements sharing similar syntax that require a more nuanced analysis and interpretation of the language used in the comments for accurate classification. This represents an additional step beyond signal detection, moving towards assessing the causality of a drug in the occurrence of adverse effects. Most of these different models performed particularly well in this task.
Data collection Data collection took place between August 2022 and November 2022. The data corresponded to publicly available usergenerated online content from four specialized websites at the time of data collection  (38) : Drugs. com, WebMD, Everyday Health, and Ask a Patient. Drugs. com, WebMD and Everyday Health are websites that provide information about medications and health to the general public. Ask a Patient is a platform where patients can share their first-hand experiences with their prescribed drugs. Data from Drugs. com, WebMD and Everyday Health were scraped using R version 4. 2. 2 and the R packages rvest  (39) , httr  (40) , and xml2  (41)  while data from Ask a Patient (and the psyTAR) database were kindly provided by Askapatient. com. A random sample of 8,000 texts, stratified by site and drugs, was extracted for manual annotation.
Data split The data set was split into training (n = 6,000), validation (n = 1,000), and test (n = 1,000) datasets. The training and validation data sets were used to adjust the LLMs during the fine-tuning step. The test set was used to evaluate the performance of the model. It is important to note that the training, the validation and the test sets were defined identically for all models in order to facilitate comparisons.
Large Language Models The pre-trained models were loaded by instanciating a given LLM configuration model using the generic sequence classification model class available in the Hugging Face library https://huggingface. co/ (42) for Python 3. 8. The different models for zero shot classification were : • distilbert-base-uncased (6-layer, 768-hidden, 12heads, 66 M parameters) • deberta-v3-base (12 layer, 768-hidden,12-heads, 86 M parameters) • deberta-v3-large (24 layer, 1024-hidden,16-heads, 304 M parameters) and for fine-tuning : • bert-base-uncase (12-layer, 768-hidden, 12-heads, 110 M parameters) • roberta-base (12-layer, 768-hidden, 12-heads, 125 M parameters) • distilbert-base-uncased (6-layer, 768-hidden, 12heads, 66 M parameters) • squeezebert-uncased (12-layer, 768-hidden, 12-heads, 51 M parameters) The input texts corresponding to user reviews were lowercased and tokenized using the appropriate pre-trained tokenizer using the AutoTokenizer class from the Transformers model library, without any additional preprocessing steps.
Prediction performance The models were evaluated using accuracy, weighted pre- ). Of these comments, 8,000 were randomly selected for manual annotation, stratified by website and antidepressant molecules. The annotation process was designed to allow for some overlap between text inputs so that inter-rater reliability could be estimated. Out of the 8,000 text inputs, 3,500 were evaluated twice, resulting in 96 label discrepancies (2. 74 %) reconciled by an expert reviewer. This gave a Krippendorff's alpha of 0. 841. The distribution of labels in the dataset are presented as a function of drugs in Table  S1. Following the data split, the training and validation sets comprised 6,000 and 1,000 labeled inputs, respectively, while 1,000 inputs were kept for testing. As expected, labels associated with gaining or losing weight were in the minority in the training and testing setswith almost 90% of the labels corresponding to an absence of mention of change in body weight in the input text.
This led to an accuracy value of 0. 897 in the validation dataset for a dummy classifier predicting the most represented class (see Table  1 ). Models performance BERT, RoBERTa, DistilBERT, and SqueezeBERT all performed well after fine tuning, with BERT and RoBERTa performing the best with an F1-score of 0. 976. Their macro F1score were 0. 865 and 0. 891, respectively. Precision and recall of each class are presented for all models in Table  S2. Dis-tilBERT performed poorly in zero-shot classification, while DeBERTa's performance seemed to scale with the number of parameters, reaching performance close to that of the smaller fine-tuned models for the large DeBERTa model with 304M parameters. Confusion matrices -calculated for the same test set for all the models -are available as Supplementary Materials in Figures  S1  and S2.
Discussion In this work, we explored the potential of large language models (LLMs) for detecting signals of body weight change as a side effects of antidepressants. An important key point of this work is that we have built a valuable human annotated database, controlled by experts, with a substantial size of 8,000 input texts to evaluate model performance. We have made our dataset publicly available to promote open science and enable other researchers to build upon our labeling efforts (doi to be released). Collecting data on health-focused sites was relevant because it is estimated that only 10% of medical content on general social networks includes information on AEs, compared to 20-25% on healthfocused platforms all AEs combined  (19). Focusing on a single AE, we observed a proportion of 10% of the reviews made on antidepressant drugs were mentioning weight-related AEs -confirming that this AE might be a concern for patients taking these treatments.
Background Evaluating academics is a challenge, and the use of indicators such as scientific impact (i. e. number of published papers and their citation rate) is complex and poorly validated. We propose a new indicator for academic medical research: the "Free lunches" index (fl-index), computed from the sum of gifts from the industry. The fl-index provides a direct and straightforward measure of industry investment consisting in regaling a clinical researcher with rewards like a leisurely meal in a Michelin-starred restaurant or a relaxing stay in a high-end resort hotel. Methods and findings 3,936 French academics were included in this observational and satirical retrospective study using the French database registering gifts received by medical doctors and Web of Science, over the years 2014-2019.
Study design and participants In France, academic duties require both active clinical activity and research activity. While clinical skills are evaluated qualitatively, the assessment of research activity relies on publication metrics. Two researchers (PES and TC) identified French professors of medicine and associate professors according to their academic discipline from university lists and the "Annuaire Santé" of the Department of Health which details medical specialties. Academic disciplines were categorized according to the national organization (Conseil National des Universités) in charge of hiring, promotion and tenure of academics in France. From an existing database of nearly all French academics' administrative data matched with Web of Science publication data  (Carayol & Carpentier, 2021; Carayol & Lanoë, 2017) , we extracted: gender, the year of the first paper (an indicator of seniority) and quantitative indicators of productivity for the 2014-2019 period (h-index increase over the period-defined as the difference between the h-index at the start and at the end of the period), the number of publications and the number of citations for these publications.
The Department of Health's open access database (https:// www. trans paren ce. sante. gouv. fr/), established according to the French Sunshine Act documents three types of financial payments by the industry in 3 respective tables (advantages, special agreements (i. e. conventions) and remunerations). For each academic, we retrieved the amount of money declared by the industry during the 2014-2019 period exclusively for direct categories of gifts (e. g. meals or accommodation, and travel) to healthcare professionals, i. e. the "advantage" table. We did not use the tables reporting "conventions" and "remunerations", because these are not complete regarding payments, as the industry often refers to the contractual obligation to ensure business confidentiality. We retrieved the financial data concerning gifts on May 18, 2020 from EurosForDocs (https:// www.
Reporting and ethics This study is reported according to the Standards for Reporting Diagnostic Accuracy  (Bossuyt et al. , 2015). The project was based on a protocol that has been approved by the CNGE [Collège National des Généralistes Enseignants] Ethics Committee (Approval Number 110321260) on 03/26/2021. The project used publicly available data, and procedures for the protection of persons complied with the data protection regulations (GDPR). It was registered on 03/27/2021 on the Open Science Framework prior to any data collection (osf. io/7d4bk) and the protocol was publicly available online. The protocol insufficiently defined the primary outcome of the h-index over the study period and was amended, before the automatic extraction from Web of Science, to become the increase in the h-index over the period.
Participants We identified 4,320 academics from 28 (out of 37) of the French Schools of Medicine that publicly list academics. Among them, 354 were not registered as healthcare professionals and could not be found in the Sunshine act database. An additional 30 were neither professors nor associate professors. 3,936 academics were included in the main analysis. Of these, 513 had no productive metrics from an automatic extraction from Web of Science. As this could have been due to either a mismatch in name or to hiring on the grounds of clinical but not scientific background, these academics were retained in the main analysis, and excluded from a sensitivity analysis performed on the 3,423 remaining academics. Web Appendix 1 summarizes the steps leading to the selection of the academics. The total value of gifts to these 3,936 academics reached a total of 77.
Strengths and weaknesses of the study The fl-index underestimates academics' market value, as the French databases for "gifts received" excludes contracts such as scientific presentations, "training", advice, consultancies, and various collaborations in clinical research or seeding  (Braillon, 2014). However, it could also overestimate certain values of fl-as the data is derived from the declarations of the industry itself with possible typos. Some mismatches and selection biases are possible, as we were not able to identify academics from 9 universities. We did not control for geographical differences. However, we included towns like Lyon which is well-known for its gastronomy and Bordeaux which is well known for its wine. Wine stimulates fruitful collaborations with academics and the industry knows this well, for instance a California winery, Ridge Vineyards, is a subsidiary 1 3 company of Otsuka Pharmaceutical, a company that markets an alcohol-weaning pill for alcohol dependence in Japan  (Miyata et al.
Data availability We are not allowed to share data because of a risk of reidentification, in the context of GPRD. Any interested parties have the right of access to any data concerning them (upon request). R code is available on Open Science Framework at https:// osf. io/ 7d4bk/. Complete data can be extracted from the transparency santé (https:// www. trans paren ce. sante. gouv. fr/) and Web of Science. The lead author and the manuscript's guarantor affirm that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
INTERPRETATION OF RESULTS The primary objective of the FI cohort was met, with a statistically significant improvement in CCIS at 3-months post-implant. Furthermore, the mean change in CCIS can be considered clinically meaningful as it exceeded the previously estimated minimally important improvement which ranges from 2 to 3 [2]. The change in symptoms is consistent with previous reports and rate of adverse events is similar compared to previously reported rates for SNM thereby confirming the safety and clinical performance of InterStim Micro for subjects with FI. These data add to the established SNM evidence with data from the rechargeable InterStim Micro. Follow-up of enrolled patients is ongoing.
Funding Medtronic Clinical Trial Yes Registration Number NCT04506866 RCT No Subjects Human Ethics Committee The study was approved by appropriate institutional review boards and ethics committees and subjects provided informed consent. This is a multi-center study with many different local IRBs/ethics committees from different institutions; as such, each individual committee is not named here. Helsinki Yes Informed Consent Yes Continence 7S1 (2023) 100893 doi: 10. 1016/j. cont. 2023. 100893.
Chemistry: General Experimental Procedures Specific rotation was measured using a 10 cm cell in a Jasco P-2000 polarimeter in DCM. NMR data were collected on a 300 MHz spectrometer. All spectra were recorded in CDCl3 as solvent. Reactions were monitored by TLC using Merck silica gel 60 F254 (0. 25 mm-thick) plates. Compounds on TLC plates were visualized under UV light at 254 nm and by immersion in a 10% sulfuric acid solution and heating with a heat gun. Purifications were performed by flash chromatography on Merck silica gel (230-400 mesh). Commercial reagent grade solvents and chemicals were used as purchased. Combined organic extracts were washed with brine, dried over anhydrous MgSO4, filtered, and concentrated under reduced pressure. (+)-Dehydroabietylamine (ca. 60%) was purchased from Aldrich (Saint Louis, MO, USA) and (-)-abietic acid >70% from TCI Europe (Zwijndrecht, Belgium).
Chemistry: General Experimental Procedures Specific rotation was measured using a 10 cm cell in a Jasco P-2000 polarimeter in DCM. NMR data were collected on a 300 MHz spectrometer. All spectra were recorded in CDCl 3 as solvent. Reactions were monitored by TLC using Merck silica gel 60 F254 (0. 25 mm-thick) plates. Compounds on TLC plates were visualized under UV light at 254 nm and by immersion in a 10% sulfuric acid solution and heating with a heat gun. Purifications were performed by flash chromatography on Merck silica gel (230-400 mesh). Commercial reagent grade solvents and chemicals were used as purchased. Combined organic extracts were washed with brine, dried over anhydrous MgSO4, filtered, and concentrated under reduced pressure. (+)-Dehydroabietylamine (ca. 60%) was purchased from Aldrich (Saint Louis, MO, USA) and (-)-abietic acid >70% from TCI Europe (Zwijndrecht, Belgium).
Calculation of Molecular Properties (Drug-Likeness) The structures of tested compounds were manually drawn in ChemDraw Professional 15. 0 software (PerkinElmer Informatics, Inc. , Waltham, MA, USA), and the SMILES notation was obtained for each molecule. To calculate the molecular polar surface area  [28]  and the parameters of Lipinski's rule of five (see Table  1 )  [29, 30] , the SMILES notation in Molinspiration online 2023 software (available from https://www. molinspiration. com/cgibin/properties, accessed on 27 January 2023, Molinspiration Cheminformatics, Slovensky Grob, Slovak Republic, 2023, https://www. molinspiration. com) was employed.
Toxicity Prediction (GUSAR) Compounds 1, 3, and 12-16 were investigated for acute rat toxicity properties in silico (see Table  2 ). The acute toxicity in rodent models of the tested compounds were predicted by GUSAR  [31]. The analysis was based on the quantitative neighborhoods of atom descriptors and the prediction of activity spectra for substances algorithm. It correlates the results with the SYMYX MDL toxicity database and further classifies the tested compounds according to the Organization for Economic Co-operation and Development (OECD) manual. Toxic doses are expressed as median lethal dose, representing the dose of the compound that is lethal to 50% of the experimental animals exposed to it. The LD 50 values are expressed as the weight of the chemical per unit of body weight (mg/kg).
0 µg/mL or 6. 7 µM), where the cytopathogenic effect was totally abolished at all the applied MOIs (0. 01-0. 001). Compounds 3 and 15 also showed important reductions in the viral titer, similar to the one induced by 16, but they also induced modification of the cell morphology in photonic microscope observation. The rest of the compounds also produced important reductions in the viral CPE, but mostly at lower MOIs (0. 001), except jiadifenoic acid C (7) which showed no activity (Figure  4 ). In silico computation is, nowadays, an important part of the drug design development process. Using the Molinspiration Cheminformatics software (Molinspiration, accessed on 27 January 2023, Slovensky Grob, Slovak Republic, 2023, http://www. molinspiration. com), through an easily obtained SMILES notation for each molecule, we can calculate certain very useful molecular properties such as the Lipinski's rule of five  [29].
The compounds 1, 3, 12, 13, 14, 15 and 16 were subjected to this software and we obtained very promising results of drug-likeness (Table  1 ). The computational prediction of compound toxicities in drug design allows the quick estimation of the lethal doses in animals, and avoids the use of animal experiments. The rat acute toxicity of the compounds 1, 3, 12, 13, 14, 15 and 16 was predicted by the in silico software tool GUSAR (General Unrestricted Structure-Activity Relationships, version 16. 0)  [31]  on the basis of PASS (Prediction of Activity Spectra for Substances) technology  [33, 34]. The quantitative in silico prediction was applied to four different types of administration: oral, intravenous (IV), intraperitoneal (IP), and subcutaneous (SC). As is shown in Table  2 , the differences in the predicted LD 50 values obtained for the four different routes, based on the structural formulae of the tested compounds, are reported along with the acute rodent toxicity classification of the compounds.
Extreme/super toxicities (class 1) are documented for compound 1 for all routes of administration, and for compound 3 in the case of intravenous administration, where the value of 0. 00 indicates that the compound falls out of the domain of applicability of the proposed route of administration. In silico computation is, nowadays, an important part of the drug design development process. Using the Molinspiration Cheminformatics software (Molinspiration, accessed on 27 January 2023, Slovensky Grob, Slovak Republic, 2023, http://www. molinspiration. com), through an easily obtained SMILES notation for each molecule, we can calculate certain very useful molecular properties such as the Lipinski's rule of five  [29]. The compounds 1, 3, 12, 13, 14, 15 and 16 were subjected to this software and we obtained very promising results of drug-likeness (Table  1 ).
The computational prediction of compound toxicities in drug design allows the quick estimation of the lethal doses in animals, and avoids the use of animal experiments. The rat acute toxicity of the compounds 1, 3, 12, 13, 14, 15 and 16 was predicted by the in silico software tool GUSAR (General Unrestricted Structure-Activity Relationships, version 16. 0)  [31]  on the basis of PASS (Prediction of Activity Spectra for Substances) technology  [33, 34]. The quantitative in silico prediction was applied to four different types of administration: oral, intravenous (IV), intraperitoneal (IP), and subcutaneous (SC). As is shown in Table  2 , the differences in the predicted LD50 values obtained for the four different routes, based on the structural formulae of the tested compounds, are reported along with the acute rodent toxicity classification of the compounds.