dataesr/scientific-paragraphs-categorization huggingface dataset:
A low-density lipoprotein (LDL) uptake assay was performed as described previously  44. Briefly, the iECs were incubated with 10 μg/ml acetylated LDL (Ac-LDL) labeled with 1,1′-dioctadecyl-3,3,3′,3′-tetramethylindo-carbocyanine perchlorate (DiI-Ac-LDL) (Invitrogen, Catalog# L3484) for 4 h at 37 °C. The DiI-Ac-LDL uptake was assessed and quantified by fluorescent microscopy. Subsequently, cells were dissociated into single cells with TrypLE TM Express Enzyme. The data acquisition was performed on a MACSQuant Flow Cytometer (Miltenyi Biotec, Bergisch Gladbach, Germany) and the results were analyzed with FlowJo software (FlowJo, LLC). Co-culture assays Transmigration assay. The transwell inserts of a 24-well plate were coated with fibronectin (31. 25 µg/mL) for 30 min and followed seeding iPS-derived endothelial cells (iECs) at density of 60,000 cells/well in the upper insert.
The iEC were seeded at 120,000 cells per well in 400 μL of iEC growth medium. The barrier integrity of iEC was monitored for 48 h when the cells were treated with dasatinib (10 nM, and 50 nM, Selleckchem, Houston, TX), TNFα inhibitor adalimumab (12 μg/ml), and dasatinib plus TNFα inhibitor. The multiple frequency data were collected for barrier modeling as recommended by Applied Biophysics. ECIS measurements were acquired from three independent biological repeats with three replicates per experiment. The panel shown in Fig. 6  is from one independent experiment with three replicates, which is representative of the three biological repeats. The results of all three independent biological repeats were graphed in panel c of the figure. Graphs were generated using GraphPad Prism software (GraphPad Software).
Acknowledgements The authors would like to thank  Rachel Vantries ,  B. A. ,  Gina Montealegre, M. D. ,  Dawn Chapelle , C. R. N. P. ,  Yan Huang, M. D. ,  April Brundidge ,  R. N. ,  Alexandra Liptakova, M. D. ,  Anna Sediva, M. D. , Ph. D. ,  Milan Macek ,  Jr, M. D. Ph. D. , and  Zdenek Sumnik, M. D. , Ph. D. for logistical help with patients and sample management;  Ondrej Hrusak, M. D. , Ph. D. ,  Tomas Kalina, M. D. Ph. D. ,  Milan Macek Jr. , M. D. , Ph. D. ,  Zdenek Sumnik, M. D. , Ph. D. ,  Anna Sediva, M. D. , Ph. D. ,  Kveta Blahova, M. D. Ph. D. ,  Alexandra Liptakova, M. D. ,  Veronika Koukolska, M. D. ,  Quan Yu ,  M. D, M. S. ,  Karen Lau, M. S. , for their contribution to this study, and the families for their cooperation. Funding was provided by the  Intramural Research Program  of the  NIH ,  NIAID ,  NHLBI ,  NIAMS  and the  Clinical Center , the  Ministry of Health of the Czech Republic  ( NV19-05-00332 ,  NU20-05-00320 ,  NV18-05-00162  and project for the conceptual development of research organization - University Hospital Motol 00064203 ),  Ministry of Education, Youth and Sports of the Czech Republic  ( LM2018132  -to M.
Data availability Next-generation sequencing data from patients 1, 2, and 3 are not publicly available due to patient privacy and confidentiality. Requests for data use for medical research should be directed to the corresponding author. Upon request, we will provide the data within two weeks provided that the requesting investigator will use appropriate protection of the patient's privacy according to data use agreement. The LYN (NM_002350. 4) variants were deposited in ClinVar (https:// www. ncbi. nlm. nih. gov/clinvar/) under accession numbers SCV002822944 for c. 1519 C > T, p. Q507*; SCV002822945 for c. 1523 A > T, p. Y508F; and SCV002822946 for c. 1524 C > G, p. Y508*. The variants were also added to the Infevers database (https://infevers. umai-montpellier. fr/web/), The Registry for Autoinflammatory Disorders Mutations, shortly after publication.
Furthermore, in their mission of prevention, the pharmacist can participate in the research and development of new diagnostic tools and treatments through the preclinical development and/or the implementation of clinical trials on innovative medical devices, molecules, or combinations in the diagnostic and treatment of malaria. Thus, there are 15 registered clinical trials on malaria in France in the Clinical-Trials. gov database accessed on 13 May 2023, of which 10 have recruited patients in the last 10 years  [53]. In Europe, but also in endemic areas, intensive cooperation is needed between pharmacists, physicians, other health professionals, and travel medicine stakeholders to ensure the proper implementation of the public health guidelines and to achieve the health workforce desired by the WHO to accelerate progress toward universal health coverage and sustainable development goals.
Sarcasm is a form of figurative speech where the intended meaning of a sentence is different from it literal meaning. Sarcastic expressions tend to confuse automatic NLP approaches in many application domains, making their detection of significant importance. One of the challenges in machine learning approaches to sarcasm detection is the difficulty of acquiring ground-truth annotations. Thus, human-annotated datasets usually contain only a few thousand texts, often being unbalanced. In this paper, we propose two different pipelines of data augmentation to generate more sarcastic data. The first one is SMERT-BERT, a modified SMERTI pipeline that uses RoBERTa as the language model for the text infilling module. The second one is SWORD (semantic text exchange by Word-Attribution), where we modified the masking module in the SMERTI pipeline by utilizing the word-attribution value.
These approaches are combined with a SLOR (syntactic log-odds ratio) metric to filter the generated sarcastic data and only select sentences with the best score. Our experiments show that the use of a SLOR filter has a significant positive contribution to the augmentation process. In particular, we achieve the best results when using the SMERT-BERT pipeline and a SLOR filter by improving the F-measure by 4. 00% on the iSarcasm dataset, compared to the baseline models.
Introduction In this paper, we explore the effects of data augmentation in sarcasm detection by generating more training data using a modified SMERTI  [1]  pipeline with RoBERTa (A Robustly Optimized BERT pretraining approach) for the text infilling module and using Word-Attribution for the mask module. We analyze the results using three different datasets: Ghosh, iSarcasm, and SemEval-18. Sarcasm is a term that refers to the use of words to ridicule, irritate, or amuse someone. It is widely used on social networks. The metaphorical and creative nature of sarcasm creates considerable difficulty to detect sarcastic sentences when using machine learning approaches  [2]. Furthermore, because sarcasm indicates a sentiment  [3] , its detection in a text is essential to predict the accurate sentiment of the text.
Thus, sarcasm detection is a valuable tool with many applications in areas such as safety, health, service, product evaluation, and sales. It is also an essential aspect of creative understanding of language  [4]  and online opinion mining  [5]. Yet even for humans, identification of sarcasm is difficult due to the high contextualization  [6]. There have been various works in sarcasm detection in recent years. Goel et al. [7]  use various deep learning models such as long short-term memory (LSTM), gated recurrent unit (GRU), and baseline convolutional neural networks (CNN) in sarcasm detection using news headlines and Reddit datasets. To further improve the performance, they proposed ensemble models and found that the weighted average ensemble gave the best performance. Another research by Amer and Siddiqu  [8]  uses three feature set engineering: context-based on features set, sarcastic based on features, and lexical based on feature, and found that by combining the three feature sets.
Although recent work in sarcasm detection has been improved, there is a problem with the scarcity of sarcastic data in sarcasm detection datasets. Due to the difficulty of acquiring ground-truth annotations in sarcasm datasets, human-annotated datasets usually contain only a few hundred or thousands of sarcastic data items and often are unbalanced with respect to the sarcastic vs. non-sarcastic expressions  [10]. One of the possible mitigations against the lack of data and their balance is to artificially enhance the existing data. However, the main challenge in performing data augmentation (especially when generating sarcastic data) is to maintain the sarcastic nature of the text and maintain the quality of the generated text. In order to achieve this, we measure the quality of the data using the SLOR (syntactic log-odd ratio) metric  [11].
SLOR works by measuring the fluency of a sentence. SLOR gives a score to a sentence S based on the log-probability of a given language model, normalized by the length of the sentence and the unigram log-probability. Kann et al. [11]  prove in their article that SLOR is a good reference-less metric for natural language generation systems. In this paper, we use data augmentation with two different pipelines, first by using the pre-trained RoBERTa  [12]  model to generate sarcastic data and secondly, by utilizing Wordattribution  [13]  for masking the original text and using pretrained RoBERTa  [12]  model to generate the sarcastic data. Our contributions are the following:.
3) We use the SLOR metric to filter the generated sarcastic data, both for SMERT-BERT and semantic text exchange by Word-Attribution (SWORD) then prove that SLOR is a good metric to improve the data augmentation of sarcasm detection on both of those pipelines. Our contributions considered as state-of-the-art due to how we utilized advanced technique such as RoBERTa, as the text infilling module for the SMERT-BERT pipeline, and also utilizing semantic text exchange by using SWORD pipeline to generate more sarcastic data. Moreover, our study includes an evaluation of the generated data using SLOR metric, which is a novel approach in the field of sarcasm detection. The remainder of this paper is organized as follows: in section II, we discuss related work, and section III introduces the datasets and the methodology that we used.
[16]  tried various NLP data augmentation techniques on Yelp reviews. The first one, "Random Insertion, Deletion, & Swap," consists of adding, then deleting a random word in the sentence, and finally taking two words and swapping them. The second one, "Semantic Text Exchange," consists of using similarity between tokens in a sentence and keywords in a dictionary made by the most occurring words in the dataset, then creating a masking module based on the similarity of the replaced word and the other words in the sentence. The two last techniques studied in  [16]  is "Synthetic Noise" and "Keyword Replacement. " It is indicated in their article that "Synthetic Noise" and "Keyword Replacement" can provide better performance. Abaskohi et al. [2]  also tried data augmentation with generator-based and mutation-based methods in NLP problems.
The first one is using the GPT-2  [17]  generative model to generate both sarcastic, and nonsarcastic data, and extracting samples from the generated sentences. The second uses synonym replacement, word elimination, and shuffling. They show that those methods do not increase F-measure performance for sarcasm detection. This is because GPT-2 is not pre-trained specifically to generate sarcastic data, thus using either GPT-2 or the new GPT-3, to generate sarcastic data will ultimately still depend on the quality and size of the training data that is used to train the GPT models. Others augmentation techniques are also already implemented in sarcasm detection. Abdullah et al. [18]  use the downsampling and augmentation method to produce a balanced dataset. Shekhawat et al. [19]  performed data augmentation using nlpaug library by taking sarcastic tweets given by organizers and using the word-replacement technique to synthesize three additional tweets from each input tweet.
According to their research, data augmentation proved to improve the performance of sarcasm detection. Data augmentation also proved to improve the performance of other text classification tasks. Moreno Monterde et al. [20]  propose two models, the first one is a binary multilabel classifier using Bayesian networks, and the second one is using BERT. The dataset that they were using was very unbalanced. Thus, they use balancing techniques in order to achieve good results. They use a synonym augmenter by swapping one random word by its synonym and keeping the label. They also suggest to only applied the technique only once to prevent overfitting. The results show that using a synonym augmenter could improve the performance of sarcasm detection. Although this approach seems to improve the performance, there is a lack of analysis on the performance difference before and after augmentation.
Also, augmenting by only replacing one word with the synonym could potentially generate synthetic data that is very similar to the original one, thus risking overfitting the model. In earlier works, Handoyo et al. [21]  augmented the data by only changing one word to its synonym. demonstrate the overfitting effects, which cause performance to decline as more augmented data are used. Stylianou et al. [22]  propose a data augmentation method for low-resource and imbalanced datasets by aligning language models to in-domain data prior to generating synthetic examples. They propose alignment to existing generic models in task-specific unlabeled data to boost the performance of text classification tasks. They found that in-domain alignment help creates better synthetic data and improve the performance of text classification.
Methodology Our data augmentation approaches: SMERT-BERT and SWORD are described in subsections IV-A and IV-B. In order to measure the impact of our approaches to sarcasm detection and to compare the results to baseline model, we use the following experimental protocol as illustrated in Fig. 1. Our experimental setup focuses on generating new sarcastic data using three different datasets: Ghosh  [26] , Isarcasm  [10] , and Semeval-18  [14]. We then continue by performing data pre-processing with six different steps, then filter it by only taking the sarcastic data, and perform data augmentation with our two different pipelines SMERT-BERT and SWORD. The training data and the augmented data were then combined, and for each dataset, a validation portion was used to evaluate the model. 1) Dataset iSarcasm dataset contains tweets that are written by participants of an online survey and is an example of intended sarcasm text as the authors of the tweets themselves have annotated the data.
Due to the nature of how this dataset is collected, iSarcasm dataset is one of the most unbalanced datasets that was used in this research. For SemEval-18 dataset, it contains sarcastic tweets that were labeled by third-party annotators and are used for perceived sarcasm detection. Even though the data is labeled manually, SemEval-18 dataset still seems fairly balanced. As for the Ghosh dataset, it was an automatically collected dataset that contain tweets having particular hashtags such as #sarcasm, #not as sarcastic, and others as non-sarcastic. The Gosh dataset were labeled automatically, thus providing a fairly balanced dataset. The distribution of sarcastic and nonsarcastic data in every dataset can be seen in Table  1. 2) Data preprocessing Pre-processing is an important part of this experiment.
It is done to make sure that the text transforms into a more digestible form so that the models can perform well. The steps of pre-processing conducted in this study are by dropping all duplicate tweets across the datasets, and deleting all URL links, hashtags, foreign language characters, stop words removal, non-English ASCII characters, and emoji. 3) Filter only data labeled as sarcastic Our experiments focus on generating sarcastic labeled text in the datasets. Thus here we filter the data and only use the texts tagged as sarcastic labeled text for performing the data augmentation.
4) Semantic text exchange Once the data has been cleaned and filtered, we apply our approaches for performing semantic text exchange by maintaining the semantic value of the sarcastic text using two different pipelines: SMERT-BERT and SWORD which we will be defining later. The amount of generated sarcastic text varies between datasets. This is because different datasets need different amounts of data to be balanced. Details of the amount of generated sarcastic text that was used and the class distribution before and after data augmentation can be seen in Table  3. International.
Data augmentation algorithms In this experiment, we apply data augmentation only in training data to prevent information leaking to the validation dataset which used to evaluated the models. Among our algorithms, SMERT-BERT takes the sarcastic pre-processed sentences as input, then applies random masking based on the length of the sentence. As for SWORD (semantic text exchange by Word-Attribution), it takes the attribution score of each word from our pre-processed sarcastic data, then masks the words with the best and worst attribution scores. We then make the mask-filling in the text infilling module, which is a pre-trained RoBERTa, adapted to mask-filling tasks. Finally, to make sure our generated sarcastic data still maintain the text quality and maintain the sarcastic nature of the text, we use the SLOR metric to filter the generated sarcastic data.
By taking the sample of data augmentation depending on the length of the chosen dataset, we compare the results when using the SLOR metric to filter the generated sarcastic data. To evaluate the quality of the generated data, we use indirect measurement approach by observing the improvement in performance after augmenting the data. The improvement here implies that we successfully preserve the quality of sarcastic text, since the model remains resilient even after merging the original training dataset with the augmented dataset.
A basic example of how SMERTI works can be seen in Fig. 2. In our research, we extend SMERTI to generate sarcastic data by using only random masking and adding text infilling using a pre-trained RoBERTa language model. As for the mask-filling module, we are using a pre-trained transformer model, RoBERTa  [12] , a more robust version of BERT  [27] , to get new sentences by using context. Details of this pipeline can be seen in Fig. 3. The figure includes two scenarios, one of which has supplementary data selection based on their SLOR score. That specific part will be detailed in subsection IV-D. As already mentioned, our pipeline is applied to the three reference datasets. For the masking, we use random masking and mask 20% of the original sentence. We determine to use 20% due to an initial experiment that tests the masking using four different configurations, from 20%, 40%, 60%, and 80% of random masking.
The results show that using 20% of random masking gives an overall better F-measure when detecting sarcastic sentences when using SMERTBERT pipeline, compared to other percentages. The same outcome can also be seen in precision, recall, and accuracy results. Our initial experiment results, which using SMERTBERT pipeline can be seen in Table  2. After the random masking process, we then use a pretrained RoBERTa language model for replacing the masked parts with the most similar words, which means the word has the embedding closest to the masked parts while maintaining the context of the sentences. To maintain the context of the sentences, RoBERTa uses contextual embedding, where the vector representation of each word in the vocabulary was obtained after training the whole sentences on the model, thus the pre-trained model also learned the contextual meaning of the word  [12].
The newly generated dataset is then used to augment the original data. The proportion of the sarcastic and nonsarcastic data, before and after augmentation can be seen in Table  3. Based on Table  3 , we can see that for the iSarcasm dataset, we only add 106 sarcastic data elements. For the Ghosh dataset, we add 3,098 sarcastic data items for SemEval-18 dataset we add 147. We add only a small amount of data, and the reason is, if we add too much generated sarcastic data, it will contribute more noise in the model and affect the performance, thus the amount of augmented data that we use is based on our experiments that give the best performance boost for every dataset. We used the exact same amount of augmented data from Table  3  for every experiment scenario. After we generated the new sarcastic data, we concatenated the generated data with the original data and build the model RoBERTa, and BERT have essentially identical architectures, the researchers made a few modest changes to RoBERTa design and training methodology to improve the model's performance  [12].
SWORD (semantic text exchange by Word-Attribution) SWORD, uses Word-Attribution for the masking, as for the rest of the pipeline, it is the same as SMERT-BERT pipeline. The details of this pipeline can be seen in Fig. 4. The SWORD only masks the words with the most positive attribute and most negative attribute scores. We only mask two of those words because of the fact that the most positive attribute word contributes to how the sentence is sarcastic, and the most negative score contributes to how the sentence is less sarcastic. We then use RoBERTa for the text infilling module and generate it with a random sample and retain only the sentence with the best SLOR. The generated sarcastic data that were explained before in Table  3 , are then concatenated with the original data and used to build the sarcasm detection model using RoBERTa pre-trained language model.
Result and discussion For measuring the model performance when classifying between sarcastic and non-sarcastic sentences, we use Fmeasure, precision, and recall. Fmeasure uses the harmonic mean of precision and recall to penalize extreme values. It is the proper metric to use in case false negatives and false positives are of the same importance. This also means that F-measure is a good metric for an imbalanced dataset. To fully evaluate the effectiveness of the model, precision and recall results are also provided. All results are in this section, including baseline, SMERT-BERT, and SWORD with and without selection of data using SLOR. After that, we analyse the results by comparing the results of the baseline method and the proposed method. We also analyse if applying SLOR filtering increases the results when compared to random samples.
Model performance The performance comparison between the baseline and the proposed methods can be seen in Table  5. For readability, the performance increase (difference between the baseline and the proposed methods) can be seen in Table  6. The latter can be entirely derived from the former. Based on Table  5 , we can see that data augmentation, whether using SMERT-BERT or SWORD pipeline, gives better performance results when compared to the baseline model. Nevertheless, when we analyse the amount of increased performance from Table  6 , we can see that the performance improvement depends on the dataset. Among the three datasets, iSarcasm gives the most significant performance boost when using SMERT-BERT with SLOR filtering, with a +0. 04 of performance gain on F-measure. iSarcasm had a characteristic of a small and imbalanced dataset.
Although SemEval-18 was also a small dataset, it was still fairly balanced, thus the most significant performance gain was at +0. 03 of F-measure. As for the Ghosh dataset, its size was big and was also fairly balanced, thus when we applied data augmentation whether when we are using SMERT-BERT or SWORD pipeline, There are no improvements on the Fmeasure. From here we can assume that data our augmentation approach is effective only when applied to a small and imbalanced dataset. The performance of the model also depends on the dataset that had been used to create the model. From Table  5  we can see that the best F-measure performance is obtained when using the Ghosh dataset with an F-measure of 0. 79. iSarcasm and SemEval-18 were overall quite less performing with the best F-measure of 0.