MuCoT: Multilingual Contrastive Training for Question-Answering in Low-resource Languages

GK Kumar, AS Gehlot, SS Mullappilly, K Nandakumar - arXiv preprint arXiv …, 2022

GKKAS Gehlot, SSMK Nandakumar - DravidianLangTech 2022, 2022

Accuracy of English-language Question Answering (QA) systems has improved significantly in recent years with the advent of Transformer-based models (eg, BERT). These models are pre-trained in a self-supervised fashion with a large English text …


Multi language Email Classification Using Transfer learning

In recent years Artificial Intelligence has become a core part of many businesses, from manufacturers to service providers, AI can be found helping to improve business processes as well as providing customized experiences and support for …


Multi-class sentiment analysis of urdu text using multilingual BERT

L Khan, A Amjad, N Ashraf, HT Chang - Scientific Reports, 2022

Sentiment analysis (SA) is an important task because of its vital role in analyzing people’s opinions. However, existing research is solely based on the English language with limited work on low-resource languages. This study introduced a new …


Multi-domain Neural Machine Translation

Today, neural machine translation (NMT) systems constitute state-of-the-art systems in machine translation. However, such translation models require relatively large train data and struggle to handle a specific domain text. A domain may consist of …


Multi-label Movie Genre Classification Using Multiple Modalities

This project explores an ensemble approach to the task of multi-label movie genre classification. The proposed model combines data from movie posters, video trailer previews, text plot summaries, and metadata using separate deep networks. Other …


Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model

M Li, F Ding, D Zhang, L Cheng, H Hu, F Luo - arXiv preprint arXiv:2211.01200, 2022

Pre-trained multilingual language models play an important role in cross-lingual natural language understanding tasks. However, existing methods did not focus on learning the semantic structure of representation, and thus could not optimize their …


Multi-modal Chest X-Ray analysis: classification and report generation using self-supervised learning

Automated medical systems for classification, localization and diagnosis are increasingly being researched and developed. Accurate and automated disease detection is beneficial both to medical personnel, who do not have to perform …


Multi-objective search for gender-fair and semantically correct word embeddings

M Hort, R Moussa, F Sarro - Applied Soft Computing, 2022

Fairness is a crucial non-functional requirement of modern software systems that rely on the use of Artificial Intelligence (AI) to make decisions regarding our daily lives in application domains such as justice, healthcare and education. In fact, these …


Multi-source Multi-domain Sentiment Analysis with BERT-based Models

… FEEL-IT, the other BERTbased model, is a sentiment classifier based on UmBERTo2, which inherits the RoBERTa architecture and is pre-trained on Commoncrawl ITA. FEEL-IT has only one head used for predicting positive and …


Multi-task learning for toxic comment classification and rationale extraction

KB Nelatoori, HB Kommanti - Journal of Intelligent Information Systems, 2022

Social media content moderation is the standard practice as on today to promote healthy discussion forums. Toxic span prediction is helpful for explaining the toxic comment classification labels, thus is an important step towards building automated …


Multi-Wiki90k: Multilingual Benchmark Dataset for Paragraph Segmentation

M Swędrowski, P Miłkowski, B Bojanowski, J Kocoń - Conference on Computational …, 2022

In this paper, we present paragraph segmentation using cross-lingual knowledge transfer models. In our solution, we investigate the quality of multilingual models, such as mBERT and XLM-RoBERTa, as well as language independent models …


Multiaztertest@ Exist-Iberlef2022: Sexism Identification in Social Networks

The automatic identification of sexism and the categorization of the message in different types of sexism in social networks is the focus of the systems taking part in the EXIST-IberLEF 2022 shared task. Detecting online sexism and encouraging …


Multidimensional tie strength and economic development

LM Aiello, S Joglekar, D Quercia - Scientific Reports, 2022

The strength of social relations has been shown to affect an individual’s access to opportunities. To date, however, the correspondence between tie strength and population’s economic prospects has not been quantified, largely because of the …


Multilingual and Multitarget Hate Speech Detection in Tweets

Social media networks have become a space where users are free to relate their opinions and sentiments which may lead to a large spreading of hatred or abusive messages which have to be moderated. This paper proposes a supervised …


Multilingual Auxiliary Tasks Training: Bridging the Gap between Languages for Zero-Shot Transfer of Hate Speech Detection Models

S Montariol, A Riabi, D Seddah - arXiv preprint arXiv:2210.13029, 2022

Zero-shot cross-lingual transfer learning has been shown to be highly challenging for tasks involving a lot of linguistic specificities or when a cultural gap is present between languages, such as in hate speech detection. In this paper, we highlight …


Multilingual Bidirectional Unsupervised Translation Through Multilingual Finetuning and Back-Translation

B Li, A Patel, C Callison-Burch, MS Rasooli - arXiv preprint arXiv:2209.02821, 2022

… Some mined web data is available even for these low-resource languages, of which we use CommonCrawl and CC-100 datasets. … For the 8 test languages we draw from Common Crawl17 for {kk, gu, is} and CC-10018 for {my, ps, lv, ne, si}. …


Multilingual fine-tuning for Grammatical Error Correction

K Pajak, D Pajak - Expert Systems with Applications, 2022

… mT5 was pre-trained on a large mC4 dataset covering 101 languages and containing 26 TB of the Common Crawl web scrape. The model was pre-trained on the span corruption task, where consecutive spans of input tokens were replaced …


Multilingual Hate Speech Detection

… It was trained on 100 languages, of a CommonCrawl Corpus, and it outperforms multilingual BERT on crosslingual classification. XLM-… The XLM-RoBERTa model, on the other hand, was trained on 2.5TB of filtered CommonCrawl data across 100 …


Multilingual hope speech detection in English and Dravidian languages

BR Chakravarthi - International Journal of Data Science and Analytics, 2022

Recent work on language technology has aimed to identify negative language such as hate speech and cyberbullying as well as improve offensive language detection to mediate social media platforms. Most of these systems rely on using machine …


Multilingual Language Model Adaptive Fine-Tuning: A Study on African Languages

JO Alabi, DI Adelani, M Mosbach, D Klakow - arXiv preprint arXiv:2204.06487, 2022

Multilingual pre-trained language models (PLMs) have demonstrated impressive performance on several downstream tasks on both high resourced and low-resourced languages. However, there is still a large performance drop for languages unseen …


Multilingual Name Entity Recognition and Intent Classification employing Deep Learning architectures

S Rizou, A Paflioti, A Theofilatos, A Vakali… - … Modelling Practice and …, 2022

… For the fastText initialization we use a model which is pre-trained in Common Crawl and Wikipedia. … For the fastText initialization we used the model in Common Crawl and Wikipedia. For the GloVe initialization we used the 300 dimensional …


Multilingual Natural Language Understanding for the FPT. AI Conversational Platform

P Le-Hong, TTL Nguyen, MT Pham, TH Vu - 2022 14th International Conference on …, 2022

This paper presents a multilingual natural language understanding model which is based on BERT and ELECTRA neural networks. The model is pre-trained and fine-tuned on large datasets of four languages: Indonesian, Malaysian, Japanese and …


Multilingual Relation Classification via Efficient and Effective Prompting

Y Chen, D Harbecke, L Hennig - arXiv preprint arXiv:2210.13838, 2022

Prompting pre-trained language models has achieved impressive performance on various NLP tasks, especially in low data regimes. Despite the success of prompting in monolingual settings, applying prompt-based methods in multilingual scenarios …


Multilingual Representation Distillation with Contrastive Learning

W Tan, K Heffernan, H Schwenk, P Koehn - arXiv preprint arXiv:2210.05033, 2022

Multilingual sentence representations from large models can encode semantic information from two or more languages and can be used for different cross-lingual information retrieval tasks. In this paper, we integrate contrastive learning into …


Multilingual Sequence-to-Sequence Models for Hebrew NLP

M Eyal, H Noga, R Aharoni, I Szpektor, R Tsarfaty - arXiv preprint arXiv:2212.09682, 2022

… 2021) in comparison, was trained on mC4, where its public replica1 is a 27TB collection of natural text in 101 languages drawn from the public Common Crawl, 66GB of which is Hebrew (4.125x more Hebrew training data compared to …


Multilingual taxonomic text classification in an incremental number of languages

Taxonomic text classification is the Natural Language Processing (NLP) branch that aims at classifying text into a hierarchically organized schema. Its applications space from pure research to industrial and commercial purposes. In this thesis, in particular …


Multilingual Transformer Encoders: a Word-Level Task-Agnostic Evaluation

F Gaschi, F Plesse, P Rastin, Y Toussaint - arXiv preprint arXiv:2207.09076, 2022

… However, it could also be explained by the fact that mBERT is trained solely on Wikipedia whereas models like XLM-R are trained on the CommonCrawl corpus which might contain texts that are less comparable across languages. It is also to be …


Multilingual Transformers for Product Matching--Experiments and a New Benchmark in Polish

A Wróblewska, S Tkachuk, S Łukasik - arXiv preprint arXiv:2205.15712, 2022

… The authors of these studies extracted offers from websites collected under the CommonCrawl project.The selection of offers was based on the schema.org8 tags, which Internet sellers had started using in the last five … 7commoncrawl.org 8schema.org …


Multilinguals at SemEval-2022 Task 11: Complex NER in Semantically Ambiguous Settings for Low Resource Languages

A Pandey, S Daw, N Unnam, V Pudi - Proceedings of the 16th International Workshop …, 2022

We leverage pre-trained language models to solve the task of complex NER for two low-resource languages: Chinese and Spanish. We use the technique of Whole Word Masking (WWM) to boost the performance of masked language modeling …


Multimodal Graph for Unaligned Multimodal Sequence Analysis via Graph Convolution and Graph Pooling

S Mai, S Xing, J He, Y Zeng, H Hu - ACM Transactions on Multimedia Computing …, 2022

Multimodal sequence analysis aims to draw inferences from visual, language and acoustic sequences. A majority of existing works focus on the aligned fusion of three modalities to explore inter-modal interactions, which is impractical in real-world …


Multimodal sentiment analysis in real-life videos

This thesis extends the emerging field of multimodal sentiment analysis of real-life videos, taking two components into consideration: the emotion and the emotion’s target. The emotion component of media is traditionally represented as a segment-based …


Multimodality for NLP-Centered Applications: Resources, Advances and Frontiers

With advancements in the methods of Natural Language Processing (NLP) by explicitly considering other modalities than just text, the resurgence of multimodal datasets has attracted significant attention. However, there remains lack of a …


Multitasking of sentiment detection and emotion recognition in code-mixed Hinglish data

S Ghosh, A Priyankar, A Ekbal, P Bhattacharyya - Knowledge-Based Systems, 2022

As the number of non-native English speakers on social media has skyrocketed in recent years, sentiment and emotion analysis on regional languages and code-mixed data has gained traction. Despite extensive research on English, the area of Hindi-English …


MUST-VQA: MUltilingual Scene-text VQA

E Vivoli, AF Biten, A Mafla, D Karatzas, L Gomez - arXiv preprint arXiv:2209.06730, 2022

… The mT5 model was trained on a 101-language Common Crawl-based multilingual variation. Only English Common Crawl is what T5 has been pre-trained on. Additionally an increase in performance is obtained by the use of GeGLU nonlinearities [31]. …


Mutual Relation of Machine Translation and Quality Estimation

Machine Translation Quality Estimation predicts quality scores for translations produced by Machine Translation systems based on source and output segments. Quality Estimation systems are usually trained in a supervised manner using training …


naab: A ready-to-use plug-and-play corpus for Farsi

S Sabouri, E Rahmati, S Gooran, H Sameti - arXiv preprint arXiv:2208.13486, 2022

… 2020) in Farsi is a 70GB cleaned text corpus from a compilation of 8 (Common Crawl - fa (Crawl… 2022) or Open Super-large Crawled ALMAnaCH coRpus is a huge multilingual corpus obtained by language classification and filtering of the …


NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis

SH Muhammad, DI Adelani, IS Ahmad, I Abdulmumin… - arXiv preprint arXiv …, 2022

Sentiment analysis is one of the most widely studied applications in NLP, but most work focuses on languages with large amounts of data. We introduce the first large-scale human-annotated Twitter sentiment dataset for the four most widely spoken …


NamedEntityRangers at SemEval-2022 Task 11: Transformer-based Approaches for Multilingual Complex Named Entity Recognition

A Miftahova, A Pugachev, A Skiba, K Artemova… - Proceedings of the 16th …, 2022

This paper presents the two submissions of NamedEntityRangers Team to the MultiCoNER Shared Task, hosted at SemEval-2022. We evaluate two state-of-the-art approaches, of which both utilize pre-trained multi-lingual language models …


NapierOne: A modern mixed file data set alternative to Govdocs1

SR Davies, R Macfarlane, WJ Buchanan - Forensic Science International: Digital …, 2022

It was found when reviewing the ransomware detection research literature that almost no proposal provided enough detail on how the test data set was created, or sufficient description of its actual content, to allow it to be recreated by other …


Natural Language Correction With Focus on Czech

Natural language correction, a subfield of natural language processing (NLP), is the task of automatically correcting user errors in written texts. It includes, but is not limited to, grammatical error correction, spelling error correction and diacritics …


Natural Language Inference with Self-Attention for Veracity Assessment of Pandemic Claims

M Arana-Catania, E Kochkina, A Zubiaga, M Liakata… - arXiv preprint arXiv …, 2022

… on the health domain focused on information retrieval from general websites through the Common Crawl corpus (commoncrawl.org). … Research challenge using claims on the health domain focused on information retrieval from general …


Natural Language Processing in toxicology: Delineating Adverse Outcome Pathways and guiding the application of New Approach Methodologies

MPF Corradi, AM de Haan, B Staumont, AH Piersma… - Biomaterials and …, 2022

Adverse Outcome Pathways (AOPs) are conceptual frameworks that tie an initial perturbation (molecular initiating event) to a phenotypic toxicological manifestation (adverse outcome), through a series of steps (key events). They provide therefore a …


Natural Questions in Icelandic

We present the first extractive question answering (QA) dataset for Icelandic, Natural Questions in Icelandic (NQiI). Developing such datasets is important for the development and evaluation of Icelandic QA systems. It also aids in the …


Navigable Proximity Graph-Driven Native Hybrid Queries with Structured and Unstructured Constraints

M Wang, L Lv, X Xu, Y Wang, Q Yue, J Ni - arXiv preprint arXiv:2203.13601, 2022

As research interest surges, vector similarity search is applied in multiple fields, including data mining, computer vision, and information retrieval. {Given a set of objects (eg, a set of images) and a query object, we can easily transform each object …


Need for Wheel Speed: Generating synthetic wheel speeds using LSTM and GANs

… One of the largest data sets available is Common crawl 1 which consists of over 5 Peta Bytes of data in total which is huge amounts of data, and it grows consistently with around 300 Tebibyte of data every two months. At this rate, it will take close to …


Nepali Encoder Transformers

U Maskey, M Bhatta, SR Bhatta, S Dhungel, BK Bal

… ● XLM-RoBERTa: XLM-RoBERTa (Conneau et al., 2020) is a multi-lingual model that is trained on large dataset of filtered CommonCrawl data covering 100 languages. …


NER in Hindi Language Using Transformer Model: XLM-Roberta

AA Choure, RB Adhao, VK Pachghare - 2022 IEEE International Conference on …, 2022

… It has been pre-trained on 2.5 TB of filtered Common-Crawl data with hundred languages. It was pre-trained with the Masked language modeling (MLM) goal. Taking a sentence as an example, the model masks 15% of the words in the input at …


NerKor+ Cars-OntoNotes+

In this paper, we present an upgraded version of the Hungarian NYTK-NerKor named entity corpus, which contains about twice as many annotated spans and 7 times as many distinct entity types as the original version. We used an extended …


Net impact of large language models trained on code

P Gherciu - … tehnico-ştiinţifică a studenţilor, masteranzilor şi …, 2022

Natural language processing has seen many improvements in recent years, particularly driven by machine learning models such as OpenAI’s GPT-3. This paper aims to present the various language models, as well as OpenAI Codex, which is …


Neural Approaches to Conversational Information Retrieval

J Gao, C Xiong, P Bennett, N Craswell - arXiv preprint arXiv:2201.05176, 2022

A conversational information retrieval (CIR) system is an information retrieval (IR) system with a conversational interface which allows users to interact with the system to seek information via multi-turn conversations of natural language, in spoken or …


Neural Language Models as What If?-Engines for HCI Research

P Hämäläinen, M Tavast, A Kunnari - 27th International Conference on Intelligent …, 2022

Collecting data is one of the bottlenecks of Human-Computer Interaction (HCI) and user experience (UX) research. In this poster paper, we explore and critically evaluate the potential of large-scale neural language models like GPT-3 in …


Neural machine translation for Hungarian

LJ Laki, ZG Yang - Acta Linguistica Academica, 2022

In the scope of this research, we aim to give an overview of the currently existing solutions for machine translation and we assess their performance on the English-Hungarian language pair. Hungarian is considered to be a challenging language for machine …


Neural Natural Language Generation: A Survey on Multilinguality, Multimodality, Controllability and Learning

Developing artificial learning systems that can understand and generate natural language has been one of the long-standing goals of artificial intelligence. Recent decades have witnessed an impressive progress on both of these problems, giving …


Neural text summarization for Hungarian

ZG Yang - Acta Linguistica Academica, 2022

… One of the most prominent representative dataset in this category is the Common Crawl. However, despite the fact that web scraping can give rise to large datasets, the filtering and the cleaning of data can be still challenging and often claimed as a …


Neural Transfer Learning For Vietnamese Sentiment Analysis Using Pre-trained Contextual Language Models

AP Le, TV Pham, TV Le, DV Huynh - 2021 IEEE International Conference on Machine …, 2021

We propose a fine-tuning methodology and a comprehensive comparison between state-of-the-art pre-trained language models (PLM) when applying to Vietnamese Sentiment Analysis. The fine-tuning architecture includes three main components: (1) …


New directions in empirical asset pricing: information, innovation, and stock returns

This dissertation is a collection of three essays that explore new directions in empirical asset pricing. The first essay studies the role of textual information in analyst reports. I show that analysts use the report text to convey soft information that …


New meaning for NLP: the trials and tribulations of natural language processing with GPT-3 in ophthalmology

S Nath, A Marie, S Ellershaw, E Korot, PA Keane - British Journal of Ophthalmology, 2022

Natural language processing (NLP) is a subfield of machine intelligence focused on the interaction of human language with computer systems. NLP has recently been discussed in the mainstream media and the literature with the advent of Generative …


New Methods and the Study of Vulnerable Groups: Using Machine Learning to Identify Immigrant-Oriented Nonprofit Organizations

Many migrants are vulnerable due to noncitizenship, linguistic or cultural barriers, and inadequate safety-net infrastructures. Immigrant-oriented nonprofits can play an important role in improving immigrant well-being. However, progress on …


News Article Classification using a Transfer Learning Approach

K Abhishek - 2022 10th International Conference on Reliability …, 2022

The increasing proliferation of electronic files in recent decades has been addressed by significant improvements in Natural Language Processing. As the text data generated is quite large, employing sophisticated algorithms that can process …


NLI Models for Assessing Facticity in Summarization Methods

Guidelines: The task is to: 1) Research state-of-the-art NLP approaches for text summarization, focusing on evaluation methods for assessment of generated summaries. Explore mainly model-based approaches. 2) Discuss available options …


NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measures

J Vamvas, R Sennrich - arXiv preprint arXiv:2204.13692, 2022

Being able to rank the similarity of short text segments is an interesting bonus feature of neural machine translation. Translation-based similarity measures include direct and pivot translation probability, as well as translation cross-likelihood, which …


No Language Left Behind: Scaling Human-Centered Machine Translation

N Team, MR Costa-jussà, J Cross, O Çelebi…

Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today. However, such efforts have coalesced around a small subset of languages, leaving …


No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Media

M Spliethöver, M Keiff, H Wachsmuth - arXiv preprint arXiv:2211.03634, 2022

… (2021), we collected news articles from Common Crawl3. Since the media bias rating history is not available, we mapped each outlet to its current rating. To extract the pure text from the collected files in WARC format, we used the library news-please …


Noise Reduction Learning based on XLNet-CRF for Biomedical Named Entity Recognition

Z Chai, H Jin, S Shi, S Zhan, L Zhuo, Y Yang, Q Lian - IEEE/ACM Transactions on …, 2022

In recent years, Biomedical Named Entity Recognition (BioNER) systems have mainly been based on deep neural networks, which are used to extract information from the rapidly expanding biomedical literature. Long-distance context …


Noise-Reduction for Automatically Transferred Relevance Judgments

The TREC Deep Learning tracks used MS MARCO Version 1 as their official training data until 2020 and switched to Version 2 in 2021. For Version 2, all previously judged documents were re-crawled. Interestingly, in the track’s 2021 edition, models …


Nominalizations with the suffixes-ee and-ation A distributional semantic analysis

… • FastText Common Crawl subword model (Bojanowski et al. 2016, Mikolov et al. 2018) … • FastText Common Crawl subword model (Bojanowski et al. 2016, Mikolov et al. 2018) … • FastText Common Crawl subword model (Bojanowski et al. 2016 …


Non-Autoregressive Machine Translation: It's Not as Fast as it Seems

J Helcl, B Haddow, A Birch - arXiv preprint arXiv:2205.01966, 2022

Efficient machine translation models are commercially important as they can increase inference speeds, and reduce costs and carbon emissions. Recently, there has been much interest in non-autoregressive (NAR) models, which promise faster …


Normalization and Attention

… Sixty percent of the training data set comes from a filtered version of Common Crawl consisting of 410 billion tokens. Other sources are 19 billion tokens from WebText2, 12 billion tokens from Books1, 55 billion tokens from Books2, and 3 …


NT5 at WMT 2022 General Translation Task

M Morishita, K Kudo, Y Oka, K Chousa, S Kiyono…

… Monolingual Corpus We also used the following provided monolingual data: News Crawl, News Commentary, and Common Crawl. We backtranslated the monolingual sentences with a targetto-source model trained only with the provided …


NULL at SemEval-2022 Task 6: Intended Sarcasm Detection Using Stylistically Fused Contextualized Representation and Deep Learning

M Rahgouy, HB Giglou, T Rahgooy, C Seals - … of the 16th International Workshop on …, 2022

… ), which is pre-trained on 2.5TB of CommonCrawl data containing 100 languages. The model learns an inner representation of 100 languages that can then be used to extract features useful for downstream tasks. The function CE ∈ R768 is a base …


NUS-IDS at CheckThat! 2022: Identifying Check-worthiness of Tweets using CheckthaT5

M Du, SD Gollapalli, SK Ng - Working Notes of CLEF, 2022

This paper describes our system CheckthaT5, which was designed in the context of the Checkthat! 2022 competition at CLEF. CheckthaT5 explores the feasibility of adapting sequence-to-sequence models for detecting check-worthy social media …


NusaCrowd: Open Source Initiative for Indonesian NLP Resources

S Cahyawijaya, H Lovenia, AF Aji, GI Winata, B Wilie… - arXiv preprint arXiv …, 2022

We present NusaCrowd, a collaborative initiative to collect and unite existing resources for Indonesian languages, including opening access to previously non-public resources. Through this initiative, we have has brought together 137 datasets and …


OCHADAI at SemEval-2022 Task 2: Adversarial Training for Multilingual Idiomaticity Detection

LK Pereira, I Kobayashi - arXiv preprint arXiv:2206.03025, 2022

We propose a multilingual adversarial training model for determining whether a sentence contains an idiomatic expression. Given that a key challenge with this task is the limited size of annotated data, our model relies on pre-trained contextual …


OcWikiDisc: a Corpus of Wikipedia Talk Pages in Occitan

A Miletić, Y Scherrer - Proceedings of the Ninth Workshop on NLP for Similar …, 2022

This paper presents OcWikiDisc, a new freely available corpus in Occitan, as well as language identification experiments on Occitan done as part of the corpus building process. Occitan is a regional language spoken mainly in the south of France and in …


Odor Descriptor Understanding through Prompting

L Sisson - arXiv preprint arXiv:2205.03719, 2022

Embeddings from contemporary natural language processing (NLP) models are commonly used as numerical representations for words or sentences. However, odor descriptor words, like "leather" or "fruity", vary significantly between their …


Offensive language detection in Tamil YouTube comments by adapters and cross-domain knowledge transfer

M Subramanian, R Ponnusamy, S Benhur… - Computer Speech & …, 2022

… XLM-RoBERTa is a self-supervised transformer model that has been pre-trained on 2.5TB of filtered CommonCrawl data from 100 languages. This model does not require lang tensors to understand which language is being used, and it should be …


Offensive Language Detection in Turkish Twitter Data with BERT Models

As insulting statements become more frequent on online platforms, these negative statements create a reaction and disturb the peace of society. Identifying these expressions as early as possible is important to protect the victims. Offensive …


On Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making

J Schoeffer, M De-Arteaga, N Kuehl - arXiv preprint arXiv:2209.11812, 2022

Explanations have been framed as an essential feature for better and fairer human-AI decision-making. In the context of fairness, this has not been appropriately studied, as prior works have mostly evaluated explanations based on their effects on …


On Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation

Modern unsupervised machine translation (MT) systems reach reasonable translation quality under clean and controlled data conditions. As the performance gap between supervised and unsupervised MT narrows, it is interesting to ask …


On the Applicability of Zero-Shot Cross-Lingual Transfer Learning for Sentiment Classification in Distant Language Pairs

This research explores the applicability of cross-lingual transfer learning from English to Japanese and Indonesian using the XLM-R pre-trained model. The results are compared with several previous works, either by models using a similar …


On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model

S Shin, SW Lee, H Ahn, S Kim, HS Kim, B Kim, K Cho… - arXiv preprint arXiv …, 2022

Many recent studies on large-scale language models have reported successful in-context zeroand few-shot learning ability. However, the in-depth analysis of when in-context learning occurs is still lacking. For example, it is unknown how in-context learning …


On the Effectiveness of Quasi Character-Level Models for Machine Translation

S Carrión-Ponz, F Casacuberta - Proceedings of the 15th biennial conference of the …, 2022

Neural Machine Translation (NMT) models often use subword-level vocabularies to deal with rare or unknown words. Although some studies have shown the effectiveness of purely character-based models, these approaches have resulted in …


On the Geometry of Concreteness

C Wartena - Proceedings of the 7th Workshop on Representation …, 2022

… For English we use the 300 dimensional fastText embeddings without subword information trained on the Common Crawl with 600 billion tokens. For German we also use 300 dimensional fastText embeddings trained on the Common Crawl and …


On the Role of Bidirectionality in Language Model Pre-Training

M Artetxe, J Du, N Goyal, L Zettlemoyer, V Stoyanov - arXiv preprint arXiv:2205.11726, 2022

Prior work on language model pre-training has explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult. In this work, we focus on bidirectionality as a key …


On the Surrogate Gap between Contrastive and Supervised Losses

H Bao, Y Nagano, K Nozawa - International Conference on Machine Learning, 2022

Contrastive representation learning encourages data representation to make semantically similar pairs closer than randomly drawn negative samples, which has been successful in various domains such as vision, language, and graphs. Recent …


One Document, Many Revisions: A Dataset for Classification and Description of Edit Intents

D Rajagopal, X Zhang, M Gamon, S Jauhar, D Yang…

… It is important to note that BART was pretrained on written text over the C4 commoncrawl corpus6. And adapting it to wikipedia comment generation task did not directly lead to performance gains. Our experiments show that pretrained BART …


Online patient feedback as a safety valve: An automated language analysis of unnoticed and unresolved safety incidents

A Gillespie, TW Reader - Risk Analysis, 2022

… Specifically, we used a publicly available language model created from the Common Crawl dataset of webpages, comprising 684,831 unique vectors represented in 300 dimensions (Explosion, 2020). While word embeddings trained on medical texts …


Online Social Networks and Media

Depression is a challenge to public health, frequently related to disability and one of the reasons that lead to suicide. Many of the ones who suffer depression use social media to obtain information or even to talk about their problem. Some studies have …


Open-domain conversational search assistants: the Transformer is all you need

R Ferreira, M Leite, D Semedo, J Magalhaes - Information Retrieval Journal, 2022

On the quest of providing a more natural interaction between users and search systems, open-domain conversational search assistants have emerged, by assisting users in answering questions about open topics in a conversational manner. In this …


OPT: Open Pre-trained Transformer Language Models

S Zhang, S Roller, N Goyal, M Artetxe, M Chen, S Chen… - arXiv preprint arXiv …, 2022

… All corpora were previously collected or filtered to contain predominantly English text, but a small amount of non-English data is still present within the corpus via CommonCrawl. We removed duplicated documents across all datasets by filtering …


Optimize_Prime@ DravidianLangTech-ACL2022: Abusive Comment Detection in Tamil

S Patankar, O Gokhale, O Litake, A Mandke, D Kadam - arXiv preprint arXiv …, 2022

This paper tries to address the problem of abusive comment detection in low-resource indic languages. Abusive comments are statements that are offensive to a person or a group of people. These comments are targeted toward individuals belonging to …


Optimize_Prime@ DravidianLangTech-ACL2022: Emotion Analysis in Tamil

O Gokhale, S Patankar, O Litake, A Mandke, D Kadam - arXiv preprint arXiv …, 2022

This paper aims to perform an emotion analysis of social media comments in Tamil. Emotion analysis is the process of identifying the emotional context of the text. In this paper, we present the findings obtained by Team Optimize_Prime in the ACL 2022 …


ORCA: A Challenging Benchmark for Arabic Language Understanding

AR Elmadany, EMB Nagoudi, M Abdul-Mageed - arXiv preprint arXiv:2212.10758, 2022

… 2020b) is a transformerbased multilingual masked language model pretrained on more than 2TB of filtered CommonCrawl data in 100 languages, including Arabic (2.9B tokens). XLM-R uses a Transformer model trained a multilingual version of masked …


Ordvektorer i lexikografiskt arbete

M Forsberg, E Sköldberg - LIVE and LEARN

… Ordvektorerna är baserade på datamängden Common Crawl, som är ett stort insamlat material med webbsidor, och Wikipedia. Det finns en rik flora av vetenskapliga arbeten som kretsar runt hur frukten av lexikografiskt arbete kan …


Out-of-Distribution Detection and Selective Generation for Conditional Language Models

J Ren, J Luo, Y Zhao, K Krishna, M Saleh… - arXiv preprint arXiv …, 2022

Machine learning algorithms typically assume independent and identically distributed samples in training and at test time. Much work has shown that high-performing ML classifiers can degrade significantly and provide overly-confident, wrong …


Overview of HIPE-2022: Named Entity Recognition and Linking in Multilingual Historical Documents

A Doucet, S Clematide - … IR Meets Multilinguality, Multimodality, and Interaction …, 2022

M Ehrmann, M Romanello, S Najem-Meyer, A Doucet… - International Conference of …, 2022

This paper presents an overview of the second edition of HIPE (Identifying Historical People, Places and other Entities), a shared task on named entity recognition and linking in multilingual historical documents. Following the success of the first CLEF-HIPE-2020 …


Overview of NTCIR-16

… Chuweb21 is a subset of the Common Crawl dataset and it contains 3,402,457 domains and 858,616,203 web pages. Secondly, two versions of relevance assessment are introduced: the Gold version given by the topic creators, and the …


Overview of the 2022 BUCC Shared Task: Bilingual Term Alignment in Comparable Specialized Corpora

O Adjali, E Morin, S Sharoff, R Rapp, P Zweigenbaum - LREC 2022 Workshop …, 2022

The BUCC 2022 shared task addressed bilingual terminology alignment in comparable corpora. Many research groups are working on this problem using a wide variety of approaches. However, as there is no standard way to measure the …


Overview of Touché 2022: Argument Retrieval

A Bondarenko, M Fröbe, J Kiesel, S Syed, T Gurcke… - 2022

T Gurcke, M Beloucif, A Panchenko, C Biemann… - Experimental IR Meets …, 2022

This paper is a report on the third year of the Touché lab on argument retrieval hosted at CLEF 2022. With the goal of supporting and promoting the research and development of new technologies for argument mining and argument analysis, we …


PANGUBOT: Efficient Generative Dialogue Pre-training from Pre-trained Language Model

F Mi, Y Li, Y Zeng, J Zhou, Y Wang, C Xu, L Shang… - arXiv preprint arXiv …, 2022

In this paper, we introduce PANGUBOT, a Chinese pre-trained open-domain dialogue generation model based on a large pre-trained language model (PLM) PANGU-alpha (Zeng et al.,2021). Different from other pre-trained dialogue models …


Papago's Submission for the WMT21 Quality Estimation Shared Task

S Lim, H Kim, H Kim - Proceedings of the Sixth Conference on Machine …, 2021

This paper describes Papago submission to the WMT 2021 Quality Estimation Task 1: Sentence-level Direct Assessment. Our multilingual Quality Estimation system explores the combination of Pretrained Language Models and Multi-task Learning …


Paragraph-based Transformer Pre-training for Multi-Sentence Inference

L Di Liello, S Garg, L Soldaini, A Moschitti - arXiv preprint arXiv:2205.01228, 2022

Inference tasks such as answer sentence selection (AS2) or fact verification are typically solved by fine-tuning transformer-based models as individual sentence-pair classifiers. Recent studies show that these tasks benefit from modeling …


Parameter-efficient feature-based transfer for paraphrase identification

X Liu, R Rzepka, K Araki - Natural Language Engineering, 2022

There are many types of approaches for Paraphrase Identification (PI), an NLP task of determining whether a sentence pair has equivalent semantics. Traditional approaches mainly consist of unsupervised learning and feature engineering, which …


Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval with Adapters

E Yang, S Nair, D Lawrie, J Mayfield, DW Oard - arXiv preprint arXiv:2212.10448, 2022

A popular approach to creating a zero-shot cross-language retrieval model is to substitute a monolingual pretrained language model in the retrieval model with a multilingual pretrained language model such as Multilingual BERT. This multilingual …


Parliamentary representative behaviour in times of crisis: a comparative perspective with a focus on Greece and the Eurozone crisis

Much scholarly discussion has been devoted to the impact of the 2009 Eurozone economic Crisis on the functioning of European democracies. Especially for those countries that bailout agreements had to be implemented in, the literature has …


Parquet Containers

Archives preserve content and support various types of analysis, study, and use. Archiving the Web and social media (eg, Twitter) content involves use of representations like Web ARChive (WARC) and JavaScript Object Notation (JSON) …


Partitioning Trillion-Edge Graphs

GM Slota, K Devine, K Madduri, S Rajamanickam - Massive Graph Analytics

Graph partitioning is a key preprocessing step for distributed-memory graph analytics and scientific computations on meshes. This chapter introduces XtraPuLP, a new distributed-memory graph partitioner designed to process trillion-edge graphs …


Pełen tekst

… For the language-individual model, we use FastText word em-beddings [11] which are trained on Common Crawl and Wikipedia dataset. For the language-consistent model, we use pre-trained mul-tilingual embeddings which are released by FastText …


Perplexed by Quality: A Perplexity-based Method for Adult and Harmful Content Detection in Multilingual Heterogeneous Web Data

T Jansen, Y Tong, V Zevallos, PO Suarez - arXiv preprint arXiv:2212.10440, 2022

As demand for large corpora increases with the size of current state-of-the-art language models, using web data as the main part of the pre-training corpus for these models has become a ubiquitous practice. This, in turn, has introduced an …


Personalized filled-pause generation with group-wise prediction models

Y Matsunaga, T Saeki, S Takamichi, H Saruwatari - arXiv preprint arXiv:2203.09961, 2022

In this paper, we propose a method to generate personalized filled pauses (FPs) with group-wise prediction models. Compared with fluent text generation, disfluent text generation has not been widely explored. To generate more human-like texts …


Perspectives on the Cognitive Unconscious

My interest in nonconscious processes goes back to my youth when I imagined that Sigmund Freud had discovered all that there was to be known about the human mind. But interest in the nonconscious and the role it plays in mind and behavior turned …


PETCI: A Parallel English Translation Dataset of Chinese Idioms

K Tang - arXiv preprint arXiv:2202.09509, 2022

… RoBERTa, for example, has part of its pre-training data from Reddit and CommonCrawl (Liu … Similarly, we expect the Wikipedia dataset to have a lower PoI than CommonCrawl. The … use the cased 300-dimensional GloVe vectors pre-trained on …


Phishing Detection Using URL-based XAI Techniques

PRG Hernandes, CP Floret, KFC De Almeida… - 2021 IEEE Symposium …, 2021

… As a result, 5,000 phishing web pages were selected based on URLs from PhishTank 1, OpenPhish 2, and another 5,000 legitimate pages based on URLs of Alexa 3 and the archive of CommonCrawl 4. Subsequently, this dataset was …


Phishing URL Detection: A novel hybrid Approach using Long Short-Term Memory and Gated Recurrent Units

BAS Dilhara - International Journal of Computer Applications

Phishing is one of the oldest types of cyber-attack, which mostly comes in the form of camouflaged URLs to delude the users to disclose their personal information for malevolent purposes of the attacker. It is one of the easiest ways of inducing people …


Phishing URL Identification Using Machine Learning, Ensemble Learning and Deep Learning Techniques

K Laxmi Prasanna, KV Pradeepthi, A Saxena - Smart Intelligent Computing and …, 2022

Demand for online applications and services is increasing day by day, whereas the security lapses in them is also becoming prominent. One of the major online threats, phishing, is consistently used to dupe users for retrieving sensitive information like …


Phishing website detection: How effective are deep learning‐based models and hyperparameter optimization?

M Almousa, T Zhang, A Sarrafzadeh, M Anwar - Security and Privacy, 2022

… The phishing webpages have been collected from PhishTank36 and OpenPhish,37 while the sources of legitimate webpages were Alexa38 and Common Crawl.The second dataset was obtained from Reference 40 that we refer to as Kumar-dataset …


Phonetic, Semantic, and Articulatory Features in Assamese-Bengali Cognate Detection

In this paper, we propose a method to detect if words in two similar languages, Assamese and Bengali, are cognates. We mix phonetic, semantic, and articulatory features and use the cognate detection task to analyze the relative informational …


Phrase-Based Affordance Detection via Cyclic Bilateral Interaction

L Lu, W Zhai, H Luo, Y Kang, Y Cao - arXiv preprint arXiv:2202.12076, 2022

… Meanwhile, for linguistic feature extraction, we first adopt the GloVe word embeddings [42] pre-trained on Common Crawl (840B tokens) to initialize the parameters of embedding layers then a LSTM is employed as the language feature …


Plansformer: Generating Symbolic Plans using Transformers

V Pallagani, B Muppasani, K Murugesan, F Rossi… - arXiv preprint arXiv …, 2022

… Large Language Models (LLM) such as BERT [16], RoBERTa [17], and GPT3 [3] are pre-trained on extensive unstructured knowledge from public data such as Wikipedia, Bookcorpus, and Commoncrawl, have shown impressive results in …


PLAtE: A Large-scale Dataset for List Page Web Extraction

A San, J Bakus, C Lockard, D Ciemiewicz, Y Ji, S Atluri… - arXiv preprint arXiv …, 2022

… We construct PLAtE by collecting list pages from Common Crawl, then annotating them on Mechanical Turk. Quantitative and qualitative analyses are performed to demonstrate PLAtE has high-quality annotations. We establish strong baseline …


Political Event Coding as Text to Text Sequence Generation

We report on the current status of an effort to produce political event data from unstructured text via a Transformer language model. Compelled by the current lack of publicly available and up-to-date event coding software, we seek to train a model …


Politics and Virality in the Time of Twitter: A Large-Scale Cross-Party Sentiment Analysis in Greece, Spain and United Kingdom

D Antypas, A Preece, JC Collados - arXiv preprint arXiv:2202.00396, 2022

Social media has become extremely influential when it comes to policy making in modern societies especially in the western world (eg, 48% of Europeans use social media every day or almost every day). Platforms such as Twitter allow users to follow …


Politics as Usual? Antecedents of Radical-Right Frames in US Electoral Discourse

… This extension of the original BERT model relies on the same general architecture but expands the training data to 63 million English news articles from CommonCrawl, 38 gigabytes of textual data from the OpenWebText corpus, and a 31-gigabyte …


Politics as Usual? Measuring Populism, Nationalism, and Authoritarianism in US Presidential Campaigns (1952–2020) with Deep Neural Language Models

B Bonikowski, Y Luo, O Stuhler - Sociological Methods & Research, 2022

… but expands the training data to 63 million English news articles from CommonCrawl, 38 gigabytes of textual data from the OpenWebText … corpus, and a 31-gigabyte subset of CommonCrawl data known as Stories. In addition, the …


POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection

Y Liu, XF Zhang, D Wegsman, N Beauchamp, L Wang - arXiv preprint arXiv …, 2022

Ideology is at the core of political science research. Yet, there still does not exist general-purpose tools to characterize and predict ideology across different genres of text. To this end, we study Pretrained Language Models using novel ideology-driven …


Post-hoc analysis of Arabic transformer models

A Abdelali, N Durrani, F Dalvi, H Sajjad - arXiv preprint arXiv:2210.09990, 2022

Arabic is a Semitic language which is widely spoken with many dialects. Given the success of pre-trained language models, many transformer models trained on Arabic and its dialects have surfaced. While there have been an extrinsic evaluation …


Poster: PhishLex: A Proactive Zero-Day Phishing Defence Mechanism using URL Lexical Features

Google reports that 68% of all Phishing URLs that are blocked by them are zero-day phishing attacks that remain undetected using traditional blacklist-based approaches. Machine learning-based (ML) techniques can improve the accuracy of detecting zero-day …


Posts on central websites need less originality to be noticed

M Coscia, C Vandeweerdt - Scientific Reports, 2022

… We only kept domains that are also found in the Common Crawl data. This reduced the data to 3.8M posts. … We downloaded the domain graph from the Common Crawl for the November 2019–January 2020 period. Since from the Reddit …


Pre-Trained Language Models and Their Applications

H Wang, J Li, H Wu, E Hovy, Y Sun - Engineering, 2022

Pre-trained language models have achieved striking success in natural language processing (NLP), leading to a paradigm shift from supervised learning to pre-training followed by fine-tuning. The NLP community has witnessed a surge of research …


Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?

ESA Lee, S Thillainathan, S Nayak, S Ranathunga… - arXiv preprint arXiv …, 2022

… mined one like from Common Crawl? Comparing mBART trained on Common Crawl versus domain-specific data, we see that for several … both in and not in mBART, 10k high-quality in-domain sentences leads to better performance than …


Pre-trained transformers: an empirical comparison

S Casola, I Lauriola, A Lavelli - Machine Learning with Applications, 2022

Pre-trained transformers have rapidly become very popular in the Natural Language Processing (NLP) community, surpassing the previous state of the art in a wide variety of tasks. While their effectiveness is indisputable, these methods are …


Pre-training large NLP-models by utilizing low-resource NLP-pipelines

The dominating approach to Natural Language Processing problems is by pre-training models on large amounts of unsupervised text and fine-tuning them for use on downstream tasks. Over the years, this amount of unsupervised text has grown …


Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection

L Di Liello, S Garg, L Soldaini, A Moschitti - arXiv preprint arXiv:2205.10455, 2022

An important task for designing QA systems is answer sentence selection (AS2): selecting the sentence containing (or constituting) the answer to a question from a set of retrieved relevant documents. In this paper, we propose three novel sentence-level …


Predicting Embedding Reliability in Low-Resource Settings Using Corpus Similarity Measures

J Dunn, H Li, D Sastre - arXiv preprint arXiv:2206.04330, 2022

This paper simulates a low-resource setting across 17 languages in order to evaluate embedding similarity, stability, and reliability under different conditions. The goal is to use corpus similarity measures before training to predict properties of …


Predicting Fine-Tuning Performance with Probing

Z Zhu, S Shahtalebi, F Rudzicz - Challenges {\&, 2022

… For example, after pre-training a deep neural model for 1000 steps on Wikipedia, is it more beneficial to continue training on Wikipedia or proceed to Common Crawl? The probing results could help answer questions of this kind, which we leave to …


Preparing a Corpus and a Question Answering System for Slovene

M Zupanic, M Zirkelbach, U Šmajdek, M Jazbinšek

Lack of proper training data is one of the key issues when developing natural language processing models based on less-resourced languages, such as Slovene. In this paper we discuss machine translation as a solution to this issue, with the …


Pretrained Language Models

… It was trained on 570gb of text (500b tokens) from common crawl, their previous data set, and some other ‘high quality’ data sets. When prompted with only a few examples (and sometimes with none at all) it outperformed SOTA on a wide variety …


Pretraining without Wordpieces: Learning Over a Vocabulary of Millions of Words

Z Feng, D Tang, C Zhou, J Liao, S Wu, X Feng, B Qin… - arXiv preprint arXiv …, 2022

… Glove embedding is pretrained on Common Crawl dataset with 42B tokens 2 with 300 dimension. Given a sequence of words, we first get the corresponding 300-dimensional representation for each word through Glove Embedding. Then, we add a linear layer …


Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy

D Ippolito, F Tramèr, M Nasr, C Zhang, M Jagielski… - arXiv preprint arXiv …, 2022

Studying data memorization in neural language models helps us understand the risks (eg, to privacy or copyright) associated with models regurgitating training data, and aids in the evaluation of potential countermeasures. Many prior works -- and …


Prior knowledge guided weakly supervised object detection and semantic segmentation

State-of-the-art recognition models in computer vision are trained using annotated training data. Collecting manual annotation for images is a time-consuming and tedious task. Annotation time and difficulty also change across computer vision tasks …


Privacy conflict analysis in web interaction models

User privacy has become an important topic with strong implications for the manner by which software systems are designed and used. However, it is not a straightforward consideration on how the instrumentation of data processing …


Privacy leakage analysis of text representations for Natural Language Processing

Natural language processing (NLP) offers powerful tools, such as text suggestions to help users be more efficient or classifiers that categorize documents by their content. These tools are usually powered by machine learning (ML) models that are trained …


Privacy Policies of IoT Devices: Collection and Analysis

M Kuznetsov, E Novikova, I Kotenko, E Doynikova - Sensors, 2022

Currently, personal data collection and processing are widely used while providing digital services within mobile sensing networks for their operation, personalization, and improvement. Personal data are any data that identifiably describe a person …


Proactive Interference-aware Resource Management in Deep Learning Training Cluster

Deep Learning (DL) applications are growing at an unprecedented rate across many domains, ranging from weather prediction, map navigation to medical imaging. However, training these deep learning models in large-scale compute clusters face …


Probing for Dutch Relative Pronoun Choice

G Bouma - Computational Linguistics in the Netherlands Journal, 2021

We propose a linguistically motivated version of the relative pronoun probing task for Dutch (where a model has to predict whether a masked token is either die or dat), collect realistic data for it using a parsed corpus, and probe the performance of four …


Probing language identity encoded in pre-trained multilingual models: a typological view

J Zheng, Y Liu - PeerJ Computer Science, 2022

… It is trained on CommonCrawl corpora with 12 layers and 768 hidden states. The vocabulary size is 250k, The tool for tokenization is free, namely, the Sentence Piece (Kudo & Richardson, 2018). The only task is the dynamic mask language model. …


Probing Pre-Trained Language Models for Cross-Cultural Differences in Values

A Arora, LA Kaffee, I Augenstein - arXiv preprint arXiv:2203.13722, 2022

… 2019) and is trained with an MLM objective on 2.5 TB of filtered CommonCrawl corpus data in 100 languages. It shows strong multilingual performance across a range of benchmarks and is commonly used for extracting multilingual sentence encodings. …


Proceedings of the 13th Conference on" Data analysis methods for software systems

J Bernatavičienė - Vilnius University Proceedings, 2022

DAMSS-2022 is the 13th International Conference on Data Analysis Methods for Software Systems, held in Druskininkai, Lithuania. Every year at the same place and time. The exception was in 2020, when the world was gripped by the Covid-19 …


Project European Language Equality (ELE) Grant agreement no. LC-01641480–101018166 ELE Coordinator Prof. Dr. Andy Way (DCU) Co-coordinator Prof. Dr …

G Backfried, M Skowron, E Navas, A Bērziņš… - 2022

D2. 14 provides an overview and describes the state of the art and developments within the field of Speech Technologies (ST). This field is interpreted to comprise technologies aimed at the processing and production of the human voice, both, from …


Prompt-GAN–Customisable hate speech and extremist datasets via radicalised neural language models

Online hate speech and violent extremism knows no borders, no political boundaries, no remorse. Researchers face an uphill battle to collect hate speech data in volumes and topical diversity suitable for training state-of-the art content-moderation systems …


Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models

M Xia, M Artetxe, J Du, D Chen, V Stoyanov - arXiv preprint arXiv:2205.15223, 2022

Pre-trained masked language models successfully perform few-shot learning by formulating downstream tasks as text infilling. However, as a strong alternative in full-shot settings, discriminative pre-trained models like ELECTRA do not fit into the paradigm …


Promt systems for wmt21 terminology translation task

A Molchanov, V Kovalenko, F Bykov - Proceedings of the Sixth Conference on …, 2021

… We first build an English-Russian system using all parallel data (except commoncrawl which we believe to be of bad quality; basic filtering is applied) including the Edinborough corpus of Russian news translated into English and …


Prospects for Dutch Emotion Detection: Insights from the new EmotioNL Dataset

L De Bruyne, O De Clercq, V Hoste - Computational Linguistics in the Netherlands …, 2021

Although emotion detection has become a crucial research direction in NLP, the main focus is on English resources and data. The main obstacles for more specialized emotion detection are the lack of annotated data in smaller languages …


Protecting Systems From Exploits Using Language-Theoretic Security

Any computer program processing input from the user or network must validate the input. Input-handling vulnerabilities occur in programs when the software component responsible for filtering malicious input---the parser---does not perform …


PsychBERT: A Mental Health Language Model for Social Media Mental Health Behavioral Analysis

V Vajre, M Naylor, U Kamath, A Shehu - 2021 IEEE International Conference on …, 2021

Mental health behaviors are now recognized as primary factors contributing to suicide. This paper puts forth a novel mental health language model to address mental health and makes several contributions. First, it proposes a taxonomy and …


Punctuation Restoration in Spanish Customer Support Transcripts using Transfer Learning

X Zhu, S Gardiner, D Rossouw, T Roldán… - arXiv preprint arXiv …, 2022

Automatic Speech Recognition (ASR) systems typically produce unpunctuated transcripts that have poor readability. In addition, building a punctuation restoration system is challenging for low-resource languages, especially for domain-specific …


Qasar: Self-Supervised Learning Framework for Extractive Question Answering

H Assem, R Sarkar, S Dutta - 2021 IEEE International Conference on Big Data (Big …, 2021

… Pretrained transformer based language models like BERT [8] are trained initially on large unlabelled data from common crawl corpus and Wikipedia dumps. With the creation of large labelled open-domain QA datasets like WikiQA [9], SQuAD [10] …


QianXun: A Novel Enterprise File Search Framework

C Si, P Qi, H Xue, Y Sun - 2022 IEEE 25th International Conference on Computer …, 2022

… Subsequently, Google proposed mT5 [7], a multilingual variant of T5 that is pre-trained on a new Common Crawl-based dataset covering 101 languages, which offers the possibility of processing Chinese texts. Then, based on the mT5, Su Jianlin trained …


QMLEx: Data Driven Digital Transformation in Marketing Analytics

A Geronazzo, M Ziegler - 2021 IEEE International Conference on Big Data (Big …, 2021

This paper presents a data driven approach to replace expert driven business processes. The QMLEx methodology combines NLP algorithms to improve data quality, ML techniques to perform the task and exploits external data sources to …


Qualitative Analysis of Text Summarization Techniques and Its Applications in Health Domain

D Yadav, N Lalit, R Kaushik, Y Singh, AK Yadav… - Computational Intelligence …, 2022

For the better utilization of the enormous amount of data available to us on the Internet and in different archives, summarization is a valuable method. Manual summarization by experts is an almost impossible and time-consuming activity …


Qualitative Measures for Ad hoc Table Retrieval

M Khodabakhsh, E Bagheri - Information Sciences, 2022

The focus of our work is the ad hoc table retrieval task, which aims to rank a list of structured tabular objects in response to a user query. Given the importance of this task, various methods have already been proposed in the literature that focus on …


Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP

T Nguyen, G Ilharco, M Wortsman, S Oh, L Schmidt - arXiv preprint arXiv:2208.05516, 2022

Web-crawled datasets have enabled remarkable generalization capabilities in recent image-text models such as CLIP (Contrastive Language-Image pre-training) or Flamingo, but little is known about the dataset creation processes. In this work, we …


Quantifying Memorization Across Neural Language Models

N Carlini, D Ippolito, M Jagielski, K Lee, F Tramer… - arXiv preprint arXiv …, 2022

… These models were trained on C4, a cleaned and filtered version of the English web pages from the Common Crawl, which totals 806 GB in size. At 11 billion parameters, the largest T5 model is the largest publicly available masked language …


Quantifying the Polarity of Noun Phrases

While sentiment analysis is a popular NLP subfield, not every area within gets the same amount of attention. For example, classifying the sentiment of texts seems to be quite in demand, however, rarely does it also include determining how positive or …


Query Refinement Prompts for Closed-Book Long-Form Question Answering

RK Amplayo, K Webster, M Collins, D Das, S Narayan - arXiv preprint arXiv …, 2022

Large language models (LLMs) have been shown to perform well in answering questions and in producing long-form texts, both in few-shot closed-book settings. While the former can be validated using well-known evaluation metrics, the latter is …


Query-Driven Graph-Based User Recommender System

Current Social Networking Systems (SNS) such as YouTube are creator-driven systems in which creators create content and users search among available content to find what they want. However, queries from users can be time-sensitive, such as …


QueryForm: A Simple Zero-shot Form Entity Query Framework

Z Wang, Z Zhang, J Devlin, CY Lee, G Su, H Zhang… - arXiv preprint arXiv …, 2022

Zero-shot transfer learning for document understanding is a crucial yet under-investigated scenario to help reduce the high cost involved in annotating document entities. We present a novel query-based framework, QueryForm, that extracts entity values from …


Question Answering in Czech via Machine Translation and Cross-lingual Transfer

Reading comprehension and question answering are computer science disciplines in the field of natural language processing and information retrieval. Reading comprehension is the ability of the model to read text, process it and understand its …


Question Generation and Answering in the Electrical Power System Components Domain

… This is English portion of Common Crawl data-set [Nag]. It has 63 M news articles crawled between … It is a subset of CommonCrawl data after filtering to match the story-like style of Winograd … To generate question corpus questions were …


Quick and (Maybe Not So) Easy Detection of Anorexia in Social Media

H Amini, E Mohammadi, L Kosseim - Early Detection of Mental Health Disorders by …, 2022

This book chapter overviews our participation to the first sub-task of eRisk 2019 shared task, which was focused on the task of early detection of anorexia in Reddit posts. Our approach was to utilize several attention-based neural sub-models to …


Quick and (Maybe Not So) Easy Detection of Anorexia in Social Media: To Explainability and Beyond


RAPIDS cuGraph

A Fender, B Rees, J Eaton - Massive Graph Analytics

… -scale PageRank computations using the Common Crawl dataset [5].The Common Crawl dataset captures link relationships between web … Our system delivered much faster performance for the Common Crawl dataset, but the largest …


Readability of English, German, and Russian Disease-Related Wikipedia Pages: Automated Computational Analysis

J Gordejeva, R Zowalla, M Pobiruchin, M Wiesner - Journal of Medical Internet …, 2022

Background Wikipedia is a popular encyclopedia for healthand disease-related information in which patients seek advice and guidance on the web. Yet, Wikipedia articles can be unsuitable as patient education materials, as investigated in previous …


Reading Comprehension

RS Roy, A Anand - Question Answering for the Curated Web, 2022

… Corpora of curated text (like Wikipedia), large volumes of uncurated social-media posts, and massive Web document collection projects such as Common Crawl,1 are used to train these models. The key idea is that rather than going through the trouble …


RECOGNITION THROUGH SPONTANEOUS SPEECH

Alzheimer’s disease (AD) is a chronic neurodegenerative disease, without any known treatment. This disease progressively destroys brain structures, such as the hippocampus and entorhinal cortex, due to the accumulation of pathological forms of …


Reducing Disambiguation Biases in NMT by Leveraging Explicit Word Sense Information

N Campolungo, T Pasini, D Emelin, R Navigli - … of the 2022 Conference of the North …, 2022

Recent studies have shed some light on a common pitfall of Neural Machine Translation (NMT) models, stemming from their struggle to disambiguate polysemous words without lapsing into their most frequently occurring senses in the …


Reducing Health Misinformation in Search Results

… But topics in the 2020 track don’t have relevant documents in C4.en.noclean because it was extracted from the April 2019 snapshot3 of the Common Crawl corpus. Therefore, we performed our experiments on the data from the 2019 track …


Reducing the labeling effort for entity resolution using distant supervision and active learning

… during the period 2012 to 2020 by extracting structured data from the Common Crawl web corpus.Along with this analysis, we are the first to … [C3] WDC Product Corpus We exploit semantically annotated productrelated data extracted from the …


Refining Low-Resource Unsupervised Translation by Language Disentanglement of Multilingual Model

XP Nguyen, S Joty, W Kui, AT Aw - arXiv preprint arXiv:2205.15544, 2022

… For each group, we use monolingual corpora from the relevant languages of the Common Crawl dataset, which contains data from totally 25 languages [21]. We limit the amount of monolingual data per language to up to 100M sentences. Similarly to …


Refining Word Embeddings with Sentiment Information for Sentiment Analysis

M Kasri, M Birjali, M Nabil, A Beni-Hssane, A El-Ansari… - Journal of ICT …, 2022

Natural Language Processing problems generally require the use of pre-trained distributed word representations to be solved with deep learning models. However, distributed representations usually rely on contextual information which prevents …


Register identification from the unrestricted open Web using the Corpus of Online Registers of English

V Laippala, S Rönnqvist, M Oinonen, AJ Kyröläinen… - Language Resources and …, 2022

This article examines the automatic identification of Web registers, that is, text varieties such as news articles and reviews. Most studies have focused on corpora restricted to include only preselected classes with well-defined characteristics …


Register variation remains stable across 60 languages

H Li, J Dunn, A Nini - Corpus Linguistics and Linguistic Theory, 2022

This paper measures the stability of cross-linguistic register variation. A register is a variety of a language that is associated with extra-linguistic context. The relationship between a register and its context is functional: the linguistic features that make up a …


Regressing Word and Sentence Embeddings for Low-Resource Neural Machine Translation

IJ Unanue, EZ Borzeshi, M Piccardi - IEEE Transactions on Artificial Intelligence, 2022

… The training set contains over 5M sentence pairs collected from the Europarl, CommonCrawl and Newscommentary parallel corpora. As validation and test sets, we have used the newstest2017 and the newstest2018 datasets, respectively. This …


Regularization-based Pruning of Irrelevant Weights in Deep Neural Architectures

B Giovanni, R Matteo, C Rossella - APPLIED INTELLIGENCE, 2022

G Bonetta, M Ribero, R Cancelliere - arXiv preprint arXiv:2204.04977, 2022

Deep neural networks exploiting millions of parameters are nowadays the norm in deep learning applications. This is a potential issue because of the great amount of computational resources needed for training, and of the possible loss of …


Related Questions Retrieval Model in Stack Overflow based on Semantic Matching

Z Qin, Y Wu, J Pei, J Lu, S Huang, L Liu - … IEEE 46th Annual Computers, Software, and …, 2022

… The GloVe-840B [23] is pre-trained on the Common Crawl corpus with many texts that are irrelevant to the software engineering domain. The SE (15G) [9] is pre-trained on a 15GB corpus collected from Stack Overflow data dumps. …


Relational Embeddings for Language Independent Stance Detection

JF de Landa, R Agerri - arXiv preprint arXiv:2210.05715, 2022

The large majority of the research performed on stance detection has been focused on developing more or less sophisticated text classification systems, even when many benchmarks are based on social network data such as Twitter. This paper …


Report 2 on ASR Systems

1 Executive Summary In this deliverable we describe the final automatic speech recognition (ASR) systems that we created for the ELITR speech translation system. In Section 2 we describe our work. In the subsections 2.1 and 2.2 we address tasks …


Representations of emotion concepts: Comparison across pairwise, appraisal feature-based, and word embedding-based similarity spaces

M Kwon, T Wager, J Phillips - Proceedings of the Annual Meeting of the Cognitive …, 2022

A question that has long interested cognitive scientists is how to best represent the different emotions we experience and attribute to others. For example, constructionist and appraisal theories propose that differences between emotions …


Reproducible scaling laws for contrastive language-image learning

M Cherti, R Beaumont, R Wightman, M Wortsman… - arXiv preprint arXiv …, 2022

… More recently, larger image-text datasets have been collected from Common Crawl [1]. The resulting datasets, LAION-400M [66] and … Both LAION-400M [66] and LAION-5B [65] are open, public image-text datasets that were composed by …


Reproduction and Replication of an Adversarial Stylometry Experiment

H Wang, P Juola, A Riddell - arXiv preprint arXiv:2208.07395, 2022

Maintaining anonymity while communicating using natural language remains a challenge. Standard authorship attribution techniques that analyze candidate authors' writing styles achieve uncomfortably high accuracy even when the number …


Research Article COVID-19 Outbreak Forecasting Based on Vaccine Rates and Tweets Classification

Y Didi, A Walha, MB Halima, A Wali - 2022

Copyright© 2022 Y. Didi et al. is is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. e …


Research Article Qualitative Analysis of Text Summarization Techniques and Its Applications in Health Domain

AK Yadav, KV Bhadane, A Kumar, B Khan - 2022

Summarizing textual information requires understanding and analyzing the linguistic, conceptual, and semantic attributes of the given information. In addition, a summary generated should succeed in incorporating the essential details and the main ideas …


Research Background

R Richner - Auto-Grader-Auto-Grading Free Text Answers, 2022

Research Background Page 1 3 Research Background Firstly, this chapter will introduce the technological background needed to understand how a state-of-the-art auto-grader may look and secondly elaborate on related work in the field of …


Rethinking Data Governance: A Labor-Oriented Approach

… Prominent examples include Flickr photos [12], Wikipedia articles [14], and the Common Crawl dataset consisting of publicly available webpages [11]. In many of such cases, users produce data without being aware of its value and potential …


Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

S Min, X Lyu, A Holtzman, M Artetxe, M Lewis… - arXiv preprint arXiv …, 2022

Large language models (LMs) are able to in-context learn--perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of …


Revisiting CCNet for Quality Measurements in Galician

JE Ortega, I de-Dios-Flores, JR Pichel, P Gamallo - International Conference on …, 2022

P Gamallo - … Processing of the Portuguese Language: 15th …

… In this article, we present our findings on reproducing the introduction of the common crawl corpus by Facebook, known as the CCNet … as the Common Crawl 2 corpus based on a “snapshot” of the web at a given time – the work on CCNet is …


Revisiting DocRED--Addressing the Overlooked False Negative Problem in Relation Extraction

Q Tan, L Xu, L Bing, HT Ng - arXiv preprint arXiv:2205.12696, 2022

The DocRED dataset is one of the most popular and widely used benchmarks for document-level relation extraction (RE). It adopts a recommend-revise annotation scheme so as to have a large-scale annotated dataset. However, we find that the …


Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understanding

A Ghaddar, Y Wu, S Bagga, A Rashid, K Bibi… - arXiv preprint arXiv …, 2022

… We collect pre-training data from the following sources: common crawl (CC), news (NEWS, ELKHAIR), and Wikipedia (WIKI).Recent … Common Crawl (CC): We used 10 shards of Common Crawl8 data from March to December 2020. After …


RigoBERTa: A State-of-the-Art Language Model For Spanish

AV Serrano, GG Subies, HM Zamorano, NA Garcia… - arXiv preprint arXiv …, 2022

… OSCAR [28] [29] is a very large multilingual corpus, obtained by language classification and filtering of the CommonCrawl7. It has a portion of … For example, javascript code, automatic writings, poor automatic translations or malformed …


RobBERT-2022: Updating a Dutch Language Model to Account for Evolving Language Use

P Delobelle, T Winters, B Berendt - arXiv preprint arXiv:2211.08192, 2022

… 2019) is based on automatically language-classified Common Crawl documents.The corpus has a release with data up until January 2022, which in contrast with previous releases is now a full documents corpus instead of the previous pre-shuffled …


Robust Domain Adaptation for Pre-trained Multilingual Neural Machine Translation Models

M Grosso, P Ratnamogan, A Mathey, W Vanhuffel… - arXiv preprint arXiv …, 2022

Recent literature has demonstrated the potential of multilingual Neural Machine Translation (mNMT) models. However, the most efficient models are not well suited to specialized industries. In these cases, internal data is scarce and expensive to …


Robust Knowledge Acquisition in Answering Information-seeking Questions At Scale

Answering information-seeking question involves retrieving relevant documents from a massive haystack of unstructured text corpora. This dissertation aims at building question answering (QA) systems that can be deployed in the wild where …


Robust Phishing Detection Against Adversaries

S Al-Ahmadi - Journal: WSEAS TRANSACTIONS ON COMPUTER …, 2022

Phishing websites have grown more recently than ever, and they become more intelligent, even against well-designed phishing detection techniques. Formerly, we have proposed in the literature a state-of-the-art URL-exclusive phishing detection …


Robustness Analysis of Grover for Machine-Generated News Detection

Advancements in Natural Language Generation have raised concerns on its potential misuse for deep fake news. Grover is a model for both generation and detection of neural fake news. While its performance on automatically discriminating …


Romantic-Computing

E Horishny - arXiv preprint arXiv:2206.11864, 2022

In this paper we compare various text generation models' ability to write poetry in the style of early English Romanticism. These models include: Character-Level Recurrent Neural Networks with Long Short-Term Memory, Hugging Face's GPT-2 …


Russian Web Tables: A Public Corpus of Web Tables for Russian Language Based on Wikipedia

P Fedorov, A Mironov, G Chernishev - arXiv preprint arXiv:2210.06353, 2022

… Therefore, relying on the Common Crawl dataset would require a multimachine environment. For example, in case of the WDC Web Table Corpus, the … Therefore, it is prohibitevely expensive for low-budged projects or individual students to use the …


SALTED: A Framework for SAlient Long-Tail Translation Error Detection

V Raunak, M Post, A Menezes - arXiv preprint arXiv:2205.09988, 2022

Traditional machine translation (MT) metrics provide an average measure of translation quality that is insensitive to the long tail of behavioral problems in MT. Examples include translation of numbers, physical units, dropped content and …


Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation

TK Lam, S Schamoni, S Riezler - arXiv preprint arXiv:2203.08757, 2022

End-to-end speech translation relies on data that pair source-language speech inputs with corresponding translations into a target language. Such data are notoriously scarce, making synthetic data augmentation by back-translation or …


Samsung R&D Institute participation in WMT 2022 General MT Task

A Dobrowolski, M Klimaszewski, A Mysliwy…

… CommonCrawl, we used only lines/paragraphs containing three or more sentences, which we split into single sentences. News-CommonCrawl … crawl and 100k sentences from CommonCrawl. Using this model, we selected sentences classified …


Sarcasm Detection Framework Using Emotion and Sentiment Features

O Vitman, Y Kostiuk, G Sidorov, A Gelbukh - arXiv preprint arXiv:2211.13014, 2022

… We used Glove Common Crawl4 pretrained vectors (42 billion words version). For words with no pre-trained vectors, we checked their stemmed versions. If no pre-trained embedding were found, random vector was initialized. …


Sarcasm detection using deep learning and ensemble learning

P Goel, R Jain, A Nayyar, S Singhal, M Srivastava - Multimedia Tools and …, 2022

Across the globe, there is a noticeable upward trend of incorporating sarcasm in everyday life. This trend can be easily attributed to the frequent use of sarcasm in everyday life, but more specifically to social media and the Internet. This study aims …


Scaling Laws and Interpretability of Learning from Repeated Data

D Hernandez, T Brown, T Conerly, N DasSarma… - arXiv preprint arXiv …, 2022

Recent large language models have been trained on vast datasets, but also often on repeated data, either intentionally for the purpose of upweighting higher quality data, or unintentionally because data deduplication is not perfect and the model is …


Scaling Laws Beyond Backpropagation

MJ Filipovich, A Cappelli, D Hesslow, J Launay - arXiv preprint arXiv:2210.14593, 2022

… The models are trained on 30B tokens of English CommonCrawl data filtered with CCNet [41], with hyper-parameters adapted from GPT-3 [29], but with a context size of 512 for increased training speed. We chose different optimization hyper-parameters …


SDW-ASL: A Dynamic System to Generate Large Scale Dataset for Continuous American Sign Language

Y Jiang - arXiv preprint arXiv:2210.06791, 2022

… For example, the GPT-3 model with 175 billion parameters was trained using the ever-growing Common Crawl dataset with close to 1 trillion words. The largest dataset for ASL is dwarfed in comparison — so far it is only around 10-thousand …


Second place solution of Amazon KDD Cup 2022: ESCI Challenge for Improving Product Search

How to improve the search experience is a hot topic that has been studied for decades in both academia and industry. KDD Cup 2022 ESCI Challenge organized by Amazon provides a large dataset with four categories of ESCI to improve …


Securing the Next Generation Web

… We evaluate Black Ostrich on a set of 8 821 unique validation patterns gathered from over 21 667 978 forms from a combination of the July 2021 Common Crawl and Tranco top 100K. For these forms and reconstructions of input elements …


Security and Privacy Challenges for Intelligent Internet of Things Devices 2022 TADW: Traceable and Antidetection Dynamic Watermarking of Deep Neural Networks

J Dong, H Wang, Z He, J Niu, X Zhu, G Wu - Security and Communication Networks, 2022

Deep neural networks (DNN) with incomparably advanced performance have been extensively applied in diverse fields (eg, image recognition, natural language processing, and speech recognition). Training a high-performance DNN model …


Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledge

D Merkx, SL Frank, M Ernestus - arXiv preprint arXiv:2202.10292, 2022

Distributional semantic models capture word-level meaning that is useful in many natural language processing tasks and have even been shown to capture cognitive aspects of word meaning. The majority of these models are purely text based, even …


SEHC: A Benchmark Setup to Identify Online Hate Speech in English

S Ghosh, A Ekbal, P Bhattacharyya, T Saha, A Kumar… - IEEE Transactions on …, 2022

… The embeddings are fetched from GloVe (300-dimension) pre-trained word embedding, which was trained on the Common Crawl corpus (840 billion tokens). We pad input arrays with zeros to make input sequences the same length. On our …


Self-admitted technical debt classification using natural language processing word embeddings

AF Sabbah, AA Hanani - International Journal of Electrical and Computer …, 2023

Recent studies show that it is possible to detect technical dept automatically from source code comments intentionally created by developers, a phenomenon known as self-admitted technical debt. This study proposes a system by which a comment …


SEMANTIC ACCOUNTS OF RISK PERCEPTION


Semantic Annotation of Numerical Data in Web Tables

A large portion of quantitative information about entities mentioned in Web pages is expressed as Web tables, and these tables often lack proper schema and annotation, which introduces challenges for the purpose of querying and further analysis. In this …


Semantic Comparisons for Natural Language Processing Applications

For social scientists and other data practitioners, the abundance of available digital text data is a rich potential source for understanding social phenomena. As a result, practitioners have increasingly used text analysis methods on relevant corpora to …


Semantic Cross Attention for Few-shot Learning

B Xiao, CL Liu, WH Hsaio - arXiv preprint arXiv:2210.06311, 2022

Few-shot learning (FSL) has attracted considerable attention recently. Among existing approaches, the metric-based method aims to train an embedding network that can make similar samples close while dissimilar samples as far as possible and …


Semantic Network: A Brief Review of its Datasets

M Trovati, S Awan - International Conference on Intelligent Networking and …, 2022

Semantic networks are graphical representations, in terms of nodes and edges, of words and phrases from the linguistics vocabulary to form a meaning and relation between them to describe the intended target or object. It is part of the computer …


Semantic projection recovers rich human knowledge of multiple object features from word embeddings

G Grand, IA Blank, F Pereira, E Fedorenko - Nature Human Behaviour, 2022

How is knowledge about word meaning represented in the mental lexicon? Current computational models infer word meanings from lexical co-occurrence patterns. They learn to represent words as vectors in a multidimensional space, wherein …


Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning

G Fan, J Wang, Y Li, D Zhang, R Miller - arXiv preprint arXiv:2210.01922, 2022

Dataset discovery from data lakes is essential in many real application scenarios. In this paper, we propose Starmie, an end-to-end framework for dataset discovery from data lakes (with table union search as the main use case). Our proposed framework …


Semantics-driven Attentive Few-shot Learning over Clean and Noisy Samples

OB Baran, RG Cinbiş - arXiv preprint arXiv:2201.03043, 2022

Over the last couple of years few-shot learning (FSL) has attracted great attention towards minimizing the dependency on labeled training examples. An inherent difficulty in FSL is the handling of ambiguities resulting from having too few training …


SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding

HT Madabushi, E Gow-Smith, M Garcia, C Scarton… - arXiv preprint arXiv …, 2022

This paper presents the shared task on Multilingual Idiomaticity Detection and Sentence Embedding, which consists of two subtasks: (a) a binary classification one aimed at identifying whether a sentence contains an idiomatic expression, and (b) a …


Semi-Supervised Lifelong Language Learning

Y Zhao, Y Zheng, B Yu, Z Tian, D Lee, J Sun, H Yu, Y Li… - arXiv preprint arXiv …, 2022

Lifelong learning aims to accumulate knowledge and alleviate catastrophic forgetting when learning tasks sequentially. However, existing lifelong language learning methods only focus on the supervised learning setting. Unlabeled data …


Sentence Simplification Capabilities of Transfer-Based Models

… T5 is trained on Colossal Clean Crawled Corpus (C4), a dataset created by applying a set of filters to English-language text sourced from the public Common Crawl web scrape. • mT5 (Xue et al. 2021): a multilingual model based on T5 (Raffel …


SEntFiN 1.0: Entity‐aware sentiment analysis for financial news

A Sinha, S Kedas, R Kumar, P Malo - Journal of the Association for Information Science and …

Fine‐grained financial sentiment analysis on news headlines is a challenging task requiring human‐annotated datasets to achieve high performance. Limited studies have tried to address the sentiment extraction task in a setting where multiple …


Sentiment Analysis of COVID-19 Vaccination Responses From Twitter Using Ensemble Learning

Q Ismail, R Obeidat, K Alissa, E Al-Sobh - 2022 13th International Conference on …, 2022

… As an alternative to BERT, we also use RoBERTa, a robust pre-trained language model for English, learned from 160GB of texts covering books, Wikipedia, CommonCrawl news, CommonCrawl stories, and web text contents [11]. Our SA …


Sentiment Analysis of Russian Reviews to Estimate the Usefulness of Drugs Using the Domain-Specific XLM-RoBERTa Model

A Sboev, A Naumov, I Moloshnikov, R Rybka - Biologically Inspired Cognitive …, 2022

This paper considers the problem of classifying Russian-language drug reviews into five classes (corresponding to the authors’ rating scores) and into two classes (the drug is helpful or useless) in terms of the sentiment analysis task. The dataset of …


Sentiment Analysis with Linguistic Knowledge

… This thesis employs FastText CBOW word and text vectors with dimensionality 300 pre-trained on Wikipedia and the Common Crawl1 web corpus with negative sampling and n-gram length 52. The SVM, Logistic Regression, and Naïve Bayes …


Sentiment Analysis: A comparative study of Deep Learning and Machine Learning

R Manshani, T Thatte, K Suryawanshi - 2022

… using the word vectors trained on Common Crawl and one hot encoding our y variable or training set. Our model contains a commencing layer of LSTM that will collect all the word embeddings corresponding to each token as an input. Our main …


SentSpace: Large-Scale Benchmarking and Evaluation of Text using Cognitively Motivated Lexical, Syntactic, and Semantic Features

G Tuckute, A Sathe, M Wang, H Yoder, C Shain… - Proceedings of the 2022 …, 2022

SentSpace is a modular framework for streamlined evaluation of text. SentSpacecharacterizes textual input using diverse lexical, syntactic, and semantic features derivedfrom corpora and psycholinguistic experiments. Core sentence …


Sequence Labeling Architectures in Diglossia

… FastText is a lightweight library pretrained on the Common Crawl dataset18 and Wikipedia in 157 languages including Arabic19 “using [Continuous Bag of Words] with position-weights, in 300 dimensions, with character n-grams of length 5, a …


SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation

W Xu, X Qian, M Wang, L Li, WY Wang - arXiv preprint arXiv:2212.09305, 2022

Is it possible to leverage large scale raw and raw parallel corpora to build a general learned metric? Existing learned metrics have gaps to human judgements, are model-dependent or are limited to the domains or tasks where human ratings are available. In this …


SEUPD@ CLEF: Team Hextech on argument retrieval for comparative questions. the importance of adjectives in documents quality evaluation

A Chimetto, D Peressoni, E Sabbatini, G Tommasin… - Working Notes Papers of the …, 2022

This report explains our approach to solve the Task 2 challenge about Argument Retrieval for Comparative Questions proposed by the third Touché lab on argument retrieval at CLEF 2022. Given a comparative topic, the task is to retrieve relevant …


SexWEs: Domain-Aware Word Embeddings via Cross-lingual Semantic Specialisation for Chinese Sexism Detection in Social Media

A Jiang, A Zubiaga - arXiv preprint arXiv:2211.08447, 2022

The goal of sexism detection is to mitigate negative online content targeting certain gender groups of people. However, the limited availability of labeled sexism-related datasets makes it problematic to identify online sexism for low-resource languages …


Shedding Light on the Dark Web: Authorship Attribution in Radical Forums

L Ranaldi, F Ranaldi, F Fallucchi, FM Zanzotto - Information, 2022

Online users tend to hide their real identities by adopting different names on the Internet. On Facebook or LinkedIn, for example, people usually appear with their real names. On other standard websites, such as forums, people often use …


Simple measures of bridging lexical divergence help unsupervised neural machine translation for low-resource languages

J Khatri, R Murthy, T Banerjee, P Bhattacharyya - Machine Translation, 2022

… We consider CommonCrawl and the BigEst Estonian Corpus from the WMT18 shared task (Bojar et al. ) to create the monolingual corpus for Estonian. For English, we consider News Crawl 2017 and News Discussions 2017 from the WMT17 shared …


Simultaneous Translation for Unsegmented Input: A Sliding Window Approach

S Sen, O Bojar, B Haddow - arXiv preprint arXiv:2210.09754, 2022

In the cascaded approach to spoken language translation (SLT), the ASR output is typically punctuated and segmented into sentences before being passed to MT, since the latter is typically trained on written text. However, erroneous segmentation …


SketchNE: Embedding Billion-Scale Networks Accurately in One Hour

Y Xie, Y Dong, J Qiu, W Yu, X Feng, J Tang

We study large-scale network embedding with the goal of generating high-quality embeddings for networks with more than 1 billion vertices and 100 billion edges. Recent attempts LightNE and NetSMF propose to sparsify and factorize the (dense) …


Sliced at SemEval-2022 Task 11: Bigger, Better? Massively Multilingual LMs for Multilingual Complex NER on an Academic GPU Budget

B Plank - Proceedings of the 16th International Workshop on …, 2022

Massively multilingual language models (MMLMs) have become a widely-used representation method, and multiple large MMLMs were proposed in recent years. A trend is to train MMLMs on larger text corpora or with more layers. In this paper we …


SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routing

C He, S Zheng, A Zhang, G Karypis, T Chilimbi… - arXiv preprint arXiv …, 2022

… We evaluate the performance of SMILE by pre-training on the “Colossal Clean Crawled Corpus” (C4), a collection of English-language text sourced from the public Common Crawl web scrape. It includes heuristics to extract only natural language (as …


Smooth inverse frequency based text data selection for medical dictation

Under-resourced domain problem is significant in automatic speech recognition, especially in small languages such as Hungarian or in fields where data is often confidential such as finance and medicine. We introduce a method using word …


SMTCE: A Social Media Text Classification Evaluation Benchmark and BERTology Models for Vietnamese

LT Nguyen, K Van Nguyen, NLT Nguyen - arXiv preprint arXiv:2209.10482, 2022

Text classification is a typical natural language processing or computational linguistics task with various interesting applications. As the number of users on social media platforms increases, data acceleration promotes emerging studies on …


Social and Web presence of Cultural Heritage Organisations in India

The present study focuses on the Social and Web presence of 27 Cultural Heritage Organisations in India. The purpose of this study is to investigate the domain authority, number of webpages, links, calculate the web impact factors and link-mentions …


SoK: On the Impossible Security of Very Large Foundation Models

EM El-Mhamdi, S Farhadkhani, R Guerraoui, N Gupta… - arXiv preprint arXiv …, 2022

Large machine learning models, or so-called foundation models, aim to serve as base-models for application-oriented machine learning. Although these models showcase impressive performance, they have been empirically found to pose …


Solving Hungarian natural language processing tasks with multilingual generative models

Generative ability is a crucial need for artificial intelligence applications, such as chatbots, virtual assistants, machine translation systems etc. In recent years, the transformer-based neural architectures gave a huge boost to generate human-like …


Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP World

S Ranathunga, N de Silva - arXiv preprint arXiv:2210.08523, 2022

Linguistic disparity in the NLP world is a problem that has been widely acknowledged recently. However, different facets of this problem, or the reasons behind this disparity are seldom discussed within the NLP community. This paper …


SOS: Systematic Offensive Stereotyping Bias in Word Embeddings

F Elsafoury, SR Wilson, S Katsigiannis, N Ramzan

Systematic Offensive stereotyping (SOS) in word embeddings could lead to associating marginalised groups with hate speech and profanity, which might lead to blocking and silencing those groups, especially on social media platforms. In this …


SPARQL querying for validating the usage of automatically georeferenced social media data as human sensors for air quality

S Andreadis, T Mavropoulos, N Pantelidis, S Vrochidis… - 2022 IEEE 14th Image …, 2022

… original XLM model, XLM-R drew inspiration from RoBERTa [19], in the sense that it was trained for a longer time on more data, ie more than 2TBs of filtered CommonCrawl data. Since the tweets collection relates to the German language, a …


SPBERTQA: A Two-Stage Question Answering System Based on Sentence Transformers for Medical Texts

NTH Nguyen, PPD Ha, LT Nguyen, K Van Nguyen… - arXiv preprint arXiv …, 2022

Question answering (QA) systems have gained explosive attention in recent years. However, QA tasks in Vietnamese do not have many datasets. Significantly, there is mostly no dataset in the medical domain. Therefore, we built a Vietnamese …


Speculative Propositions for Digital Writing Under the New Autonomous Model of Literacy

B Robinson - Postdigital Science and Education, 2022

The proliferation of powerful new forms of automated assistive writing technologies, natural language generation technologies in particular, raises crucial questions about the future of literacy living and learning. This article situates such technologies …


Speech-to-Speech Translation For A Real-world Unwritten Language

PJ Chen, K Tran, Y Yang, J Du, J Kao, YA Chung… - arXiv preprint arXiv …, 2022

… With the trained text and speech encoders, we perform data mining between Hokkien speech from Hokkien dramas and English Common Crawl text, and between the former and Librivox English audio7. We post-process the mined data in …


SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations

PA Duquenne, H Gong, N Dong, J Du, A Lee… - arXiv preprint arXiv …, 2022

We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of …


Speeding up Inference Time of Neural Machine Translation

Large qualitative gains were recently made in machine translation tasks thanks to Transformers models. However, in practice, these models suffer from high latency, such that they often are hardly usable in practical applications. This thesis study …


Stability of a Multilingual Sentiment Analysis based on Word-to-Word Translations

N Horibe, K Fujihira - International Journal of Service and Knowledge …, 2022

People’s sentiments are known to have a large impact on changes in stock prices, products sales, and trends. Since web users generally state their opinion in various languages, it is important to develop a method of multilingual sentiment analysis for …


Stacked Model based Argument Extraction and Stance Detection using Embedded LSTM model

P Rajula, CC Hung, SP Ponzetto - Working Notes Papers of the CLEF, 2022

In this paper, we present our submission approach for the third Touché lab at CLEF 2022 [1], shared task 2 on Argument Retrieval for Comparative Questions, which tackles answering comparative questions based on argument retrieval of text …


Stance Detection of Political Tweets with Transformer Architectures

P Gunhal, A Bashyam, K Zhang, A Koster, J Huang…

The online actions and words of a person can reveal their political sentiments and how they may vote at the polls. For decades, the dominant strategy of determining voter sentiment on policies relied on slow and often inaccurate polling. The creation …


State-of-the-art in Open-domain Conversational AI: A Survey

T Adewumi, F Liwicki, M Liwicki - arXiv preprint arXiv:2205.00965, 2022

We survey SoTA open-domain conversational AI models with the purpose of presenting the prevailing challenges that still exist to spur future research. In addition, we provide statistics on the gender of conversational AI in order to guide the ethics …


Stateful Memory-Augmented Transformers for Dialogue Modeling

Q Wu, Z Yu - arXiv preprint arXiv:2209.07634, 2022

Transformer encoder-decoder models have shown impressive performance in dialogue modeling. However, as Transformers are inefficient in processing long sequences, dialogue history length often needs to be truncated. To address this …