Statistical Dataset Evaluation: Reliability, Difficulty, and Validity

C Wang, Q Dong, X Wang, H Wang, Z Sui - arXiv preprint arXiv:2212.09272, 2022

Datasets serve as crucial training resources and model performance trackers. However, existing datasets have exposed a plethora of problems, inducing biased models and unreliable evaluation results. In this paper, we propose a model-agnostic …


Statistical detection of format dialects using the weighted Dowker complex

M Robinson, LW Li, C Anderson, S Huntsman - arXiv preprint arXiv:2201.08267, 2022

… The good files were largely sourced from Common Crawl [15], while the bad files were drawn from various … sources: some were found in the wild (from Common Crawl), some were malicious files created by the Test and Evaluation Team, and …


StoryNet: A 5W1H-Based Knowledge Graph to Connect Stories

Stories are a powerful medium through which the human community has exchanged information since the dawn of the information age. They have taken multiple forms like articles, movies, books, plays, short films, magazines, mythologies, etc. With the …


Strategies for Adapting Multilingual Pre-training for Domain-Specific Machine Translation

Pretrained multilingual sequence-to-sequence models have been successful in improving translation performance for mid-and lower-resourced languages. However, it is unclear if these models are helpful in the domain adaptation setting, and if so …


Streaming neural machine translation systems from English into European languages

[EN] Machine translation is one of the most active areas within Artificial Intelligence, particularly in the field of Machine Learning. Recently, this area has been the focus of attention by major technology figures such as Google, Facebook, Microsoft, etc …


Streaming neural machine translation systems from European languages into English


Streaming, fast and accurate on-device Inverse Text Normalization for Automatic Speech Recognition

Y Gaur, N Kibre, J Xue, K Shu, Y Wang, I Alphanso, J Li… - arXiv preprint arXiv …, 2022

… The data is drawn from a coprora of email text, a collection of email addresses, common crawl data, and search queries. The selection of data from common crawl is loosely constrained to ITN-relevant strings, with some heuristics to avoid “mis-formatted” …


Stress Detection from Social Media Articles: New Dataset Benchmark and Analytical Study

Stress detection is a basic and essential task for examining the mental health of a given population. With the rapid digitalization leading to text-based forms of communication gaining dominance over spoken ones, there is now the chance to …


Subspace-based Set Operations on a Pre-trained Word Embedding Space

Y Ishibashi, S Yokoi, K Sudoh, S Nakamura - arXiv preprint arXiv:2210.13034, 2022

Word embedding is a fundamental technology in natural language processing. It is often exploited for tasks using sets of words, although standard methods for representing word sets and set operations remain limited. If we can leverage the …


Subword-based Cross-lingual Transfer of Embeddings from Hindi to Marathi and Nepali

N Bafna, Z Žabokrtský - Proceedings of the 19th SIGMORPHON Workshop on …, 2022

Word embeddings are growing to be a crucial resource in the field of NLP for any language. This work introduces a novel technique for static subword embeddings transfer for Indic languages from a relatively higher resource language to a …


SummVD: An efficient approach for unsupervised topic-based text summarization

G Shenouda, A Bossard, O Ayoub, C Rodrigues - … of the 2nd Conference of the Asia …, 2022

This paper introduces a new method, SummVD, for automatic unsupervised extractive summarization. This method is based on singular value decomposition, a linear method in the number of words, in order to reduce the dimensionality of word …


SumREN: Summarizing Reported Speech about Events in News

RG Reddy, H Elfardy, HP Chan, K Small, H Ji - arXiv preprint arXiv:2212.01146, 2022

… We then collected a list of news articles discussing these events and retained articles that are present in Common Crawl (CC) News.We ended up with a total of 633 news articles corresponding to 132 major events. …


Survey of NLP in Pharmacology: Methodology, Tasks, Resources, Knowledge, and Tools

D Trajanov, V Trajkovski, M Dimitrieva, J Dobreva… - arXiv preprint arXiv …, 2022

Natural language processing (NLP) is an area of artificial intelligence that applies information technologies to process the human language, understand it to a certain degree, and use it in various applications. This area has rapidly developed in the …


Survey on bimodal speech emotion recognition from acoustic and linguistic information fusion

BT Atmaja, A Sasou, M Akagi - Speech Communication, 2022

Speech emotion recognition (SER) is traditionally performed using merely acoustic information. Acoustic features, commonly are extracted per frame, are mapped into emotion labels using classifiers such as support vector machines for machine …


Symptom or culprit? Social media, air pollution, and violence

This paper provides the first causal evidence that hostile activities online lead to physical violence. Given the recently documented relationship between pollution and social media, I exploit exogenous variation in local air quality as the first step to …


SYNONYM DETECTION USING SYNTACTIC DEPENDENCY AND NEURAL EMBEDDINGS

Recent advances on the Vector Space Model have significantly improved some NLP applications such as neural machine translation and natural language generation. Although word co-occurrences in context have been widely used in counting-/predicting-based …


Synopses of Movie Narratives: a Video-Language Dataset for Story Understanding

Y Sun, Q Chao, B Li - arXiv preprint arXiv:2203.05711, 2022

Despite recent advances of AI, story understanding remains an open and under-investigated problem. We collect, preprocess, and publicly release a video-language story dataset, Synopses of Movie Narratives(SyMoN), containing 5,193 video summaries …


Synthesis and Evaluation of a Domain-specific Large Data Set for Dungeons & Dragons

A Peiris, N de Silva - arXiv preprint arXiv:2212.09080, 2022

This paper introduces the Forgotten Realms Wiki (FRW) data set and domain specific natural language generation using FRW along with related analyses. Forgotten Realms is the de-facto default setting of the popular open ended tabletop …


Table Pretraining: A Survey on Model Architectures, Pretraining Objectives, and Downstream Tasks

H Dong, Z Cheng, X He, M Zhou, A Zhou, F Zhou, A Liu… - arXiv preprint arXiv …, 2022

Since a vast number of tables can be easily collected from web pages, spreadsheets, PDFs, and various other document types, a flurry of table pretraining frameworks have been proposed following the success of text and images, and they have …


Tackling Social Value Tasks with Multilingual NLP

In recent years, deep learning applications have shown promise in tackling social value tasks such as hate speech and misinformation in social media. Neural networks provide an efficient automated solution that has replaced hand-engineered …


TAG: Learning Circuit Spatial Embedding From Layouts

K Zhu, H Chen, WJ Turner, GF Kokai, PH Wei, DZ Pan… - arXiv preprint arXiv …, 2022

Analog and mixed-signal (AMS) circuit designs still rely on human design expertise. Machine learning has been assisting circuit design automation by replacing human experience with artificial intelligence. This paper presents TAG, a new paradigm of …


Tagging terms in text

… We used the embeddings that are pre-trained on Common Crawl (for all languages). Finally, the hugely successful transformer-based architectures are supported in FlairNLP as well, through HuggingFace (Wolf et al. …


Talk to me: Database Vocalization Summaries using BART and PEGASUS

A Elkhouly, P Joshi, S Rathod, S Aziz, V Kalra…

Zeta bytes of data are generated on a daily basis and this is growing continuously. Naturally, it is imperative for all kinds of stakeholders to have tools that help them gain insights into this data. So far text summarization work has been done using …


TamilEmo: Finegrained Emotion Detection Dataset for Tamil

C Vasantharajan, S Benhur, PK Kumarasen… - arXiv preprint arXiv …, 2022

Emotional Analysis from textual input has been considered both a challenging and interesting task in Natural Language Processing. However, due to the lack of datasets in low-resource languages (ie Tamil), it is difficult to conduct research of …


Tasks Using the Hugging Face Library

SM Jain - Introduction to Transformers for NLP, 2022

… It produces a tidier version of the enormous Common Crawl dataset, which is referred to as the Colossal Cleaned Common Crawl (C4) . This dataset is approximately 100,000 times more extensive than Wikipedia. … It prepares the body …


TaTa: A Multilingual Table-to-Text Dataset for African Languages

S Gehrmann, S Ruder, V Nikolaev, JA Botha… - arXiv preprint arXiv …, 2022

Existing data-to-text generation datasets are mostly limited to English. To address this lack of data, we create Table-to-Text in African languages (TaTa), the first large multilingual table-to-text dataset with a focus on African languages. We created …


Teaching AI when to care about gender

J Powell, K Sentz, E Moyer, M Klein - Code4Lib Journal, 2022

… For many deep learning/NLP tasks, it is standard practice to download pre-trained models such as GoogleNews, Common Crawl, and wikipedia word embeddings models and use them for deep learning tasks. Since pretrained models are trained …


Technical Report on Web-based Visual Corpus Construction for Visual Document Understanding

D Kim, T Hong, M Yim, Y Kim, G Kim - arXiv preprint arXiv:2211.03256, 2022

We present a dataset generator engine named Web-based Visual Corpus Builder (Webvicob). Webvicob can readily construct a large-scale visual corpus (ie, images with text annotations) from a raw Wikipedia HTML dump. In this report, we validate that …


Technology behind text to image generators

P Fernandez - Library Hi Tech News, 2022

… Common Crawl, which has petabytes of data taken by scraping websites. Copyright law in the USA can be frustratingly ambiguous, but factors in Common Crawl’s … In other words, Common Crawl creates large data sets used for …


Technology, Vintage-Specific Human Capital, and Labor Displacement: Evidence from Linking Patents with Occupations

L Kogan, D Papanikolaou, LDW Schmidt, B Seegmiller

We develop a granular, occupation-specific measure of technological progress that relies only on textual descriptions of patent documents and the tasks performed by workers in an occupation. Our measure primarily identifies labor-saving innovations …


Test collections for web-scale datasets using Dynamic Sampling

… Other large scale web collections, like the Common Crawl dataset, have also been used in various TREC tracks. It might be interesting to study the performance of Dynamic Sampling on these alternate datasets like Common Crawl1. Additionally, in …


TeSum: Human-Generated Abstractive Summarization Corpus for Telugu

A Urlana, N Surange, P Baswani, P Ravva…

Expert human annotation for summarization is definitely an expensive task, and can not be done on huge scales. But with this work, we show that even with a crowd sourced summary generation approach, quality can be controlled by aggressive …


Text Algorithms in Economics

This paper provides an overview of the methods used for algorithmic text analysis in economics, with a focus on three key contributions. First, the paper introduces methods for representing documents as high-dimensional count vectors over …


Text Analysis of Airline Tweets

… For the FastText approach, we used gensim to load the vector file from the common crawl models downloaded from fasttext (FastText, 2020) and converted it to a bin model because the loading time of the vector was more. We then continued to …


Text Characterization Toolkit (TCT)

D Simig, T Wang, V Dankers, P Henderson, K Batsuren… - … of the 2nd Conference of the …, 2022

We present a tool, Text Characterization Toolkit (TCT), that researchers can use to study characteristics of large datasets. Furthermore, such properties can lead to understanding the influence of such attributes on models’ behaviour. Traditionally, in …


Text Characterization Toolkit

D Simig, T Wang, V Dankers, P Henderson, K Batsuren… - arXiv preprint arXiv …, 2022

In NLP, models are usually evaluated by reporting single-number performance scores on a number of readily available benchmarks, without much deeper analysis. Here, we argue that - especially given the well-known fact that benchmarks often …


Text Embeddings by Weakly-Supervised Contrastive Pre-training

L Wang, N Yang, X Huang, B Jiao, L Yang, D Jiang… - arXiv preprint arXiv …, 2022

… data sources such as CommunityQA, Common Crawl and Scientific papers, and perform … Simple heuristic rules are applied to filter data from Reddit and Common Crawl. For example, … Reddit and Common Crawl. For more details and examples …


Text Representations and Word Embeddings

R Egger - Applied Data Science in Tourism, 2022

Today, a vast amount of unstructured text data is consistently at our disposal. Owing to the rapid increase in user-generated content on the one hand and powerful, ready-to-use machine learning algorithms on the other hand, automated text analysis can now be …


Text to Image GANs with RoBERTa and Fine-grained Attention Networks

… The additional data included CommonCrawl News dataset, Web text corpus and Stories from Common Crawl. The RoBERTa makes use of similar architecture as BERT Model but uses the byte-level BPE as the tokenizer [4]. The ’roberta-base’ is …


Text Vectorization Method Based on Concept Mining Using Clustering Techniques

A Mansour, J Mohammad, Y Kravchenko - 2022 VI International Conference on …, 2022

… word vectors trained with sub-word information on Wikipedia 2017, UMBC webbase corpus, and statmt.org news data set (16B tokens); word2vec model2 which was trained with Google News corpus (3 billion words) and Glove model …


Text-to-Text Transfer Transformer Phrasing Model Using Enriched Text Input

M Řezáčková, J Matoušek - International Conference on Text, Speech, and …, 2022

… In our experiments, we used the Text-to-Text Transfer Transformer (T5) model (following the architecture presented in [16, 25]) pre-trained from a huge amount of unlabeled text from the Czech CommonCrawl corpus [27]. For training the model itself, we used …


Textual Inference Identification in the Malayalam Language Using Convolutional Neural Network

S Renjit, SM Idicula - Advanced Computing and Intelligent Technologies, 2022

… fastText [11] has a collection of pre-trained word vectors in 157 languages, where each language word collection is from Wikipedia and common crawl. Such collections provide a good start for language processing tasks in low-resource …


The (Moral) Language of Hate

B Kennedy, P Golazizian, J Trager, M Atari, J Hoover… - 2022

… In the present study, we extract the FastText embeddings — via the pre-trained set of embeddings10, which are trained on a combination of Wikipedia and Common Crawl data — of terms in the Weaponized Word lexicon, for the 19 …


THE ADVANCE TECHNIQUES USED IN CYBER SECURITY FOR PHISHING DETECTION

The Internet has become a necessary part of our lives; however, it has also provided opportunities to carry out malicious activities anonymously like Phishing. Phishers try to trick their victims through social engineering or create mock-up websites to …


The AISP-SJTU Translation System for WMT 2022

G Liu, Q Zhu, X Chen, R Feng, J Ren, R Wu, Q Miao… - Proceedings of the Seventh …, 2022

… For monolingual data, we select data from News Crawl, Common Crawl and Extended Common Crawl, and the amount of data after pro… Crawl and Common Crawl. Then we used the JA-EN ensemble model to generate the hypotheses as the …


The case for 4-bit precision: k-bit Inference Scaling Laws

T Dettmers, L Zettlemoyer - arXiv preprint arXiv:2212.09720, 2022

… Furthermore, we find that across more than 35,000 zero-shot experiments, the Pearson correlation coefficient between The Pile Common Crawl perplexity and zero-shot performance is -0.94. … In this section, we present data for evaluation on The Pile …


The Causal News Corpus: Annotating Causal Relations in Event Sentences from News

FA Tan, A Hürriyetoğlu, T Caselli, N Oostdijk, T Nomoto… - arXiv preprint arXiv …, 2022

Despite the importance of understanding causality, corpora addressing causal relations are limited. There is a discrepancy between existing annotation guidelines of event causality and conventional causality corpora that focus more on linguistics …


The Curious Case of Control

E Stengel-Eskin, B Van Durme - arXiv preprint arXiv:2205.12113, 2022

… The training data is based on Common Crawl, though similarly to GPT-3 Davinci, the details of the training data filtering process are unclear. Relevant differences to GPT-3 are in the tokenization (which includes multi-word expressions) and use of …


The Dark Side of the Language: Pre-trained Transformers in the DarkNet

L Ranaldi, A Nourbakhsh, A Patrizi, ES Ruzzetti… - arXiv preprint arXiv …, 2022

Pre-trained Transformers are challenging human performances in many natural language processing tasks. The gigantic datasets used for pre-training seem to be the key for their success on existing tasks. In this paper, we explore how a range of …


The Effect of Pre-trained Language Models on Critical Case Detection from Radiology Reports

… As the embedding layer of the bi-LSTM we use following alternatives: 1) fastText pre-trained model for Turkish on common crawl 3 2) fastText model trained from scratch using our unsupervised dataset of 190 thousand radiology reports 3) BERT …


The emergence of gender associations in child language development

B Prystawski, E Grant, A Nematzadeh, SWS Lee…

Gender associations have been a long-standing research topic in psychological and social sciences. Although it is known that children learn aspects of gender association at a young age, it is not well understood how they might emerge through …


The fujitsu dmath submissions for wmt21 news translation and biomedical translation tasks

A Martínez - Proceedings of the Sixth Conference on Machine …, 2021

… For example, we noted that the Hausa Extended Common Crawl corpus published for WMT21 contained a large number of Japanese … Despite its noisy nature, we decided to use the Extended Common Crawl Hausa corpus. The results …


The Ghost in the Machine has an American accent: value conflict in GPT-3

RL Johnson, G Pistilli, N Menédez-González… - arXiv preprint arXiv …, 2022

The alignment problem in the context of large language models must consider the plurality of human values in our world. Whilst there are many resonant and overlapping values amongst the world's cultures, there are also many conflicting, yet …


The hitchhiker's guide Method handbook for quantification of online lin-guistic data in a country-specific context Official research report, Linguistic Explorations of …

… A very significant non-profit provider to consider alongside these commercial actors is CommonCrawl (USA). As for CrowdTangle and DataSift… The CommonCrawl corpus contains petabytes of data including raw web page data …


The hitchhiker's guide to web-mediated text. Method handbook for quantification of online linguistic data in a country-specific context. Official research report …

… While a data provider could be any entity, also nonprofit or government-funded ones (CommonCrawl is one such example, described in more detail below), a vendor is typically a company operating in sectors like market intelligence, social …


The IARPA BETTER Program Abstract Task Four New Semantically Annotated Corpora from IARPA's BETTER Program

… In Phase I, sentences were randomly chosen from newswire texts in the target languages harvested from the Common Crawl. An Amazon Mechanical Turk pipeline was set up to confirm the suitability of the extracted sentences according to …


THE IMPACT OF ONLINE AFFILIATE MARKETING ON WEB SEARCH

J Bevendorff, M Wiegmann, M Potthast, B Stein

… Based on several pageand domain-level text quality features, which we calculated at scale on four different Common Crawl versions, we … number of pages crawled from domains that fall into the “affiliate spam” genre might improve the quality of …


The Increasing Frequency of Terms Denoting Political Extremism in US and UK News Media

D Rozado, E Kaufmann - Social Sciences, 2022

The term political extremism is commonly used to refer to political attitudes considered to be outside the ideological mainstream. This study leverages computational content analysis of big data to longitudinally examine (1970–2019) …


The Indeterminate Self and Large Language Models: A Nietzschean Critique of GPT-3

… For GPT-3, the largest part of the training data (60%) comes from the web scraping repository Common Crawl12 (Brown et al., 2020, p. 9). The exact sources of the Common Crawl data set are not disclosed, and given the sheer amount it is …


The Infinite Index: Information Retrieval on Generative Text-To-Image Models

N Deckers, M Fröbe, J Kiesel, G Pandolfo, C Schröder… - arXiv preprint arXiv …, 2022

The text-to-image model Stable Diffusion has recently become very popular. Only weeks after its open source release, millions are experimenting with image generation. This is due to its ease of use, since all it takes is a brief description of the …


The language of opinion change on social media under the lens of communicative action

C Monti, LM Aiello, G De Francisci Morales, F Bonchi - Scientific Reports, 2022

Which messages are more effective at inducing a change of opinion in the listener? We approach this question within the frame of Habermas’ theory of communicative action, which posits that the illocutionary intent of the message (its pragmatic …


The Minimum Wage as an Anchor: Effects on Determinations of Fairness by Humans and AI

DG Soatto - arXiv preprint arXiv:2210.10585, 2022

I study the role of the minimum wage as an anchor for perceptions of the fairness of wages by both human subjects and AI. Through surveys on human subjects enrolled in the crowdsourcing platform Prolific.co and queries submitted to the large …


The mininglamp machine translation system for wmt21

S Zhao, X Li, M Wu, J Hao - Proceedings of the Sixth Conference on Machine …, 2021

This paper describes Mininglamp neural machine translation systems of the WMT2021 news translation tasks. We have participated in eight directions translation tasks for news text including Chinese to/from English, Hausa to/from …


The MORAL INTEGRITY CORPUS: A Benchmark for Ethical Dialogue Systems

C Ziems, AY Jane, YC Wang, A Halevy, D Yang

Conversational agents have come increasingly closer to human competence in open-domain dialogue settings; however, such models often reflect insensitive, hurtful, or entirely incoherent viewpoints that erode a user’s trust in the moral integrity of the system …


The multiplex relations between cities: a lexicon-based approach to detect urban systems

W Tongjing, E Meijers, H Wang - Regional Studies, 2022

… Common Crawl is one of the largest publicly available web archives, a … the Common Crawl web archive as our preliminary data source to avoid selection bias. We extracted a 139 Gigabyte text corpus containing 91 million Chinese webpages …


The NiuTrans Machine Translation Systems for WMT22

W Shan, Z Cao, Y Han, S Wu, Y Hu, T Xiao, A Ma…

This paper describes the NiuTrans neural machine translation systems of the WMT22 General MT constrained task. We participate in four directions, including Chinese→ English, English→ Croatian, and Livonian↔ English. Our models are …


The Norwegian Colossal Corpus: A Text Corpus for Training Large Norwegian Language Models

Norwegian has been one of many languages lacking sufficient available text to train quality language models. In an attempt to bridge this gap, we introduce the Norwegian Colossal Corpus (NCC), which comprises 49GB of clean Norwegian …


The only chance to understand: machine translation of the severely endangered low-resource languages of Eurasia

A Mossolova, K Smaïli - The Fifth Workshop on Technologies for Machine …, 2022

Numerous machine translation systems have been proposed since the appearance of this task. Nowadays, new large language modelbased algorithms show results that sometimes overcome human ones on the rich-resource languages …


The Problem with XSD Binary Floating Point Datatypes in RDF

… in the September 2020 Common Crawl archive,13 a freely available web crawl archive. We selected it because of its large size, an expected large proportion of literals, the uniform access of the whole corpus, its heterogeneous original sources (15e6domains) …


The Role of negative information when learning dense word vectors

… Figure 1.1 shows a 2-dimensional PCA projection of a set of words vectors drawn from our LexVec Common Crawl vectors (the LexVec model is presented in Chapter 3), illustrating how a vector space model represents meaning. Words that are …


The semantics of-ee and-ation: a distributional semantic approach

… Common Crawl subword model • 2 million pre-trained word vectors • Contains subword information to create new vectors based on …


The Seventh Workshop on Search-Oriented Conversational Artificial Intelligence (SCAI'22)

G Penha, S Vakulenko, O Dusek, L Clark, V Pal… - Proceedings of the 45th …, 2022

… The two datasets also differ with respect to the information source — QReCC has a larger and more diverse collection of 54M passages based on Commoncrawl, whereas TopiOCQA has 26M passages based on Wikipedia. While the corpus of …


The sins of the parents are to be laid upon the children: biased humans, biased data, biased models

Technological innovations have become a key driver of societal advancements. Nowhere is this more evident than in the field of machine learning (ML), which has developed algorithmic models that shape our decisions, behaviors, and outcomes …


The State of Profanity Obfuscation in Natural Language Processing

D Nozza, D Hovy - arXiv preprint arXiv:2210.07595, 2022

Work on hate speech has made the consideration of rude and harmful examples in scientific publications inevitable. This raises various problems, such as whether or not to obscure profanities. While science must accurately disclose what it does, the …


The UMD Neural Machine Translation Systems at WMT17 Bandit Learning Task

We describe the University of Maryland machine translation systems submitted to the WMT17 German-English Bandit Learning Task. The task is to adapt a translation system to a new domain, using only bandit feedback: the system receives a German …


The usefulness of NLP techniques for predicting peaks in firefighter interventions due to rare events

S Cerna, C Guyeux, D Laiymani - Neural Computing and Applications, 2022

… This is mainly because the LSTM and CNN models were trained only with the vocabulary of the bulletins while the transformers were pre-trained with an extensive vocabulary of the French language (Common Crawl Oscar corpus). Thus, this set of …


The VolcTrans System for WMT22 Multilingual Machine Translation Task

X Qian, K Hu, J Wang, Y Liu, X Pan, J Cao, M Wang - arXiv preprint arXiv:2210.11599, 2022

This report describes our VolcTrans system for the WMT22 shared task on large-scale multilingual machine translation. We participated in the unconstrained track which allows the use of external resources. Our system is a transformerbased multilingual …


The YiTrans End-to-End Speech Translation System for IWSLT 2022 Offline Shared Task

This paper describes the submission of our end-to-end YiTrans speech translation system for the IWSLT 2022 offline task, which translates from English audio to German, Chinese, and Japanese. The YiTrans system is built on large-scale pre-trained …


ThEconSum: an Economics-domained Dataset for Thai Text Summarization and Baseline Models

S Jumpathong, A Takhom, P Boonkwan… - 2022 17th International Joint …, 2022

Language resources as datasets are an essential component in developing an effective automatic text summarization (ATS) system. Some public datasets are relatively uncommon when compared with popular languages, due to the complexity …


Thirumurai: A Large Dataset of Tamil Shaivite Poems and Classification of Tamil Pann

… Such models are trained on petabytes of data, such as the Common Crawl data, which has over 12 years of internet data. Due to the size of the data they are being trained on, this not only improves the accuracy of the models, but it also allows them …


Three-order normalized PMI and other lessons in tensor analysis of verbal selectional preferences

We investigate several questions in transitive verb structure representation by decomposing tensors populated with different subject-verb-object association measures, including a novel generalization of normalized pointwise mutual …


TL; DR: Mining Reddit to Learn Automatic Summarization

Recent advances in automatic text summarization have used deep neural networks to generate high-quality abstractive summaries, but the performance of these models strongly depends on large amounts of suitable training data. We propose a …


To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering

D Dua, E Strubell, S Singh, P Verga - arXiv preprint arXiv:2212.10381, 2022

… To understand the impact of the pre-trainining vs fine-tuning corpus, we also compare the performance of the FiD reader initialized from T5 pretrained on common-crawl dataset(C4) compared to one that was pre-trained on pubmed articles. After …


Topic Classification in Indonesian-language Tweets using Fast-Text Feature Expansion with Support Vector Machine (SVM)

IM Fadhil, Y Sibaroni - 2022 International Conference on Data Science and Its …, 2022

… The word vector used in this study uses a pretrained Indonesian word vector that can be used from research [7] where the corpus used to train the model comes from Wikipedia and Common Crawl with the number of words contained in the word …


Towards a Cleaner Document-Oriented Multilingual Crawled Corpus

J Abadji, PO Suarez, L Romary, B Sagot - arXiv preprint arXiv:2201.06642, 2022

… Table 1: Comparison of occurrences of news-related terms between OSCAR and our corpus in a sample of 100 CommonCrawl shards. *: For the Burmese language, we use the whole 21.09 and 22.XX corpus since it is a low resource language …


Towards Analyzing the Bias of News Recommender Systems Using Sentiment and Stance Detection

M Alam, A Iana, A Grote, K Ludwig, P Müller… - arXiv preprint arXiv …, 2022

… More specifically, we use a pre-trained German word2vec model trained on the Common Crawl and Wikipedia dataset [12] to learn latent representations of the words in an article. The article’s vector representation is computed as the average of …


Towards Arabic Sentence Simplification via Classification and Generative Approaches

N Khallaf, S Sharoff - arXiv preprint arXiv:2204.09292, 2022

This paper presents an attempt to build a Modern Standard Arabic (MSA) sentence-level simplification system. We experimented with sentence simplification using two approaches: (i) a classification approach leading to lexical simplification pipelines …


Towards Automatic Generation of Messages Countering Online Hate Speech and Microaggressions

M Ashida, M Komachi - Proceedings of the Sixth Workshop on Online Abuse …, 2022

With the widespread use of social media, online hate is increasing, and microaggressions are receiving attention. We explore the potential for using pretrained language models to automatically generate messages that combat the …


Towards automatic generation of Piping and Instrumentation Diagrams (P&IDs) with Artificial Intelligence

E Hirtreiter, LS Balhorn, AM Schweidtmann - arXiv preprint arXiv:2211.05583, 2022

Developing Piping and Instrumentation Diagrams (P&IDs) is a crucial step during the development of chemical processes. Currently, this is a tedious, manual, and time-consuming task. We propose a novel, completely data-driven method for the …


Towards better structured and less noisy Web data: Oscar with Register annotations

V Laippala, A Salmela, S Rönnqvist, AF Aji, LH Chang… - Proceedings of the Eighth …, 2022

… The documents for the annotation were randomly sampled from a recent Common Crawl2 dataset. The annotation was done using a custom … The documents were retrieved from Common Crawl using the same pipeline as the …


Towards Controllable Protein design with Conditional Transformers

N Ferruz, B Höcker - arXiv preprint arXiv:2201.07338, 2022

The 21st century is presenting humankind with unprecedented environmental and medical challenges. The ability to design novel proteins tailored for specific purposes could transform our ability to respond timely to these issues. Recent …


Title: Towards fairness AI: A data-centric approach Subtitle: Assessment and mitigation techniques to tackle bias in datasets The consequences of bias and injustice have received more attention, even though AI is increasingly employed in …


Towards Fine (r)-grained Identification of Event Coreference Resolution Types

L De Langhe, O De Clercq, V Hoste - Computational Linguistics in the Netherlands …, 2022

… RobBERT on the other hand was pretrained on 6.6B tokens of Commoncrawl webdata (Suárez et al. 2019). Since the Commoncrawl data consists of individual lines and not every line contains more than one sentence, we anticipate that this …


Towards Latvian WordNet

P Paikens, M Grasmanis, A Klints, I Lokmane…

In this paper we describe our current work on creating a WordNet for Latvian based on the principles of the Princeton WordNet. The chosen methodology for word sense definition and sense linking is based on corpus evidence and the existing Tezaurs …


Towards Lithuanian grammatical error correction

L Stankevičius, M Lukoševičius - arXiv preprint arXiv:2203.09963, 2022

Everyone wants to write beautiful and correct text, yet the lack of language skills, experience, or hasty typing can result in errors. By employing the recent advances in transformer architectures, we construct a grammatical error correction model for …


Towards Multi-Lingual Visual Question Answering

S Changpinyo, L Xue, I Szpektor, AV Thapliyal… - arXiv preprint arXiv …, 2022

Visual Question Answering (VQA) has been primarily studied through the lens of the English language. Yet, tackling VQA in other languages in the same manner would require considerable amount of resources. In this paper, we propose scalable …


Towards Performance of NLP Transformers on URL-Based Phishing Detection for Mobile Devices

… , and the legitimate webpage sources are Alexa and CommonCrawl.org. In 2020, Shirazi et al. … .com and 10,955 legitimate URLs from CommonCrawl.org. The phishing URLs include sites … Since the CommonCrawl.org corpus contains …


Towards Simple and Efficient Task-Adaptive Pre-training for Text Classification

A Ladkat, A Miyajiwala, S Jagadale, R Kulkarni… - arXiv preprint arXiv …, 2022

Language models are pre-trained using large corpora of generic data like book corpus, common crawl and Wikipedia, which is essential for the model to understand the linguistic characteristics of the language. New studies suggest using Domain …


Towards the Next 1000 Languages in Multilingual Machine Translation: Exploring the Synergy Between Supervised and Self-Supervised Learning

A Siddhant, A Bapna, O Firat, Y Cao, MX Chen… - arXiv preprint arXiv …, 2022

Achieving universal translation between all human language pairs is the holy-grail of machine translation (MT) research. While recent progress in massively multilingual MT is one step closer to reaching this goal, it is becoming evident that …


Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers

Autograding short textual answers has become much more feasible due to the rise of NLP and the increased availability of question-answer pairs brought about by a shift to online education. Autograding performance is still inferior to human grading …


Towards Understanding and Practices of Ethical Artificial Intelligence

The successes of machine learning (ML) and artificial intelligence (AI) models encourage their widespread deployments in high-stakes domains--from public transportation to social decision-making such as autonomous driving, criminal justice …


Toxicity Detection for Indic Multilingual Social Media Content

M Jhaveri, D Ramaiya, HS Chadha - arXiv preprint arXiv:2201.00598, 2022

… • XLM-RoBERTa - It is a transformer-based masked language model trained on 100 languages, using more than two terabytes of filtered CommonCrawl data. [3] • MuRIL - A multilingual language model specifically built for Indic languages trained …


Toxicity detection in online Georgian discussions

N Lashkarashvili, M Tsintsadze - … Journal of Information Management Data Insights, 2022

… They trained models using Wikipedia and Common Crawl dataset. They used character n-grams with a size equal to 5 and the dimension of embeddings was set to 300. We found that 8,385, about 1 / 4 of the total of 38,733 raw unique words were …


Tradeoff Options for Bipartite Graph Partitioning

J Mackenzie, M Petri, A Moffat - IEEE Transactions on Knowledge and Data …, 2022

… CC-News-En: A snapshot of English news documents from the Common Crawl between 2016 and 2018 [32]. As was also the case with other work on reordering of inverted indexes, short postings lists (fewer than 4096 elements) and long postings …


Traditional Machine Learning and Deep Learning-based Text Classification for Turkish Law Documents using Transformers and Domain Adaptation

O Akça, G Bayrak, AM Issifu, MC Ganіz - … on INnovations in Intelligent SysTems and …, 2022

… We use their models which are pre-trained on a general common crawl corpus in Turkish. We than fine-tune this general domain model using our corpus that consists of a collection of Turkish legal texts to leverage the power of transfer learning. For …


Trainability and real-world knowledge

Until this point, we concentrated on the lexicon, conceived of as the repository of shared linguistic information. In 8.1 we take on the problem of integrating real-world knowledge, nowadays typically stored in knowledge graphs as billions of RDF triples …


Training a Deep Contextualized Language Model for International Classification of Diseases, 10th Revision Classification via Federated Learning: Model …

PF Chen, TL He, SC Lin, YC Chu, CT Kuo, F Lai… - JMIR Medical Informatics, 2022

Background: The automatic coding of clinical text documents by using the International Classification of Diseases, 10th Revision (ICD-10) can be performed for statistical analyses and reimbursements. With the development of natural …


Training a T5 Using Lab-sized Resources

MR Ciosici, L Derczynski - arXiv preprint arXiv:2208.12097, 2022

Training large neural language models on large datasets is resourceand time-intensive. These requirements create a barrier to entry, where those with fewer resources cannot build competitive models. This paper presents various techniques for making …


Training language models for deeper understanding improves brain alignment

KL Aw, M Toneva - arXiv preprint arXiv:2212.10898, 2022

… sentences on the C4 dataset, a collection of roughly 750 GB of English texts sourced from the public Common Crawl open repository. … , a collection of roughly 750 GB of English texts sourced from the public Common Crawl open repository …


TRANS-KBLSTM: An External Knowledge Enhanced Transformer BiLSTM model for Tabular Reasoning

Natural language inference on tabular data is a challenging task. Existing approaches lack the world and common sense knowledge required to perform at a human level. While massive amounts of KG data exist, approaches to integrate them …


Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling

K Song, Y Leng, X Tan, Y Zou, T Qin, D Li - arXiv preprint arXiv:2205.12986, 2022

… WMT14 English-German comprises 4.5M bilingual data collected from Europarl v7, Common Crawl corpus and News Commentary. We concatenate newstest2012 and newstest2013 as the valid set, and choose newstest2014 as the test set for WMT14 …


Transfer Effects in Scandinavian Languages

The field of natural language processing with machine learning has gone through a quantum leap in the past decade. Giants like Google and Amazon have paved the way for powerful models trained on a vast amount of data. Even though the …


Transfer language selection for zero-shot cross-lingual abusive language detection

J Eronen, M Ptaszynski, F Masui, M Arata, G Leliwa… - Information Processing & …, 2022

We study the selection of transfer languages for automatic abusive language detection. Instead of preparing a dataset for every language, we demonstrate the effectiveness of cross-lingual transfer learning for zero-shot abusive language …


Transfer Learning Approach to Prediction of Rate of Penetration in Drilling

FJ Pacis, S Alyaev, A Ambrus, T Wiktorski - International Conference on …, 2022

The rate of penetration (ROP) is a key performance indicator in the oil and gas drilling industry as it directly translates to cost savings and emission reductions. A prerequisite for a drilling optimization algorithm is a predictive model that provides …


Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models

S Nair, E Yang, D Lawrie, K Duh, P McNamee… - arXiv preprint arXiv …, 2022

… We also conducted experiments using the new CLIR Common Crawl Collection (HC4) [14], where the documents are newswire articles from Common Crawl in Chinese or Persian. Throughout, English queries are used to search a collection in a non-English …


Transfer Learning for Automatic Sexism Detection with Multilingual Transformer Models

D Liakhovets, M Schütz, J Böck, M Andresel… - 2022

… • XLM-R is also a multilingual model and, similarly to mBERT, is trained on 100 languages with a dataset containing CommonCrawl data. The architecture consists of a combination of the XLM [24] and RoBERTa [25] transformer models. The latter is …


Transfer Learning in Natural Language Processing through Interactive Feedback

Machine learning models cannot easily adapt to new domains and applications. This drawback becomes detrimental for natural language processing (NLP) because language is perpetually changing. Across disciplines and languages, there are …


Transfer Learning of Transformers for Spoken Language Understanding

J Švec, A Frémund, M Bulín, J Lehečka - … Conference on Text, Speech, and Dialogue, 2022

… For the Czech language, we have collected the CommonCrawl corpus at the end of August 2020. It contains 47.9 GB of clean text, 20.5 M unique URLs and 6.7 B running words. The Czech T5 training procedure followed the original procedure …


Transformer Language Models without Positional Encodings Still Learn Positional Information

A Haviv, O Ram, O Press, P Izsak, O Levy - arXiv preprint arXiv:2203.16634, 2022

Transformers typically require some form of positional encoding, such as positional embeddings, to process natural language sequences. Surprisingly, we find that transformer language models without any explicit positional encoding are still …


Transformer models used for text-based question answering systems

K Nassiri, M Akhloufi - Applied Intelligence, 2022

The question answering system is frequently applied in the area of natural language processing (NLP) because of the wide variety of applications. It consists of answering questions using natural language. The problem is, in general, solved by …


Transformer-Based Abstractive Summarization for Reddit and Twitter: Single Posts vs. Comment Pools in Three Languages

IS Blekanov, N Tarasov, SS Bodrunova - Future Internet, 2022

… One of the major contributions of the T5 model’s authors was the process of collecting and preprocessing data based on the CommonCrawl initial dataset. As a result of multiple steps of processing, including duplicate, stopwords, and special …


Translation between Molecules and Natural Language

C Edwards, T Lai, K Ros, G Honke, H Ji - arXiv preprint arXiv:2204.11817, 2022

Joint representations between images and text have been deeply investigated in the literature. In computer vision, the benefits of incorporating natural language have become clear for enabling semantic-level control of images. In this work, we present …


Tree-based Search Graph for Approximate Nearest Neighbor Search

X Fan, X Wang, K Lu, L Xue, J Zhao - arXiv preprint arXiv:2201.03237, 2022

Nearest neighbor search supports important applications in many domains, such as database, machine learning, computer vision. Since the computational cost for accurate search is too high, the community turned to the research of approximate …


TrollsWithOpinion: A taxonomy and dataset for predicting domain-specific opinion manipulation in troll memes

S Suryawanshi, BR Chakravarthi, M Arcan, P Buitelaar - Multimedia Tools and …, 2022

Memes have become a de-facto media device in online communication. Unfortunately, memes are also used for trolling, which intends to demean, harass, or bully targeted individuals. As a result of which, the targeted individual could fall prey …


Trustworthiness of Health-Related Information on Arabic Social Media

Social media (SM) platforms play a vital role in disseminating health-related information. However, evidence suggests that Twitter posts (ie, tweets) are often inaccurate; for example, research from Saudi Arabia indicates that 50% of health-related …


TSMind: Alibaba and Soochow University's Submission to the WMT22 Translation Suggestion Task

X Ge, K Wang, J Wang, N Xiao, X Duan, Y Zhao… - arXiv preprint arXiv …, 2022

This paper describes the joint submission of Alibaba and Soochow University, TSMind, to the WMT 2022 Shared Task on Translation Suggestion (TS). We participate in the English-German and English-Chinese tasks. Basically, we utilize …


Tune in: The afrl wmt21 news-translation systems

G Erdmann, J Gwinnup, T Anderson - Proceedings of the Sixth Conference on …, 2021

… The corpus used for the initial model consisted of commoncrawl, paracrawl v1, and newscommentary-v13 from wmt19 and was processed … Dirt cheap web-scale parallel text from the common crawl. In Proceedings of the 51st Annual Meeting of …


Turkish abstractive text summarization using pretrained sequence-to-sequence models

B Baykara, T Güngör - Natural Language Engineering, 2022

The tremendous amount of increase in the number of documents available on the Web has turned finding the relevant piece of information into a challenging, tedious, and time-consuming activity. Accordingly, automatic text summarization has become …


TWEET-FID: An Annotated Dataset for Multiple Foodborne Illness Detection Tasks

R Hu, D Zhang, D Tao, T Hartvigsen, H Feng… - arXiv preprint arXiv …, 2022

Foodborne illness is a serious but preventable public health problem -- with delays in detecting the associated outbreaks resulting in productivity loss, expensive recalls, public safety hazards, and even loss of life. While social media is a promising source …


TweetNLP: Cutting-Edge Natural Language Processing for Social Media

J Camacho-Collados, K Rezaee, T Riahi, A Ushio… - arXiv preprint arXiv …, 2022

In this paper we present TweetNLP, an integrated platform for Natural Language Processing (NLP) in social media. TweetNLP supports a diverse set of NLP tasks, including generic focus areas such as sentiment analysis and named entity …


TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations

X Zhang, Y Malkov, O Florez, S Park, B McWilliams… - arXiv preprint arXiv …, 2022

We present TwHIN-BERT, a multilingual language model trained on in-domain data from the popular social network Twitter. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision, but also …


Two-Stream Network for Sign Language Recognition and Translation

Y Chen, R Zuo, F Wei, Y Wu, S Liu, B Mak - arXiv preprint arXiv:2211.01367, 2022

Sign languages are visual languages using manual articulations and non-manual elements to convey information. For sign language recognition and translation, the majority of existing approaches directly encode RGB videos into hidden …


Uncertainty Estimation for Language Reward Models

A Gleave, G Irving - arXiv preprint arXiv:2203.07472, 2022

Language models can learn a range of capabilities from unsupervised training on text corpora. However, to solve a particular problem (such as text summarization) it is typically necessary to fine-tune them on a task-specific dataset. It is often easier for …


Uncertainty-Aware Cross-Lingual Transfer with Pseudo Partial Labels

S Lei, X Zhang, J He, F Chen, CT Lu - Findings of the Association for Computational …, 2022

… 2020) improves over mBERT by training longer with more data from CommonCrawl, and without the NSP objective. Recently, two self-learning based methods were proposed for cross-lingual transfer. Dong and de Melo …


Understanding HTML with Large Language Models

I Gur, O Nachum, Y Miao, M Safdari, A Huang… - arXiv preprint arXiv …, 2022

… For this task, we derive a dataset from CommonCrawlCommonCrawl does not include renderings or annotations that would reveal what text in the HTML is associated with which elements. Instead, we infer descriptions of various elements …


UniCon: Unidirectional Split Learning with Contrastive Loss for Visual Question Answering

Y Sun, H Ochiai - arXiv preprint arXiv:2208.11435, 2022

Visual question answering (VQA) that leverages multi-modality data has attracted intensive interest in real-life applications, such as home robots and clinic diagnoses. Nevertheless, one of the challenges is to design robust learning for different client …


Unified Pretraining Framework for Document Understanding

J Gu, J Kuen, VI Morariu, H Zhao, N Barmpalios, R Jain… - arXiv preprint arXiv …, 2022

Document intelligence automates the extraction of information from documents and supports many business applications. Recent self-supervised learning methods on large-scale unlabeled document datasets have opened up promising directions …


Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks

J Lu, C Clark, R Zellers, R Mottaghi, A Kembhavi - arXiv preprint arXiv:2206.08916, 2022

We propose Unified-IO, a model that performs a large variety of AI tasks spanning classical computer vision tasks, including pose estimation, object detection, depth estimation and image generation, vision-and-language tasks such as region …


Universal Conditional Masked Language Pre-training for Neural Machine Translation

P Li, L Li, M Zhang, M Wu, Q Liu - arXiv preprint arXiv:2203.09210, 2022

Pre-trained sequence-to-sequence models have significantly improved Neural Machine Translation (NMT). Different from prior works where pre-trained models usually adopt an unidirectional decoder, this paper demonstrates that pre-training a …


UNLT: Urdu Natural Language Toolkit

J Shafi, HR Iqbal, RMA Nawab, P Rayson - Natural Language Engineering, 2022

This study describes a Natural Language Processing (NLP) toolkit, as the first contribution of a larger project, for an under-resourced language—Urdu. In previous studies, standard NLP toolkits have been developed for English and many other …


Unsupervised Context Aware Sentence Representation Pretraining for Multi-lingual Dense Retrieval

N Wu, Y Liang, H Ren, L Shou, N Duan, M Gong… - arXiv preprint arXiv …, 2022

Recent research demonstrates the effectiveness of using pretrained language models (PLM) to improve dense retrieval and multilingual dense retrieval. In this work, we present a simple but effective monolingual pretraining task called …


Unsupervised Dense Retrieval Deserves Better Positive Pairs: Scalable Augmentation with Query Extraction and Generation

R Meng, Y Liu, S Yavuz, D Agarwal, L Tu, N Yu… - arXiv preprint arXiv …, 2022

… In this section, we introduce several data argumentation methods to create pseudo queries given a document (eg, a passage from a Wikipedia article or a web document from CommonCrawl), without using any annotated queries or questions …


Unsupervised Extreme Multi Label Classification of Stack Overflow Posts

Knowing the topics of a software forum post, such as those on StackOverflow, allows for greater analysis and understanding of the large amounts of data that come from these communities. One approach to this problem is using extreme multi label …


Unsupervised Sentence Textual Similarity with Compositional Phrase Semantics

Z Wang, J Dou, Y Zhang - arXiv preprint arXiv:2210.02284, 2022

… 2014): We use the 300D GloVe vectors trained on Common Crawl (840B tokens, 2.2M vocabulary) 15. • fastText (Bojanowski et al.… 2017): We use the 300D fastText vectors trained on Common Crawl (600B tokens) without subword information 16. …


Unsupervised Visual Domain Adaptation by Self-Guided Learning

… video action understanding Common Crawl 2020 … In the field of natural language processing, the training datasets used by the leading model GPT-3 [16] include the 410 billion token Common Crawl archive 1 and 86 billion tokens from …


UoR-NCL at SemEval-2022 Task 3: Fine-Tuning the BERT-Based Models for Validating Taxonomic Relations

T Markchom, H Liang, J Chen - Proceedings of the 16th International Workshop on …, 2022

In human languages, there are many presuppositional constructions that impose a constrain on the taxonomic relations between two nouns depending on their order. These constructions create a challenge in validating taxonomic relations in real-world …


UPV at TREC Health Misinformation Track 2021

Health misinformation on search engines is a significant problem that could negatively affect individuals or public health. To mitigate the problem, TREC organizes a health misinformation track. This paper presents our submissions to this …


Use the Metadata, Luke!–An Experimental Joint Metadata Search and N-gram Trend Viewer for Personal Web Archives

B Indig, Z Sárközi-Lindner, M Nagy - Proceedings of the 2nd International Workshop …, 2022

… This puts pressure on website operators, as everyone is forced to create an archive of their own, the very issue that was intended to be avoided by shared archives such as Common Crawl that have become trapped in a rivalry of quantity …


Using a Pre-trained SimpleT5 Model for Text Simplification in a limited Corpus

J Monteiro, M Aguiar, S Araújo - Proceedings of the Working Notes of CLEF, 2022

In this paper, we describe our approach for solving Task 3 of the SimpleText Lab, organized as part of the Clef 2022 conference. The SimpleText Lab addresses issues of automatic text simplification of scientific texts in order to make scientific …


Using Context-Free Grammar to Generate Synthetic Technical Short Texts

T Bikaun, M Stewart, M Hodkiewicz - Australasian Joint Conference on Artificial …, 2022

… Pre-trained language models take advantage of hundreds of millions of open-domain training examples derived from sources such as the English Wikipedia, Toronto BookCorpus, and Common Crawl Corpus to learn powerful representations over …


Using Contextual Information to Process Out-of-Vocabulary Words in Spoken Dialog Systems

Speech is a very efficient channel for communication, but it is a challenge to cope with words which are unknown to one of the dialogue partners. Human beings are very good at dealing with words that are not in their vocabulary, for example by …


Using Deep Learning Techniques, A Framework for Estimating the Nutritional Value of Food in Real TIME

S Bahadur, R Kushwaha, S Sharma, SG Anuradha - International Journal of Health Sciences

… The HTML format is used to store the raw text data acquired from Google and Common Crawl. HTML tags, CSS, JavaScript code, and comments have been removed during the preprocessing process. To extract individual words, such meta-data …


Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

S Smith, M Patwary, B Norick, P LeGresley… - arXiv preprint arXiv …, 2022

… Other Datasets: In addition to Common Crawl data, we leveraged a number of other previously generated datasets. From The Pile, we … To compound this issue, the URLs scraped in different Common Crawl snapshots are not necessarily unique …


Using distributional thesaurus to enhance transformer-based contextualized representations for low resource languages

G Venkatesh, A Jana, S Remus, Ö Sevgili… - Proceedings of the 37th …, 2022

… XLM-RoBERTa [8] is a large multilingual model, based on the RoBERTa architecture, pretrained on 2.5TB of filtered CommonCrawl data. Experiments confirm that it outperforms mBERT on a wide range of cross-lingual tasks. Alongside …


Using Emotion Embeddings to Transfer Knowledge Between Emotions, Languages, and Annotation Formats

G Chochlakis, G Mahajan, S Baruah, K Burghardt… - arXiv preprint arXiv …, 2022

… It decreases α to 0.3 and switches to the CommonCrawl dataset, which has a more balanced language distribution. It also disposes of language embeddings and TLM. XLM-T [2] extends XLM-R by finetuning it on tweets to perform multilingual …


Using Information-Seeking Argument Mining to Improve Service

B Skiera, S Yan, J Daxenberger, M Dombois… - Journal of Service Research, 2022

If service providers can identify reasons users are in favor of or against a service, they have insightful information that can help them understand user behavior and what they need to do to change such behavior. This article argues that the novel text-mining …


Using Informative Data Subsets for Efficient Training of Large Language Models: An Initial Study

HK Renduchintala, K Killamsetty, S Bhatia, M Aggarwal…

Abstract Language Models (LMs) are pretrained on large unlabeled corpora through selfsupervision tasks and have become ubiquitous to several NLP applications. Recent trends indicate that the generalization capability of Large LMs (LLMs) …


Using Keyqueries to Reduce Misinformation in Health-Related Search Results

M Fröbe, S Günther, A Bondarenko, J Huck, M Hagen - 2022

In the scenario of health-related searches, we investigate whether explicit relevance feedback by experts can guide query expansion methods to formulate queries that return fewer misleading or wrong results. In contrast to standard query expansion …


Using Natural Language Processing Techniques to Improve Manual Test Case Descriptions

Despite the recent advancements in test automation, software testing often remains a manual, and costly, activity in many industries. Manual test cases, often described only in natural language, consist of one or more test steps, which are instructions …


Using Transformer models for gender attribution in Polish

K Kaczmarek, J Pokrywka, F Graliński - 2022 17th Conference on Computer Science …, 2022

… We searched the contents of CommonCrawl 2019 (as provided to us by the authors of [6]2) for the six-gram fragments of the HSSS test set, obviously taking into account the fact that feminine gender-specific forms were modified during the …


Utilizing Deep Learning for Detecting Adverse Drug Events in Structured and Unstructured Regulatory Drug Data Sets

BM Knisely, Q Hatim, M Vaughn-Cooke - Pharmaceutical Medicine, 2022

Background The US Food and Drug Administration (FDA) collects and retains several data sets on post-market drugs and associated adverse events (AEs). The FDA Adverse Event Reporting System (FAERS) contains millions of AE reports …


Utilizing subjectivity level to mitigate identity term bias in toxic comments classification

Z Zhao, Z Zhang, F Hopfgartner - Online Social Networks and Media, 2022

Toxic comment classification models are often found biased towards identity terms, ie, terms characterizing a specific group of people such as “Muslim” and “black”. Such bias is commonly reflected in false positive predictions, ie, non-toxic comments with …


UTSA NLP at SemEval-2022 Task 4: An Exploration of Simple Ensembles of Transformers, Convolutional, and Recurrent Neural Networks

X Zhao, A Rios - arXiv preprint arXiv:2203.14920, 2022

The act of appearing kind or helpful via the use of but having a feeling of superiority condescending and patronizing language can have have serious mental health implications to those that experience it. Thus, detecting this condescending and …


UWaterlooMDS at the TREC 2021 Health Misinformation Track

M ABUALSAUD, IX CHEN, K GHAJAR, LNHI PHAN…

… Using the hosts in M as a base, we expanded this list using the common crawl host graph 8. The hosts graph contains roughly 4 million nodes … We do this by calculating PageRank scores in a subset of the common crawl host-level graph. The …


Vega-MT: The JD Explore Academy Translation System for WMT22

C Zan, K Peng, L Ding, B Qiu, B Liu, S He, Q Lu… - arXiv preprint arXiv …, 2022

C Zanℜ, K Peng, L Dingℜ, B Qiu, B Liu, S He, Q Lu…

We describe the JD Explore Academy's submission of the WMT 2022 shared general translation task. We participated in all high-resource tracks and one medium-resource track, including Chinese-English, German-English, Czech-English, Russian-English …


Vicomtech at DA-VINCIS: Detection of Aggressive and Violent Incidents from Social Media in Spanish

P Turón, N Perez, A García-Pablos, E Zotova… - 2022

This paper describes the participation of the Vicomtech NLP team in the DA-VINCIS shared task. This shared task is focused on mentions of violent events in Spanish tweets, and proposes two subtasks: first, detecting whether a violent incident is …


Video Games as a Corpus: Sentiment Analysis using Fallout New Vegas Dialog

M Hämäläinen, K Alnajjar, T Poibeau - 2022

We present a method for extracting a multilingual sentiment annotated dialog data set from Fallout New Vegas. The game developers have preannotated every line of dialog in the game in one of the 8 different sentiments: anger, disgust, fear, happy …


Vietnamese Hate and Offensive Detection using PhoBERT-CNN and Social Media Streaming Data

K Quoc Tran, A Trong Nguyen, PG Hoang, CD Luu… - Neural Computing and …, 2022

KQ Tran, AT Nguyen, PG Hoang, CD Luu, TH Do… - arXiv preprint arXiv …, 2022

… XLM-RoBERTa (XLM-R) [28]: is a multilingual model trained using over two terabytes of cleaned and filtered CommonCrawl data. Upsampling low-resource languages during training and vocabulary generation, generating a more extensive …


Visualization of 2D fractal structures associated with the Riemann zeta function

I Belovas, M Sabaliauskas, L Kuzma - DAMSS: 13th conference on data analysis …, 2022

DAMSS-2022 is the 13th International Conference on Data Analysis Methods for Software Systems, held in Druskininkai, Lithuania. Every year at the same place and time. The exception was in 2020, when the world was gripped by the Covid-19 …


Visuelle Exploration von indirekten Befangenheiten bei der Verarbeitung natürlicher Sprachen durch Transformer Modelle

… Common Crawl corpora are composed of text dataset collected from web pages and contains several billion tokens … This metric was used to measure the biases within the Common Crawl pre-trained GloVe model, an ELMo model, and the bert-base …


ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generation

L Phan, H Tran, H Nguyen, TH Trinh - arXiv preprint arXiv:2205.06457, 2022

We present ViT5, a pretrained Transformer-based encoder-decoder model for the Vietnamese language. With T5-style self-supervised pretraining, ViT5 is trained on a large corpus of high-quality and diverse Vietnamese texts. We benchmark ViT5 on …


VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

H Ding, C Liu, S Wang, X Jiang - IEEE Transactions on Pattern Analysis and Machine …, 2022

We propose a Vision-Language Transformer (VLT) framework for referring segmentation to facilitate deep interactions among multi-modal information and enhance the holistic understanding to vision-language features. There are different …


Vocabulary Volume: A New Metric for Assessing Vocabulary Knowledge

This paper presents Vocabulary Volume, a new metric to assess vocabulary knowledge. The existing metrics for vocabulary knowledge assessment rely on word difficulty, which is often defined in terms of the use frequency of words. In addition to …


Voices of First Responders—Nationwide Public Safety Communication Survey Findings

Public safety is in the midst of a major transition in the communication technology that is used due to the ongoing development and deployment of the Nationwide Public Safety Broadband Network (NPSBN). Once established, the NPSBN will …


Voicing challenges: GDPR and AI research

K Quezada-Tavarez, L Dutkiewicz, N Krack - Open Research Europe, 2022

EU data protection rules could be difficult for researchers to navigate, particularly when processing massive datasets containing personal data for Artificial Intelligence (AI) developments. This article examines how data protection intersects with AI …


WarpGate: A Semantic Join Discovery System for Cloud Data Warehouses

T Cong, J Gale, J Frantz, HV Jagadish, Ç Demiralp

… In the database community, Web table corpora [11, 18] extracted from the Common Crawl contribute the largest publicly accessible relational table datasets. Efforts exploring representation learning for relational tables [8, 17] have trained …


Web Data Commons-Schema. org Table Annotation Benchmark

… The Web Data Commons project regularly extracts such data from the Common Crawl and provides it for public download. The Schema.org Table Corpus was created using schema.org data from the December 2020 version of the Web Data …


Web-Based Communication of Wooden Sport Equipment: An Analysis Based on Six Olympic Sports

F Negro, S Blanc, S Bruzzese, A Falaschi, F Ruffinatto… - Forests, 2022

Wood has traditionally been used to make sports equipment and playing surfaces. While its use is outdated in several sports, in many others, wood remains well-established or even the material of choice, eg, to make basketball floors or table tennis rackets …


Webformer: Pre-training with Web Pages for Information Retrieval

Y Guo, Z Ma, J Mao, H Qian, X Zhang, H Jiang, Z Cao… - Proceedings of the 45th …, 2022

Pre-trained language models (PLMs) have achieved great success in the area of Information Retrieval. Studies show that applying these models to ad-hoc document ranking can achieve better retrieval effectiveness. However, on the Web, most …


WebFormer: The Web-page Transformer for Structure Information Extraction

Q Wang, Y Fang, A Ravula, F Feng, X Quan, D Liu - arXiv preprint arXiv:2202.00217, 2022

… We evaluate WebFormer on SWDE and Common Crawl benchmarks, which shows superior performance over several state-of-the-art methods. The experimental results also demonstrate the effectiveness of WebFormer in modeling …


Webis at TREC 2021: Deep Learning, Health Misinformation, and Podcasts Tracks

A Bondarenko, M Fröbe, M Gohsen, S Günther…

… Mean and PageRank of the domain provided by the Common Crawl,2 as well as six query … up to 1000 anchor texts extracted from six Common Crawl snapshots from the years 2016 to … Here, we use only anchor texts extracted from the …


WeLM: A Well-Read Pre-trained Language Model for Chinese

H Su, X Zhou, H Yu, Y Chen, Z Zhu, Y Yu, J Zhou - arXiv preprint arXiv:2209.10372, 2022

… monthly shards released by Common Crawl to construct a general… common crawl. In Figure 2, we visualize the document length and topic distribution in the training data. Topics are detected with pre-trained topic classification models. We …


What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structured

A Henlein, A Mehler - arXiv preprint arXiv:2204.05673, 2022

Transformer-based models are now predominant in NLP. They outperform approaches based on static models in many respects. This success has in turn prompted research that reveals a number of biases in the language models …


What is it like to program with artificial intelligence?

A Sarkar, AD Gordon, C Negreanu, C Poelitz… - arXiv preprint arXiv …, 2022

… For example, the Common Crawl project1 produces around 20 TB of text data (from web pages) monthly, but labelled task-specific data is less prevalent. This makes unsupervised training appealing which leads to the concept of pre-trained LMs (J. Li …


What Is Love?: Text Analytics on Romance Literature From the Perspective of Authors

CH Naing, X Zhao, KH Gan, NH Samsudin - … of Research on Opinion Mining and Text …, 2022

Descriptions of love can be found in a wide range of literature. The meaning of love that a reader grasps from reading a literary work is mostly the result of self-understanding and is very likely different from the one that the author tried to express. Therefore, it …


What Language Model to Train if You Have One Million GPU Hours?

T Le Scao, T Wang, D Hesslow, L Saulnier, S Bekman… - Challenges {\&, 2022

… crawls with curated highquality sources significantly improves zeroshot generalization over pretraining datasets constructed from Common Crawl only. … First, we have found that complimenting Common Crawl data with high-quality cross-domain …


WHAT NEURAL NETWORKS KNOW ABOUT LINGUISTIC COMPLEXITY

SA Sharoff - Russian Journal of Linguistics, 2022

SS Aleksandrovich - Russian Journal of Linguistics, 2022

Linguistic complexity is a complex phenomenon, as it manifests itself on different levels (complexity of texts to sentences to words to subword units), through different features (genres to syntax to semantics), and also via different tasks (language …


What Really Matters? Evaluating the Importance of Skills for Data Analysts

… First, the job postings utilized in Dong and Triche’s study were not comprehensive but rather only those captured by the Common Crawl tool to facilitate the longitudinal nature of their study. Second, the wildcard searches used by Dong and Triche …


When danger strikes: A linguistic tool for tracking America's collective response to threats

VK Choi, S Shrestha, X Pan, MJ Gelfand - Proceedings of the National Academy of …, 2022

In today’s vast digital landscape, people are constantly exposed to threatening language, which attracts attention and activates the human brain’s fear circuitry. However, to date, we have lacked the tools needed to identify threatening language …


When Pairs Meet Triplets: Improving Low-Resource Captioning via Multi-Objective Optimization

Y Wu, S Zhao, Y Zhang, X Yuan, Z Su - ACM Transactions on Multimedia Computing …, 2022

Image captioning for low-resource languages has attracted much attention recently. Researchers propose to augment the low-resource caption dataset into (image, rich-resource language, and low-resource language) triplets and develop the dual attention …


Where do I Belong in Six Centuries of Literature?

J Silva, S Grilo, M Bolrinha, R Santos, L Gomes…

Enhancing the availability of corpora and processing tools for language research is a central endeavour of the CLARIN research infrastructure. In this chapter we report on how PORTULAN CLARIN, with the support of the national institute for the …


Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines

A Isenko, R Mayer, J Jedele, HA Jacobsen - arXiv preprint arXiv:2202.08679, 2022

Preprocessing pipelines in deep learning aim to provide sufficient data throughput to keep the training processes busy. Maximizing resource utilization is becoming more challenging as the throughput of training processes increases with hardware …


Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection

S Gururangan, D Card, SK Drier, EK Gade, LZ Wang… - arXiv preprint arXiv …, 2022

… Web dumps like Common Crawl offer the promise of more diverse text than what is available in curated resources. However, much of the web consists of frequently replicated boilerplate (eg, privacy policies), code (eg, HTML and Javascript) …


Why Do Neural Language Models Still Need Commonsense Knowledge to Handle Semantic Variations in Question Answering?

S Kwon, C Kang, J Han, J Choi - arXiv preprint arXiv:2209.00599, 2022

… Finally, RoBERTa is pretrained with approximately 160GB of data including the BERT pretraining corpora as well as three additional corpora (CommonCrawl News dataset27, open-source recreation of the WebText corpus28 and STORIES corpus29) …


Why Globally Re-shuffle? Revisiting Data Shuffling in Large Scale Deep Learning

TT Nguyen, F Trahay, J Domke, A Drozd, E Vatai… - IEEE International Parallel & …, 2022

Stochastic gradient descent (SGD) is the most prevalent algorithm for training Deep Neural Networks (DNN). SGD iterates the input data set in each training epoch processing data samples in a random access fashion. Because this puts enormous …


WikiMulti: a Corpus for Cross-Lingual Summarization

P Tikhonov, V Malykh - arXiv preprint arXiv:2204.11104, 2022

Cross-lingual summarization (CLS) is the task to produce a summary in one particular language for a source document in a different language. We introduce WikiMulti - a new dataset for cross-lingual summarization based on Wikipedia …


Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning

P Villalobos, J Sevilla, L Heim, T Besiroglu… - arXiv preprint arXiv …, 2022

… 5) CommonCrawl CommonCrawl releases monthly crawls which for the past four years have contained 5-10TB of compressed plaintext.Half of the URLs in … At 200M words per GB of text, CommonCrawl produces between 8.3e12 and 4.4e13 …


Within-pair factors might explain the inconsistent effects of animacy on paired-associates recall

MJ Serra, CM DeYoung - Psychonomic Bulletin & Review, 2022

The tendency for people to have better memory for animate (living) concepts than inanimate (nonliving) concepts in memory tasks involving free recall and recognition suggests that animacy status can be an important predictor of memory. To date …


Witscript 2: A System for Generating Improvised Jokes Without Wordplay

A previous paper presented Witscript, a system for generating conversational jokes that rely on wordplay. This paper extends that work by presenting Witscript 2, which uses a large language model to generate conversational jokes that rely on common …


Word Embeddings for Automatic Equalization in Audio Mixing

S Venkatesh, D Moffat, ER Miranda - arXiv preprint arXiv:2202.08898, 2022

… It trains on the World Wide Web using Common Crawl, which is a larger corpus of text. Tok2Vec is a word embedding model provided by a company called spaCy [31]. We did not find the entire details regarding its implementation, but the model is …


Word formation supports efficient communication: The case of compounds

Compounding is a common type of word formation extensively studied in linguistics and cognitive psychology. A growing line of research suggests that the lexicon supports efficient communication by balancing informativeness and simplicity. We …


Work-in-Progress: Computing Sentence Similarity for Short Texts using Transformer models

V Ramnarain-Seetohul, V Bassoo, Y Rosunally - 2022 IEEE Global Engineering …, 2022

… The OpenAI GPT-3, an autoregressive language model with 175 billion parameters, is trained with the Common Crawl corpus of data, web text, books, and Wikipedia [10]. The OpenAI offers an API to use the GPT-3 model for registered users …


WuDaoMM: A large-scale Multi-Modal Dataset for Pre-training models

S Yuan, Z Shuai, L Jiahong, X Zhao, Z Hanyu, T Jie - arXiv preprint arXiv:2203.11480, 2022

… LAION-400M is a larger-scale vision-language corpus, which uses Common Crawl as the data source and applies a series of exquisite data cleaning rules. In contrast, Chinese-based multi-modal datasets are relatively rare and the scale of …


XDoc: Unified Pre-training for Cross-Format Document Understanding

J Chen, T Lv, L Cui, C Zhang, F Wei - arXiv preprint arXiv:2210.02849, 2022

… 2022a), we take advantage of the large-scale dataset Common Crawl3, which contains petabytes of web pages in raw formats. Specifically, text contents and HTML tags are both available for each web page. According to (Li et al., 2022a), the …


XLCNN: pre-trained transformer model for malware detection

Κ Γιαπαντζής - 2022

The present thesis describes a Transformer-based neural network model that was developed in order to detect malicious software. We believe that the scientific community should take advantage of the contribution of Transformer models in the …


XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond

Abstract Language models are ubiquitous in current NLP, and their multilingual capacity has recently attracted considerable attention. However, current analyses have almost exclusively focused on (multilingual variants of) standard benchmarks …


XTREME-S: Evaluating Cross-lingual Speech Representations

A Conneau, A Bapna, Y Zhang, M Ma, P von Platen… - arXiv preprint arXiv …, 2022

We introduce XTREME-S, a new benchmark to evaluate universal cross-lingual speech representations in many languages. XTREME-S covers four task families: speech recognition, classification, speech-to-text translation and retrieval. Covering …


Ynu-hpcc at semeval-2022 task 2: Representing multilingual idiomaticity based on contrastive learning

K Liu, J Wang, X Zhang - Proceedings of the 16th International Workshop on …, 2022

This paper will present the methods we use as the YNU-HPCC team in the SemEval-2022 Task 2, Multilingual Idiomaticity Detection and Sentence Embedding. We are involved in two subtasks, including four settings. In subtask B of sentence …


You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings

Z Talat, A Névéol, S Biderman, M Clinciu, M Dey… - Challenges {\&, 2022

Evaluating bias, fairness, and social impact in monolingual language models is a difficult task. This challenge is further compounded when language modeling occurs in a multilingual context. Considering the implication of evaluation biases for large …


Zero Shot Crosslingual Eye-Tracking Data Prediction using Multilingual Transformer Models

Eye tracking data during reading is a useful source of information to understand the cognitive processes that take place during language comprehension processes. Different languages account for different brain triggers, however there seems to be …


Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization

ZY Dou, N Peng - arXiv preprint arXiv:2201.00136, 2022

… Concretely, first, they create a cloze question corpus by masking noun phrases and named entities in statements sampled from Wikipedia, and a natural question corpus by mining questions containing some common wh-words from CommonCrawl …


Zero-Shot Ranking Socio-Political Texts with Transformer Language Models to Reduce Close Reading Time

K Akdemir, A Hürriyetoğlu - arXiv preprint arXiv:2210.09179, 2022

We approach the classification problem as an entailment problem and apply zero-shot ranking to socio-political texts. Documents that are ranked at the top can be considered positively classified documents and this reduces the close reading time …


Zhichunroad at semeval-2022 task 2: Adversarial training and contrastive learning for multiword representations

X Cui, W Xiong, S Wang - Proceedings of the 16th International Workshop on …, 2022

This paper presents our contribution to the SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding. We explore the impact of three different pre-trained multilingual language models in the SubTaskA. By enhancing …