(Commercial) Automatic Speech Recognition as a Tool in Sociolinguistic Research

N Markl - University of Pennsylvania Working Papers in …, 2022

As speech datasets used in sociolinguistic research increase in size, laborious and time-intensive manual orthographic transcription is a challenge, limiting the amount of (transcribed) data which can be analysed. In this paper, I discuss the use of (commercial) …


2 Related Work

M Kasri, M Birjali, M Nabil, A Beni-Hssane, A El-Ansari…

Word embedding aims to learn distributed vector representations of words by exploiting the existence of a huge amount of contextual information in a large text corpus using different techniques. This idea starts with [30]. They proposed a neural …


22 ZYX

As is the case with many people, my first direct lessons about failure and errors happened at kindergarten and school. In my case, it was primarily due to my (back then undiagnosed) ADHD and, since it wasn’t my mother tongue, my precarious …


4 Touché Task 1: Conversational Argument Retrieval

A Bondarenko, M Fröbe, M Beloucif, L Gienapp…

… Other argument retrieval systems, such as ArgumenText [40] and TARGER [9], use the larger Common Crawl, requiring additionally also argument … scenarios, based on billions of sentences from the Common Crawl, however, it still lacks a …


5 The Role of Language Prediction Models in Contractual Interpretation: The Challenges and Future Prospects of GPT-3

M Katrak - Legal Analytics: The Future of Analytics in Law, 2022

… For training GPT-3, researchers used the Common Crawl dataset, which constituted nearly a trillion words. The Common Crawl dataset contains petabytes of data collected over the years and includes webpage data, text extracts, and …


" Way back then": A Data-driven View of 25+ years of Web Evolution

V Agarwal, N Sastry - arXiv preprint arXiv:2202.08239, 2022

… For example, the use of historical crawls of the web such as the Common Crawl5. [30] is a brief history of web crawlers. [9] considers … Similarly, the common crawl12 represents a huge dataset of Web URLs and their content, that can provide …


A California Effect For Artificial Intelligence

… (Respectively, they used 4.9 terabytes of English and Chinese text and images, 1.56 trillion words compiled from various datasets, and a combination of English Wikipedia with ten years of Common Crawl data.) As the inconsistent units show …


A code-mixed task-oriented dialog dataset for medical domain

S Dowlagar, R Mamidi - Computer Speech & Language, 2022

In the healthcare domain, medical and patient interactions form a crucial part of the diagnosis. Initially, the AI models developed for healthcare centered only on monolingual data. However, such models do not cater to the multilingual regions …


A Comparative Study of Pre-trained Encoders for Low-Resource Named Entity Recognition

Y Chen, J Mikkelsen, A Binder, C Alt, L Hennig - arXiv preprint arXiv:2204.04980, 2022

Pre-trained language models (PLM) are effective components of few-shot named entity recognition (NER) approaches when augmented with continued pre-training on task-specific out-of-domain data or fine-tuning on in-domain data. However, their …


A Comparative Study of Pre-trained Word Embeddings for Arabic Sentiment Analysis

M Zouidine, M Khalil - 2022 IEEE 46th Annual Computers, Software, and …, 2022

In this paper, we conduct a series of experiments to systematically study both context-independent and context-dependent word embeddings for the purpose of Arabic sentiment analysis. We use pretrained word embeddings as fixed features extractors to provide …


A comparative study on vectorization methods for non-functional requirements classification

P Leelaprute, S Amasaki - Information and Software Technology, 2022

… The pre-trained model with 840 billion tokens of Common Crawl was used. This study used a vectorization method [30] based on the … For fastText, the pre-trained model with Common Crawl and Wikipedia was used. This study also used a …


A Comparative Text Classification Study with Deep Learning-Based Algorithms

Ö Köksal, Ö Akgül - 2022 9th International Conference on Electrical and …, 2022

… The vectors are trained from Wikipedia and Common Crawl, and each has a length of 300. Furthermore, the vectors are generated for n-grams, n length sequences of characters, instead of words. This property allows FastText to …


A Comparison of Approaches for Imbalanced Classification Problems in the Context of Retrieving Relevant Documents for an Analysis

S Wankmüller - arXiv preprint arXiv:2205.01600, 2022

One of the first steps in many text-based social science studies is to retrieve documents that are relevant for the analysis from large corpora of otherwise irrelevant documents. The conventional approach in social science to address this …


A COMPREHENSIVE ANALYSIS OF SUBWORD TOKENIZERS FOR MORPHOLOGICALLY RICH LANGUAGES

… The corpus contains 33 gigabyte (GB) of text which is obtained by filtering the Common Crawl corpus. As shown in Table 4.1, each tokenizer can produce different tokens for the same word. To compare the tokenizers, we used BOUN Treebank [30] …


A Comprehensive Survey of Natural Language Generation Advances from the Perspective of Digital Deception

K Jones, E Altuncu, VNL Franqueira, Y Wang, S Li - arXiv preprint arXiv:2208.05757, 2022

… Common Crawl: The Common Crawl dataset is a massive collection of petabytes of web data currently hosted by Amazon (https://commoncrawl.… Due to its size, Common Crawl has been popularly used in the training and pre-training of many …


A Comprehensive Survey on Multi-hop Machine Reading Comprehension Approaches

A Mohammadi, R Ramezani, A Baraani - arXiv preprint arXiv:2212.04072, 2022

… To decompose multi-hop question 𝑄 to simpler corpus 𝐷, First some candidate sub-question from a simple corpus 𝑆 will be created by mining 10M possible sub-questions from Common Crawl with a classifier. It then trains a decomposition model on the …


A Corpus for Suggestion Mining of German Peer Feedback

Peer feedback in online education becomes increasingly important to meet the demand for feedback in large scale classes, such as eg Massive Open Online Courses (MOOCs). However, students are often not experts in how to write helpful …


A Critique Empirical Evaluation of Relevance Computation for Focused Web Crawlers

JDPNR Mary, S Balasubramanian, RSP Raj - Brazilian Archives of Biology and …, 2022

HIGHLIGHTS This paper presents a survey on focused web crawlers. This paper presents the challenges in focused crawling research. This paper presents the highlights and hindrances of existing focused web crawlers. This paper also …


A Data-driven Approach to Natural Language Processing for Contemporary and Historical French

… Each monthly Common Crawl snapshot is in itself a massive multilingual corpus, where every single file contains data coming from multiple web … Common Crawl has already been successfully used to train language models, even multilingual …


A Deep Learning approach to real-world Entity Linking: extracting and matching organisation mentions from unstructured text

… Specifically, the authors built a clean CommonCrawl Corpus composed of text scraped from the web in 100 different languages, amounting to around 2.5 TB of text data. The amount of data obviously varies from language to language, but Conneau …


A Dual Attention-Based Representation for the Detection of Abusive Language in Texts and Memes. Technical Report No. CCC-22-005

… For word representation we used pretrained fastText embeddings [83], trained with subword information on Common Crawl, which have been recognized as useful for this task according to the study presented in [8]. All the non-BERT based DNNs …


A Format-Aware Reducer for Scriptable Rewriting of PDF Files

P Anantharaman, S Cheung, N Boorman, ME Locasto

Sanitizing untrusted input is a significant unsolved problem in defensive cybersecurity and input handling. Even if we assume that a safe, provably correct parser exists to validate the input syntax, processing logic may still require the …


A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication

AS Luccioni, F Corry, H Sridharan, M Ananny, J Schultz… - 2022

… , given their sheer size (C4 represents 2.3 TB of data, whereas the Common Crawl has 139TB), filtering them is complex and time-consuming, … In practice, documenting and deprecating these datasets is akin to a game of whack-a-mole …


A Gated Recurrent Unit based architecture for recognizing ontology concepts from biological literature

P Devkota, SD Mohanty, P Manda - BioData Mining, 2022

Annotating scientific literature with ontology concepts is a critical task in biology and several other domains for knowledge discovery. Ontology based annotations can power large-scale comparative analyses in a wide range of applications ranging …


A generating model for Finnish nominal inflection using distributional semantics

Finnish nouns are characterized by rich inflectional variation, with obligatory marking of case and number, with optional possessive suffixes and with the possibility of further cliticization. We present a model for the conceptualization of …


A Holistic Approach to Undesired Content Detection in the Real World

T Markov, C Zhang, S Agarwal, T Eloundou, T Lee… - arXiv preprint arXiv …, 2022

… PUB consists of around 90k public examples including both samples from academic datasets and Web data (Common Crawl) labeled by our annotators. SYN adds additional 69k curated synthetic examples. MIX contains all examples in SYN …


A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model

I Lakim, E Almazrouei, IA Alhaol, M Debbah, J Launay - Challenges {\&, 2022

… We use Common Crawl (CC) for acquiring large amounts of web data. Each CC dump is on average around 10TB, and we discard it immediately after processing it. On average, it takes 24 hours to fully process a dump: we used 21 dumps from CC …


A Hybrid Deep Learning Approach to Detect Bangla Social Media Hate Speech

T Ghosh, AAK Chowdhury, M Banna, H Al, M Nahian… - Proceedings of International …, 2022

Social media has become an integral part of our day-to-day life. In our activities or posts on social media, the presence of hate speech written in the native language or English has increased significantly. It often leads to the spread of negativity …


A hybrid feature-based approach for classification of Fake News in Sinhala on social media

… According to research work on evaluating the word embedding for the Sinhala language, FastText trained on the Common Crawl dataset is the best word embedding for Sinhala compared to the Pre-trained Facebook FastText model, Word2Vec, and …


A Hybrid Phishing Detection System Using Deep Learning-based URL and Content Analysis

M Korkmaz, E Kocyigit, OK Sahingoz, B Diri - Elektronika ir Elektrotechnika, 2022

… With a new dataset (1 million URLs, half of which was obtained from PhishTank and the rest from the CommonCrawl database, and the dataset contains 10,000 images) which researchers used in [21], CNN and LSTM were tested in Intelligent …


A hybrid quantum approach to leveraging data from HTML tables

P Jiménez Aguirre, JC Roldán Salvador… - Knowledge and Information …, 2022

P Jiménez, JC Roldán, R Corchuelo - Knowledge and Information Systems, 2022

… Unfortunately, a recent analysis of the 32.04 million domains in the November 2019 Common Crawl has revealed that only 11.92 million domains provide such tags [5], which means that there are roughly 20.12 million domains that do not …


A Knowledge-Enhanced Adversarial Model for Cross-lingual Structured Sentiment Analysis

Q Zhang, J Zhou, Q Chen, Q Bai, J Xiao, L He - arXiv preprint arXiv:2205.15514, 2022

… XLM-RoBERTa model is pre-trained on the 100 languages with 2.5TB of filtered CommonCrawl data. Moreover, [35] finetuned XLM-RoBERTa models on named-entity-recognition (NER) task with multilingual datasets. However, most existing works use one of …


A Large and Diverse Arabic Corpus for Language Modeling

… The Open Super-large Crawled ALMAnaCH coRpus (OSCAR) is a multi-lingual corpus, extracted from Common Crawl (CC) by performing … N-gram counts and language models from the common crawl. In Proceedings of the Language Resources …


A LEGAL FRAMEWORK FOR ARTIFICIAL INTELLIGENCE FAIRNESS REPORTING

JQ Yap, E Lim - The Cambridge Law Journal

Clear understanding of artificial intelligence (AI) usage risks and how they are being addressed is needed, which require proper and adequate corporate disclosure. We advance a legal framework for AI Fairness Reporting to which companies can and …


A Lite Romanian BERT: ALR-BERT

DC Nicolae, RK Yadav, D Tufiş - Computers, 2022

… OSCAR—OSCAR, or Open Super-large Crawled ALMAnaCH corpora, is a massive multilingual corpus derived from the Common Crawl corpus using language categorization and filtering [16]. There are approximately 11 GB of text in …


A Literature Survey of Recent Advances in Chatbots. Information 2022, 13, 41

… This led to the development of pretrained systems such as BERT (Bidirectional Encoder Representations from transformers) [39] and GPT (Generative Pre-trained Transformer), which were trained with huge language datasets, such as Wikipedia …


A Longitudinal Study of Semantic Networks in Schizophrenia and other Psychotic Disorders using the Word Association Task

AS Pintos, CLM Hui, S De Deyne, C Cheung, WT Ko… - Schizophrenia Bulletin …, 2022

Background and Hypothesis The underpinnings of language deviations in psychotic symptoms (eg, formal thought disorder, delusions) remains unclear. We examined whether the semantic networks underlying word associations are useful predictors of …


A Machine Learning approach for Fake News Detection.

WH Bisen, A Paunikar, B Thakur, A Garg, K Nangliya - International Journal of Next …, 2022

A vital role is played by the media in the circulation of public information about the happenings. A fast spread of information using social media and social networks or websites is possible due to the quick development of the internet. With no worry …


A mechanism for personalized Automatic Speech Recognition for less frequently spoken languages: the Greek case

P Antoniadis, E Tsardoulias, A Symeonidis

Automatic Speech Recognition (ASR) has become increasingly popular since it significantly simplifies human-computer interaction, providing a more intuitive way of communication. Building an accurate, general-purpose ASR system is a challenging …


A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models

J Sun, P Fernandes, X Wang, G Neubig - arXiv preprint arXiv:2210.07111, 2022

… Both ByT5 and mT5 are pretrained on the multilingual Common Crawl (mC4) corpus1 using the span reconstruction objective proposed by Raffel et al. (2020). ByT5 operates on the raw UTF8 bytes of the input without any downsampling …


A Multi-modal Approach to Mining Intent from Code-Mixed Hindi-English Calls in the Hyperlocal-Delivery Domain

J Mathew, P Sahu, B Singhal, A Joshi, KR Medikonda… - International Conference on …, 2022

In this work we outline an approach to mine insights from calls between delivery partners (DP) and customers involved in hyperlocal food delivery in India. Incorrect addresses/ locations or other impediments prompt the DPs to call customers leading …


A Multi-task Multi-modal Framework for Sentiment and Emotion aided Cyberbully Detection

K Maity, A Kumar, S Saha - IEEE Internet Computing, 2022

… The model uses a BERT based architecture pretrained by the Wikipedia and Common Crawl, along with PMINDIA and Dakshina corpora … So to overcome this issue, we have chosen MuRIL BERT, which has been pre-trained by Wikipedia and …


A Multilingual Approach to Scene Text Visual Question Answering

D Karatzas - Document Analysis Systems: 15th IAPR International …

J Brugués i Pujolràs, L Gómez i Bigordà, D Karatzas - International Workshop on …, 2022

… The provided FastText embeddings are either trained on Wikipedia (Wiki) or Common Crawl (CC) corpora and are available in 157 languages Footnote 3 . We have used the monolingual embeddings trained in both Wiki and CC, and we have …


A Multilingual Simplified Language News Corpus

R Hauser, J Vamvas, S Ebling, M Volk - 2nd Workshop on Tools and Resources for …, 2022

Simplified language news articles are being offered by specialized web portals in several countries. The thousands of articles that have been published over the years are a valuable resource for natural language processing, especially for efforts …


A multistage retrieval system for health-related misinformation detection

M Fernández-Pichel, DE Losada, JC Pichel - Engineering Applications of Artificial …, 2022

Web search is widely used to find online medical advice. As such, health-related information access requires retrieval algorithms capable of promoting reliable documents and filtering out unreliable ones. To this end, different types of …


A Novel Approach to Train Diverse Types of Language Models for Health Mention Classification of Tweets

PI Khan, I Razzak, A Dengel, S Ahmed - arXiv preprint arXiv:2204.06337, 2022

Health mention classification deals with the disease detection in a given text containing disease words. However, non-health and figurative use of disease words adds challenges to the task. Recently, adversarial training acting as a means of …


A Paradigm Shift from “Human Writing” to “Machine Generation” in Personality Test Development: an Application of State-of-the-Art Natural Language Processing

P Lee, S Fyffe, M Son, Z Jia, Z Yao - Journal of Business and Psychology, 2022

Natural language processing (NLP) techniques have become increasingly popular in areas of psychological assessment. Recently, researchers have sought to use NLP techniques for automatic item generation (AIG) in the personality domain …


A Part-of-Speech Tagger for Yiddish: First Steps in Tagging the Yiddish Book Center Corpus

S Kulick, N Ryant, B Santorini, J Wallenberg - arXiv preprint arXiv:2204.01175, 2022

We describe the construction and evaluation of a part-of-speech tagger for Yiddish (the first one, to the best of our knowledge). This is the first step in a larger project of automatically assigning part-of-speech tags and syntactic structure to Yiddish text for …


A Persian ASR-based SER: Modification of Sharif Emotional Speech Database and Investigation of Persian Text Corpora

A Yazdani, Y Shekofteh - arXiv preprint arXiv:2211.09956, 2022

Speech Emotion Recognition (SER) is one of the essential perceptual methods of humans in understanding the situation and how to interact with others, therefore, in recent years, it has been tried to add the ability to recognize emotions to human-machine …


A Projection-Based Asymmetric Similarity Measure for Distributional Semantic Models

… We apply our model to GloVe vector representations (trained on the Common Crawl 42B corpus) of pairs of words taken from two databases on free association norms, and compute the correlation between the human responses and our model’s …


A Review on Phishing Websites Revealing through Machine Learning

AS Sengar, A Bhola, RK Shukla, A Gupta - … on System Modeling & Advancement in …, 2021

Phishing is a frequent assault in which unsuspecting people’s unique, private, and sensitive information is stolen through fake websites. The primary objective of phishing websites’consistent resource allocators isto steal unique, private, and …


A Review on Text-Based Emotion Detection--Techniques, Applications, Datasets, and Future Directions

S Kusal, S Patil, J Choudrie, K Kotecha, D Vora… - arXiv preprint arXiv …, 2022

Artificial Intelligence (AI) has been used for processing data to make decisions, interact with humans, and understand their feelings and emotions. With the advent of the internet, people share and express their thoughts on day-to-day activities and …


A Roadmap for Big Model

S Yuan, H Zhao, S Zhao, J Leng, Y Liang, X Wang… - arXiv preprint arXiv …, 2022

… C4 is a colossal, cleaned version of Common Crawl’s web crawl corpus. It is based on Common Crawl dataset and was used to train the … CLUECorpus2020 is a high-quality Chinese pre-training corpus obtained by cleaning the Chinese part of …


A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation

Y Chen, F Wei, X Sun, Z Wu, S Lin - arXiv preprint arXiv:2203.04287, 2022

This paper proposes a simple transfer learning baseline for sign language translation. Existing sign language datasets (eg PHOENIX-2014T, CSL-Daily) contain only about 10K-20K pairs of sign videos, gloss annotations and texts, which …


A Study of Commonsense Reasoning with Language Models

Artificial Intelligence (AI) has gone through an increasing growth in the past decades, which in the present day translates to its usage in almost every sector of society. From its inception, AI pursues the reproduction of human intelligence. Currently, AI¬equipped …


A Study of Implicit Bias in Pretrained Language Models against People with Disabilities

PN Venkit, M Srinath, S Wilson - Proceedings of the 29th International Conference on …, 2022

Pretrained language models (PLMs) have been shown to exhibit sociodemographic biases, such as against gender and race, raising concerns of downstream biases in language technologies. However, PLMs’ biases against people with disabilities (PWDs) …


A Study of Implicit Language Model Bias Against People With Disabilities


A Study of Various Word Embeddings in Deep Learning

P Shah, S Shah, S Joshi - 2022 3rd International Conference for Emerging …, 2022

With the number of users and reviews rising on the internet, and the liberty was given to post anything on it, it is still a major issue to identify the sentiment of the reviews. The traditional machine learning models are taking a lot of computational …


A Study on Chinese-English Machine Translation Based on Transfer Learning and Neural Networks

C Li - Wireless Communications and Mobile Computing, 2022

The existing Chinese-English machine translation has problems such as inaccurate word translation and difficult translation of long sentences. To this end, this paper proposes a new machine translation model based on bidirectional Chinese-English …


A Study on Extracting Named Entities from Fine-tuned vs. Differentially Private Fine-tuned BERT Models

A Diera, N Lell, A Garifullina, A Scherp - arXiv preprint arXiv:2212.03749, 2022

… During prompt creation, we sampled a 100 character length string either from the Common Crawl dataset (naive prompting) or from the test set (informed prompting). We set the sequence length to 256 tokens. We removed the tokens of the prompt …


A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective

E Al-Hossami, S Shaikh - arXiv preprint arXiv:2202.04847, 2022

In this survey paper, we overview major deep learning methods used in Natural Language Processing (NLP) and source code over the last 35 years. Next, we present a survey of the applications of Artificial Intelligence (AI) for source code, also …


A Survey on Gender Bias in Natural Language Processing

K Stanczak, I Augenstein - arXiv preprint arXiv:2112.14168, 2021

Language can be used as a means of reproducing and enforcing harmful stereotypes and biases and has been analysed as such in numerous research. In this paper, we present a survey of 304 papers on gender bias in natural language …


A Survey on NLP resources, tools and techniques for Marathi Language Processing

P Lahoti, N Mittal, G Singh - Transactions on Asian and Low-Resource Language …, 2022

… using goclassy architecture through language classiication and iltering the Common Crawl2 corpus. The current version of OSCAR corpus … CC-Net corpus applied an automatic pipeline to extract massive high-quality monolingual text from …


A Survey on Phishing Website Detection Using Deep Neural Networks

V Sharma, T Halevi - International Conference on Human-Computer …, 2022

Phishing is a social engineering attack, where an attacker poses as a legitimate individual or institution and convinces a victim to divulge their details through human interaction. There has been a steep rise in phishing cases across the globe. A report …


A Survey on Semantics in Automated Data Science

U Khurana, K Srinivas, H Samulowitz - arXiv preprint arXiv:2205.08018, 2022

Data Scientists leverage common sense reasoning and domain knowledge to understand and enrich data for building predictive models. In recent years, we have witnessed a surge in tools and techniques for {\em automated machine learning} …


A survey on text classification: Practical perspectives on the Italian language

A Gasparetto, A Zangari, M Marcuzzo, A Albarelli - PloS one, 2022

Text Classification methods have been improving at an unparalleled speed in the last decade thanks to the success brought about by deep learning. Historically, state-of-the-art approaches have been developed for and benchmarked against English datasets …


A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions

B Qin, B Hui, L Wang, M Yang, J Li, B Li, R Geng… - arXiv preprint arXiv …, 2022

… WDC WebTables [79] is a large-scale table collection, which contains over 233 million tables and has been extracted from the July 2015 version of the CommonCrawl. Those tables are classified as either relational (90 million), entity (139 …


A Symmetric Fusion Learning Model for Detecting Visual Relations and Scene Parsing

X Liu, X Jing, Z Zheng, W Du, X Ding, Q Zhu - Scientific Programming, 2022

… We employ a pretrained word vector fastText trained on Common Crawl [38] to implement our purpose. Unlike the word vector models that ignore the morphological features inside words, the fastText model utilizes a bag of n-grams to …


A Systematic Review on Machine Learning and Deep Learning Models for Electronic Information Security in Mobile Networks

C Gupta, I Johri, K Srinivasan, YC Hu, SM Qaisar… - Sensors, 2022

Today’s advancements in wireless communication technologies have resulted in a tremendous volume of data being generated. Most of our information is part of a widespread network that connects various devices across the globe. The …


A thesis that writes itself: On the threat of AI-generated essays within academia

… 3 The CommonCrawl dataset is a large dataset of web crawl data that is freely available. The dataset is made up of data from various web crawls and contains a wealth of information on the web. The dataset used by GPT-3 is 570GB. …


A Tool for Study on Impact of Big Data Technologies on firm performance

C Lotfi, S Srinivasan, M Ertz, I Latrous

Organizations can use big data analytics to evaluate large data volumes and collect new information. It aids in answering basic inquiries concerning business operations and performance. It also aids in the discovery of unknown patterns in massive …


A Transferable and Automatic Tuning of Deep Reinforcement Learning for Cost Effective Phishing Detection

O Lavie, A Shabtai, G Katz - arXiv preprint arXiv:2209.09033, 2022

… The first dataset is Bahnsen [16], which contains 1.2M benign URLs taken from the Common Crawl corpus, and 1.146M phishing URLs from PhishTank. The second dataset is Wang [19], which contains 245K phishing URLs collected from …


A Transformer-based Sequence-labeling Approach to the Slovenian Cross-domain Automatic Term Extraction

THH Tran, M Martinc, A Repar, A Doucet, S Pollak

Automatic term extraction (ATE) is a popular research task that eases the time and effort of manually identifying terms from domainspecific corpora by providing a list of candidate terms. In this paper, we treat terminology extraction as a sequence-labeling …


A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model

X Sun, T Ge, S Ma, J Li, F Wei, H Wang - arXiv preprint arXiv:2201.10707, 2022

Synthetic data construction of Grammatical Error Correction (GEC) for non-English languages relies heavily on human-designed and language-specific rules, which produce limited error-corrected patterns. In this paper, we propose a generic and …


A Universal Deduplication Architecture for Secure and Efficient Cloud Storage

K Saric, G Ramachandran, S Pal, R Jurdak, S Nepal - Proceedings of the Fourth IEEE …, 2022

… Bloom filters populated with lists of filenames found in the Common Crawl dataset returned query responses in under 5 milliseconds on a typical laptop while configured with a false positive probability of 1 in 1015 (1 in a quadrillion) This is a …


A Use Case of Patent Classification Using Deep Learning with Transfer Learning

R Henriques, A Ferreira, M Castelli - Journal of Data and Information Science, 2022

Abstract: Purpose: Patent classification is one of the areas in Intellectual Property Analytics (IPA), and a growing use case since the number of patent applications has been increasing worldwide. We propose using machine learning algorithms to …


A User Study on Clarifying Comparative Questions

A Bondarenko, E Shirshakova, M Hagen - Proceedings of CHIIR, 2022

… From the obtained entities, we selected the pairs with the highest sentence-wise co-occurrence frequencies in the Common Crawl snapshot 2014154 (eg, ‘drummer’ and ‘guitarist’ for occupation or ‘amoxicillin’ and ‘ciprofloxacin’ for antibiotics). As for …


A Warm Start and a Clean Crawled Corpus--A Recipe for Good Language Models

V Snæbjarnarson, HB Símonarson, PO Ragnarsson… - arXiv preprint arXiv …, 2022

… We also demonstrate how to directly extract Icelandic text from the Common Crawl corpus in … that in mC4, which is also sourced from Common Crawl, there are 107 labelled languages, with … We believe that by using text extracted from the …


A Web-Scale Analysis of the Community Origins of Image Memes

D Morina, MS Bernstein - Proceedings of the ACM on Human-Computer …, 2022

… So, we adopted a common revision called harmonic centrality [43], endorsed and used by the Common Crawl foundation in their web-wide web crawl snapshot [13]. We obtained harmonic centrality scores for each community from a full web crawl by …


Abstractive Summarization of Broadcast News Stories for Estonian

We present an approach for generating abstractive summaries for Estonian spoken news stories in a low-resource setting. Given a recording of a radio news story, the goal is to create a summary that captures the essential information in a short format …


Accurate Dependency Parsing and Tagging of Latin

Having access to high-quality grammatical annotations is important for downstream tasks in NLP as well as for corpus-based research. In this paper, we describe experiments with the Latin BERT word embeddings that were recently be made …


Accurately Identifying Cerebroarterial Stenosis from Angiography Reports Using Natural Language Processing Approaches

CH Lin, KC Hsu, CK Liang, TH Lee, CS Shih, YC Fann - Diagnostics, 2022

Patients with intracranial artery stenosis show high incidence of stroke. Angiography reports contain rich but underutilized information that can enable the detection of cerebrovascular diseases. This study evaluated various natural language …


Active Exploration based on Information Gain by Particle Filter for Efficient Spatial Concept Formation

A Taniguchi, Y Tabuchi, T Ishikawa, LE Hafi… - arXiv preprint arXiv …, 2022

Autonomous robots are required to actively and adaptively learn the categories and words of various places by exploring the surrounding environment and interacting with users. In semantic mapping and spatial language acquisition conducted using …


Ad creative generation using reinforced generative adversarial network

S Terzioğlu, KN Çoğalmış, A Bulut - Electronic Commerce Research, 2022

… T5 and PEGASUS models were trained on 750 GB of English-language text from the public Common Crawl web scrape. BART model was trained on the CNN/Daily Mail dataset, which contains over 300k unique news articles from CNN and the Daily …


Adam Mickiewicz University at WMT 2022: NER-Assisted and Quality-Aware Neural Machine Translation

A Nowakowski, G Pałka, K Guttmann, M Pokrywka - arXiv preprint arXiv:2209.02962, 2022

This paper presents Adam Mickiewicz University's (AMU) submissions to the constrained track of the WMT 2022 General MT Task. We participated in the Ukrainian $\leftrightarrow$ Czech translation directions. The systems are a weighted …


Adam mickiewicz university's english-hausa submissions to the wmt 2021 news translation task

A Nowakowski, T Dwojak - Proceedings of the Sixth Conference on Machine …, 2021

This paper presents the Adam Mickiewicz University’s (AMU) submissions to the WMT 2021 News Translation Task. The submissions focus on the English↔ Hausa translation directions, which is a low-resource translation scenario between distant …


Adapting Automatic Speech Recognition to the Radiology Domain for a Less-Resourced Language: The Case of Latvian

N Gruzitis, R Dargis, VJ Lasmanis, G Garkaje, D Gosko - Intelligent Sustainable …, 2022

Automatic speech recognition (ASR) is becoming available to more and more languages. Training a quality ASR system typically requires significant amount of language resources—representative speech and text corpora. Recent advances in …


Adapting Pretrained Models for Machine Translation

Pre-trained language models received extensive attention in recent years. However, it is still challenging to incorporate a pre-trained model such as BERT into natural language generation tasks. This work investigates a recent method called adapters …


Adapting Pretrained Text-to-Text Models for Long Text Sequences

W Xiong, A Gupta, S Toshniwal, Y Mehdad, W Yih - arXiv preprint arXiv:2209.10052, 2022

We present an empirical study of adapting an existing pretrained text-to-text model for long-sequence inputs. Through a comprehensive study along three axes of the pretraining pipeline -- model architecture, optimization objective, and pretraining …


Adapting Transformers for Multi-Label Text Classification

H Fallah, P Bellot, E Bruno, E Murisasco - 2022

… , in addition to text crawled from the internet (Common-Crawl), on wikipedia article and books, thus getting more vocabulary coverage of MFHAD than CamemBERT that is pre-trained on a corpus derived from Common-Crawl. The …


Addressing religious hate online: from taxonomy creation to automated detection

A Ramponi, B Testa, S Tonelli, E Jezek - PeerJ Computer Science, 2022

… et al., 2020), a RoBERTa-based model pretrained on 2.5TB of CommonCrawl raw text containing 100 languages. We use the bert-base-multilingual-cased … OSCAR is a large-scale multilingual corpus based on filtered Common Crawl data that has …


Adversarial Cross-domain Community Question Retrieval

A Guo, X Li, N Pang, X Zhao - Transactions on Asian and Low-Resource Language …, 2022

… In practice, we choose the embedding set (Common Crawl, glove.840B.300d), which contains 840B tokens and 2.2M vocabulary. The word embeddings are of 300 dimensions. The word embedding is updated during the training process. …


AfriBERTa: Towards Viable Multilingual Language Models for Low-resource Languages

… Given that there is significant overlap between the African language corpora in Common Crawl and the BBC News data that we crawled, we … We observe that the quality of the dataset from Common Crawl is very low, confirming recent findings …


AfroLID: A Neural Language Identification Tool for African Languages

I Adebara, AR Elmadany, M Abdul-Mageed, AA Inciarte - arXiv preprint arXiv …, 2022

Language identification (LID) is a crucial precursor for NLP, especially for mining web data. Problematically, most of the world's $7000$+ languages today are not covered by LID technologies. We address this pressing issue for Africa by …


AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages

BFP Dossou, AL Tonja, O Yousuf, S Osei, A Oppong… - arXiv preprint arXiv …, 2022

In recent years, multilingual pre-trained language models have gained prominence due to their remarkable performance on numerous downstream Natural Language Processing tasks (NLP). However, pre-training these large multilingual language …


AI Personification: Estimating the Personality of Language Models

SR Karra, S Nguyen, T Tulabandhula - arXiv preprint arXiv:2204.12000, 2022

… GPT-3 was pretrained on an open-source dataset called Common Crawl, and other text corpora from sources such as Wikipedia (a popular online encyclopedia). TransformerXL: TransformerXL is a transformer-based language model capable of …


AI WITH ALIEN CONTENT AND ALIEN METASEMANTICS

… by the Common Crawl archiving service. Give the role that the Common Crawl database plays in the training of GPT-3, maybe the Common Crawl … in some way we can’t fully comprehend) by the way in which Common Crawl extracts and stores …


AI-based Structured Web Data Extraction

In this thesis, we explore current approaches for automatic web data extraction, define their limitations, and aim to overcome them. We propose a deep learning model to extract structured data from graph and visual representations of web pages …


AI-supported Natural Language Processing in project management–capabilities and research agenda

AI-based natural language processing (NLP) models show remarkable performance in tasks like question answering or text generation in general. We argue that recent NLP-AI models will play a major role in the transformation of our societies, an …


ALEXSIS-PT: A New Resource for Portuguese Lexical Simplification

K North, M Zampieri, T Ranasinghe - arXiv preprint arXiv:2209.09034, 2022

Lexical simplification (LS) is the task of automatically replacing complex words for easier ones making texts more accessible to various target populations (eg individuals with low literacy, individuals with learning disabilities, second language …


Algal bloom monitoring using multi-spectral satellite data

D Grendaitė, L Petkevičius - DAMSS 2022: 13th conference on data analysis …, 2022

DAMSS-2022 is the 13th International Conference on Data Analysis Methods for Software Systems, held in Druskininkai, Lithuania. Every year at the same place and time. The exception was in 2020, when the world was gripped by the Covid-19 …


Aligning Word Vectors on Low-Resource Languages with Wiktionary

M Izbicki - Proceedings of the Fifth Workshop on Technologies for …, 2022

… these languages are of particular interest to the Wiktionary community for their historical importance, and thus have a lot of entries; but their historical nature also means there are few webpages written in these languages, and so the word vectors …


An Analysis of Social Biases Present in BERT Variants Across Multiple Languages

A Milios, P BehnamGhader - arXiv preprint arXiv:2211.14402, 2022

… GreekBERT [15] is trained on the Greek Wikipedia, the Greek portion of Oscar [16] (a filtered and cleaned multilingual version of the Common Crawl dataset), and the Greek portion of the European Parliament Proceedings Parallel Corpus [17] …


An Approach to Generate Realistic HTTP Parameters for Application Layer Deception

M Sahin, C Hébert, R Cabrera Lozoya - International Conference on Applied …, 2022

… While the names of the honey HTML elements are ideally chosen by the user, authors also implement a suggestion tool based on a Markov model of URLs gathered from the Common Crawl [2] dataset. However, the paper does not provide …


An Automatic Speech Recognition System for Bengali Language based on Wav2Vec2 and Transfer Learning

TT Showrav - arXiv preprint arXiv:2209.08119, 2022

An independent, automated method of decoding and transcribing oral speech is known as automatic speech recognition (ASR). A typical ASR system extracts featured from audio recordings or streams and run one or more algorithms to map …


An empirical study of the impact of log parsers on the performance of log-based anomaly detection

Y Fu, M Yan, Z Xu, X Xia, X Zhang, D Yang - Empirical Software Engineering, 2023

… The word vectors are pre-trained on the Common Crawl Corpus dataset, and the dimension of word vectors is 300. Secondly, it uses the TF-IDF weight to sum up all word vectors in a log event to generate the log event vector. At the same time, the …


An Empirical Study on the Fairness of Pre-trained Word Embeddings

… from 42 billion and 840 billion tokens of Common Crawl corpus. Pre-trained embeddings trained on … + UMBCWeb Base + statmt.org News and 600 billion tokens from Common Crawl. … Also according to these data, we can infer that …


An Ensemble Approach to Acronym Extraction using Transformers

P Sharma, H Saadany, L Zilio, D Kanojia, C Orasan - 2022

… 2019) is a multilingual contextualised Language Model (LM) pre-trained on filtered CommonCrawl data from 100+ languages. Each … feed-forward hidden states, 8 heads; and is pre-trained on CommonCrawl data in over 100 languages …


An ensemble multilingual model for toxic comment classification

G Xie - … Conference on Algorithms, Microchips and Network …, 2022

… Unlabeled text in 100 languages is extracted from CommonCrawl datasets, totaling 2.5TB of text. It is trained in a RoBERTa fashion, that is, only using the MLM objective. XLM-R is a transformer-based multilingual masked language model pre-trained …


An Ensemble of Arabic Transformer-based Models for Arabic Sentiment Analysis

In recent years, sentiment analysis has gained momentum as a research area. This task aims at identifying the opinion that is expressed in a subjective statement. An opinion is a subjective expression describing personal thoughts and feelings. These …


An ensemble transformer-based model for Arabic sentiment analysis

O Mohamed, AM Kassem, A Ashraf, S Jamal… - Social Network Analysis and …, 2023

… XLM-R is a multilingual MLM trained on 2.5 TB of clean Common Crawl data in 100 languages. It provides a solid improvement over previous multilingual models such as mBERT. It performs well in various tasks such as classification, sequence …


An image and text-based multimodal model for detecting fake news in OSN's

SK Uppada, P Patel - Journal of Intelligent Information Systems, 2022

… RoBERTa is pre-trained on Books Corpus and English Wikipedia; in addition to this dataset, RoBERTa is trained on CommonCrawl, Web text corpus, and stories from Common Crawl datasets. A dense output layer with softmax as an activation …


An Information Minimization Based Contrastive Learning Model for Unsupervised Sentence Embeddings Learning

S Chen, J Zhou, Y Sun, L He - arXiv preprint arXiv:2209.10951, 2022

Unsupervised sentence embeddings learning has been recently dominated by contrastive learning methods (eg, SimCSE), which keep positive pairs similar and push negative pairs apart. The contrast operation aims to keep as much information …


An integrated data framework for policy guidance during the coronavirus pandemic: Towards real-time decision support for economic policymakers

JO Dörr, J Kinne, D Lenz, G Licht, P Winker - PloS one, 2022

… Specifically, XLM-RoBERTa has been trained on more than two terabyte of filtered common-crawl data [39]. It has acquired its basic language understanding using the masked language model approach [40], ie given a sequence of text—eg a …


An Interactive Visual Demo of Bias Mitigation Techniques for Word Representations From a Geometric Perspective

A Rathore, S Dev, V Srikumar, JM Phillips, Y Zheng… - NeurIPS 2021 Competitions …, 2022

Abstract Language representations are known to encode and propagate biases, ie, stereotypical associations between words or groups of words that may cause representational harm. In this demo, we utilize interactive visualization to increase …


An Interpretable Word Sense Classifier for Human Explainable Chatbot

RK Yadav, L Jiao, OC Granmo, M Goodwin - International Conference on Agents and …, 2022

… The three models employ static embedding obtained from two sources, CommonCrawl and Base, and are trained using 1 layer neural network … This is because of the enriched information in CommonCrawl in comparison to Base word …


An Unsupervised Approach to Genuine Health Information Retrieval Based on Scientific Evidence

R Upadhyay, G Pasi, M Viviani - International Conference on Web Information …, 2022

… The original dataset is constituted by CommonCrawl news, Footnote 21 sampled from January, 1st 2020 to April 30th, 2020, which contains health-related news articles from all over the world. For our experiments, given the large volume of the …


Analysing Environmental Narratives Computationally

RS Purves, O Koblet, B Adams - Unlocking Environmental Narratives: Towards …, 2022

This book’s origins lie in a desire to showcase, through an interdisciplinary approach, the potential for computational methods in analysing text that describes the environment. Our argument is that these computational methods need not be …


Analysis and Application of Language Models to Human-Generated Textual Content

… The authors trained the model on corpora of different sizes; the larger includes 42 billion tokens of web data from Common Crawl, tokenized and lowercased. A clear correlation between corpus size and performance is observed for syntactic tasks but …


Analysis and control of online interactions through neural natural language processing

Natural Language Processing is motivated by applications where computers should gain a semantic and syntactic understanding of human language. Recently, the field has been impacted by a paradigm shift. Deep learning architectures coupled with …


Analysis of Scientific Literature of LDOW Workshops: A Scientometric and NLP approach

This paper contributes to compiling and publishing a structured dataset from the scientific literature of the Linked Data on the Web (LDOW) workshop series. This workshop was the primary venue for publishing the frontier topics related to …


Analysis of Semantic Shift Before and After COVID-19 in Spanish Diachronic Word Embeddings

ER Betancourt, EC Murillo - 2022 XVLIII Latin American Computer Conference …, 2022

… Como origen de los datos se usó el proyecto CommonCrawl (https://commoncrawl.org/), el cual regularmente recolecta documentos de … Se descargaron desde CommonCrawl los documentos correspondientes a la semana 51 del 2018 y la …


Analysis of the Full-Size Russian Corpus of Internet Drug Reviews with Complex NER Labeling Using Deep Learning Neural Networks and Language Models

A Sboev, S Sboeva, I Moloshnikov, A Gryaznov… - Applied Sciences, 2022

The paper presents the full-size Russian corpus of Internet users’ reviews on medicines with complex named entity recognition (NER) labeling of pharmaceutically relevant entities. We evaluate the accuracy levels reached on this …


Analysis of the Semantic Vector Space Induced by a Neural Language Model and a Corpus

… The language model that we use for this study is XLM-RoBERTa [1], which is a transformer-based model pre-trained on a large corpus (2.5TB of filtered CommonCrawl data) in a self-supervised fashion. The model uses a tokenizer based on …


Analysis of Word-level Embeddings for Indic Languages on AI4Bharat-IndicNLP Corpora

D Goswami, S Malviya, R Mishra, US Tiwary - 2021 IEEE 8th Uttar Pradesh Section …, 2021

This paper presents the analysis of non-contextual word embeddings trained on AI4Bharat-IndicNLP corpus containing 2.7 billion words covering 10 Indian languages. We share the pre-trained embeddings for research and development in …


Analytical Engines With Context-Rich Processing: Towards Efficient Next-Generation Analytics

V Sanca, A Ailamaki - arXiv preprint arXiv:2212.07517, 2022

As modern data pipelines continue to collect, produce, and store a variety of data formats, extracting and combining value from traditional and context-rich sources such as strings, text, video, audio, and logs becomes a manual process where such …


Analyzing Antisemitism and Islamophobia using a Lexicon-based Approach

M Ali, S Zannettou - Workshop Proceedings of the 16th International AAAI …, 2022

The spread of Antisemitic and Islamophobic content in a longstanding problem, in particular within fringe Web communities. In this work, we attempt to analyze the spread of Antisemitic and Islamophobic content on 4chan’s Politically Incorrect …


Analyzing Gender Representation in Multilingual Models

H Gonen, S Ravfogel, Y Goldberg - arXiv preprint arXiv:2204.09168, 2022

Multilingual language models were shown to allow for nontrivial transfer across scripts and languages. In this work, we study the structure of the internal representations that enable this transfer. We focus on the representation of gender …


Analyzing the Mono-and Cross-Lingual Pretraining Dynamics of Multilingual Language Models

T Blevins, H Gonen, L Zettlemoyer - arXiv preprint arXiv:2205.11758, 2022

… This dataset consists of filtered Common Crawl data for 100 languages, with a wide range of data quantities ranging from 0.1 GiB for languages like Xhosa and Scottish Gaelic to over 300 Gib for English. As with XLM-R, we train on CC100 for 1.5M …


Analyzing the Web: Are Top Websites Lists a Good Choice for Research?

T Alby, R Jäschke - International Conference on Theory and Practice of …, 2022

… We present a heuristic-driven alternative based on the Common Crawl host-level web graph while also taking language-… the Common Crawl graph due to 94.9% of hosts in the Common Crawl graph being unknown to the other data sources. Instead …


ANNA: Enhanced Language Representation for Question Answering

C Jun, H Jang, M Sim, H Kim, J Choi, K Min, K Bae - arXiv preprint arXiv:2203.14507, 2022

… of 127,490 wordpieces that are extracted from the English Common Crawl corpus (Raffel et al.… corpora such as a ColossalCleaned version of Common Crawl (C4) corpus (Raffel et al.… for the deletion of documents written in non-English words in …


Annotation Projection-based Dependency Parser Development for Nepali

P Rai, S Chatterji - Transactions on Asian and Low-Resource Language …, 2022

Building computational resources and tools for the under-resourced languages is strenuous for any Natural Language Processing (NLP) task. This paper presents the first dependency parser for an under-resourced Indian language, Nepali. A …


Answering Binary Causal Questions Using Role-oriented Concept Embedding

H Kayesh, MS Islam, J Wang - IEEE Transactions on Artificial Intelligence, 2022

Answering binary causal questions is a challenging task, and it requires rich background knowledge to answer such questions. Extracting useful causal features from the background knowledge base and applying them effectively in a model is a …


Answering Consumer Health Questions on the Web

… In this work, we implement a document filtering technique based on topic-sensitive PageRank that uses a web graph of the hosts in common crawl. We develop a new passage extraction technique that performs query-based contextualized sentence …


APIRO: A Framework for Automated Security Tools API Recommendation

ZT Sworna, C Islam, MA Babar - arXiv preprint arXiv:2201.07959, 2022

… Moreover, for augmenting data with the same Aug and same action, we propose to use a wide variety of data sources, ie, thesaurus and embedding (eg, PPDB, GloVe Common Crawl). Hence, a set of 𝛼 number of Data Augmentation …


AppClassNet: a commercial-grade dataset for application identification research

C Wang, A Finamore, L Yang, K Fauvel, D Rossi - ACM SIGCOMM Computer …, 2022

… Similarly, in the NLP field the Common Crawl [2] project gathered several hundreds of billions of text tokens. This is in stark contrast with the networking field where a commonly identified limit to AI deployment is the lack of publicly available …


Appendix C: Data Augmentation and Pretraining to Improve Neural Headline Generation in Low-Resource Setting

M Martinc, S Montariol, L Pivovarova, E Zosa - Cross-Lingual Embeddings for Less …

We tackle the problem of neural headline generation in a low-resource setting, where only limited amount of data is available to train a model. We compare the ideal high-resource scenario on English with results obtained on a smaller subset of …


Application of convolutional deep neural network for human detection in through the wall radar signals

D Navakauskas, J Skirelis, E Šabanovič, M Kazlauskas… - DAMSS 2022: 13th …, 2022

DAMSS-2022 is the 13th International Conference on Data Analysis Methods for Software Systems, held in Druskininkai, Lithuania. Every year at the same place and time. The exception was in 2020, when the world was gripped by the Covid-19 …


Application of neural network language models based on distributive semantics for ontological modeling of the domain

MG Shishaev, VV Dikovitsky, VK Pimeshkov - Journal of Physics: Conference Series, 2022

… Common Crawl'. The network architecture included 5 layers with a 631-500-350-250-1503 funnel, the ReLU activation function was used, and training was carried out in 15 epochs. The results of testing the resulting model on verification data are shown in …


Applications of machine learning for COVID-19 misinformation: a systematic review

AR Sanaullah, A Das, A Das, MA Kabir, K Shu - Social Network Analysis and Mining, 2022

… (2021) had chosen crawl-300d-2M4 embedding Footnote 4 , a 2 million word vectors which are trained with subword information on Common Crawl (… Footnote 5 which is trained on Common Crawl consisting of 2 million words. Hossain et al. (2020) …


Applying Text Analytics to Insider Risk Analysis: A Case Study on Keyword Generation

C Gardner, WR Clacyomb - 2022 IEEE International Carnahan Conference on …, 2022

Recent advancements in text analytics demonstrate significant gains in many natural language processing tasks, taking advantage of newer neural network architectures and transfer learning. In this article, we present findings from a literature review and …


Approaching what and how people with mental disorders communicate in social media–Introducing a multi-channel representation

ME Aragón, AP López-Monroy, LC González… - Neural Computing and …, 2022

… This model was pre-trained with high-dimensional corpora from Twitter, Common Crawl, and Wikipedia. GloVe presents a lower dimensionality in their word vectors, ranging between 50 and 300 dimensions. GloVe pre-trained with information from …


Arabic abstractive text summarization using RNN-based and transformer-based architectures

M Bani-Almarjeh, MB Kurdy - Information Processing & Management, 2023

… To build our dataset, we used Common Crawl to download the archived … Common Crawl are to make it easy to rebuild our dataset by other researchers and to comply with the copyrights of the respective crawled websites. The data are …


Arabic Fake News Detection: A Fact Checking Based Deep Learning Approach

F Harrag, MK Djahli - Transactions on Asian and Low-Resource Language …, 2022

Fake news stories can polarize society, particularly during political events. They undermine confidence in the media in general. Current NLP systems are still lacking the ability to properly interpret and classify Arabic fake news. Given the high stakes …


AraNPCC: The Arabic Newspaper COVID-19 Corpus

A Al-Thubaity, S Alkhereyf, A Bahanshal - Politics

This paper introduces a corpus for Arabic newspapers during COVID-19: AraNPCC. The AraNPCC corpus covers 2019 until 2021 via automatically-collected data from 12 Arab countries. It comprises more than 2 billion words and 7.2 million texts …


ArSphere: Arabic word vectors embedded in a polar sphere

S Rizkallah, AF Atiya, S Shaheen, HED Mahgoub - International Journal of Speech …, 2022

Word embeddings mean the mapping of words into vectors in an N-dimensional space. ArSphere: is an approach that designs word embeddings for the Arabic language. This approach overcomes one of the shortcomings of word embeddings (for …


Artificial intelligence and the affective labour of understanding: The intimate moderation of a language model

C Perrotta, N Selwyn, C Ewin - New Media & Society, 2022

Interest in artificial intelligence (AI) language models has grown considerably following the release of ‘generative pre-trained transformer’ (GPT). Framing AI as an extractive technology, this article details how GPT harnesses human labour and …


Artificial Intelligence, Deepfakes, and Disinformation

CAI Content Authenticity Initiative GAN generative adversarial network GPT-3 Generative Pre-Trained Transformer 3 OSINT open-source intelligence technique in American society: increasing disagreement in evaluations of facts and analytical …


Artificial Intelligence: A Medium that Hides Its Nature

A Huxor - Artificial Intelligence and Its Discontents, 2022

… This technology builds a vast language model based on a corpus of many written texts, drawn from the Internet which themselves previously created by humans, including texts garnered by crawling the web and made available at Common Crawl …


Artificial Intelligent Context-Aware Machine-Learning Tool to Detect Adverse Drug Events from Social Media Platforms

D Roosan, AV Law, MR Roosan, Y Li - Journal of Medical Toxicology, 2022

… We selected FastText embeddings with 300-dimensional vectors and 3 million vocabularies trained with subword information on Common Crawl with 600 Billion tokens to capture the context of our corpus. We used fastText, which has pre-trained …


Artificial Knowing Otherwise

O Keyes, K Creel - Feminist Philosophy Quarterly, 2022

… Machine learning’s reliance on free, large-scale resources (some prominent examples include Flickr content for facial and object recognition, Wikipedia for text analysis and image classification, and CommonCrawl for web pages) means that …


Asking Clarifying Questions: To benefit or to disturb users in Web search?

J Zou, A Sun, C Long, M Aliannejadi, E Kanoulas - Information Processing & …, 2023

Modern information-seeking systems are becoming more interactive, mainly through asking Clarifying Questions (CQs) to refine users’ information needs. System-generated CQs may be of different qualities. However, the impact of asking multiple CQs of …


Assembly Models for SimpleText Task 2: Results from Wuhan University Research Group

… The pre-trained embedding we choose is trained on Common Crawl, which is from the public domain. There can be pre-trained word embeddings for technology and medical fields, as are the domains covered by the task corpus. Thus, one work …


Assessment of Massively Multilingual Sentiment Classifiers

K Rajda, Ł Augustyniak, P Gramacki, M Gruza… - arXiv preprint arXiv …, 2022

… We also included models trained on multilingual corpus like Wikipedia (Wiki) or Common Crawl (CC) as well as models trained with the use of parallel datasets. Selected models differ in size - from LASER with 52M parameters to LaBSE with …


Assisting Children with Special Needs in Their Daily Interaction with Other People

Children and adults with special needs may find it difficult to recognize danger and threats as well as socially complex situations. They are thus at risk of becoming victims of exploitation and violence. In addition, they may find themselves …


ASurvey ON GPT-3

This paper provides an introductory survey to GPT-3. We cover some of the historical development behind this technology, some of the key features of GPT-3, and discuss the machine learning model and the datasets used. We survey both academic and …


Attack or Block? Repertoires of Digital Censorship in Autocracies

L Kawerau, NB Weidmann, A Dainotti - Journal of Information Technology & Politics, 2022

… When a given host is visited by CommonCrawl multiple times and has the same IP address in two adjacent observations, we assume that the host had the same address on every day between those two observations. We call this period a “stable …


Attention Fusion: a light yet efficient late fusion mechanism for task adaptation in NLU

Fine-tuning a pre-trained language model using annotated data has become the de-facto standard for adapting general-purpose pretrained models like BERT to downstream tasks. However, given the trend of larger pretrained models, fine-tuning these …


Attention-Based Bi-LSTM Network for Abusive Language Detection

KB Nelatoori, HB Kommanti - IETE Journal of Research, 2022

… These GloVe embeddings are trained on Common Crawl which contains 42 billion tokens and a Twitter corpus with 2 billion tweets. GloVe trained on Twitter contains 25, 50, 100 and 200-dimensional word embeddings with 1.2 million …


Attention-Based Models for Classifying Small Data Sets Using Community-Engaged Research Protocols: Classification System Development and Validation Pilot …

BJ Ferrell, SE Raskin, EB Zimmerman, DH Timberline… - JMIR Formative Research, 2022

Background Community-engaged research (CEnR) is a research approach in which scholars partner with community organizations or individuals with whom they share an interest in the study topic, typically with the goal of supporting that community’s …


Attribute Representation in Neural Language Models

… Data and preprocessing To train our multilingual model, we use the CommonCrawl dataset from the CoNLL 2017 shared task [96] to obtain monolingual plain text in various languages. To represent words across different languages in a …


Autoblog 2021: The Importance of Language Models for Spontaneous Lecture Speech

A Hernandez, P Klumpp, B Das, A Maier, SH Yang - International Conference on Text …, 2022

SH Yang - Text, Speech, and Dialogue: 25th International …, 2022

… This corpus contains text from news articles, European parliament speeches, and text from Common Crawl [15]. The transcripts from the training set of TED-LIUM 3 [7] based on 2,351 TED talks are used to build a lecture-style LM. While this model is …


Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?

R Dong, Z Qi, L Zhang, J Zhang, J Sun, Z Ge, L Yi… - arXiv preprint arXiv …, 2022

The success of deep learning heavily relies on large-scale data with comprehensive labels, which is more expensive and time-consuming to fetch in 3D compared to 2D images or natural languages. This promotes the potential of utilizing models …


Automated Essay Scoring using Transformers

K Gupta - arXiv preprint arXiv:2210.12809, 2022

Despite being investigated for over five decades, the task of automated essay scoring continues to draw a lot of attention in the NLP community, in part because of its commercial and educational values as well as the associated research …


Automated Fake News Detection using cross-checking with reliable sources

Z Ghadiri, M Ranjbar, F Ghanbarnejad, S Raeisi - arXiv preprint arXiv:2201.00083, 2022

… We trained this word embedding on Common Crawl and Wikipedia [9]. For two semantically close tweets, their embedding vector should be relatively close. To find the distance between two tweets, we use the cosine between their corresponding vectors. …


AUTOMATED GENERATION OF MANDARIN EDUCATIONAL MULTIMEDIA CONTENT FROM EXISTING ENGLISH CONTENT

MJ Israel, A Qin, Y Zhang, J Xin, N Shaghaghi - ICERI2022 Proceedings, 2022

With the advancement of digital technology, multimedia contents have proliferated in recent years. Large collections of such multimedia content are accrued both in public on the internet and in private enterprise settings. However, the vast majority of …


Automated Identification of Toxic Code Reviews: How Far Can We Go?

J Sarker, AK Turzo, M Dong, A Bosu - arXiv preprint arXiv:2202.13056, 2022

Toxic conversations during software development interactions may have serious repercussions on a Free and Open Source Software (FOSS) development project. For example, victims of toxic conversations may become afraid to express …


AUTOMATED LABELING OF MITRE ATT&CK TACTICS AND TECHNIQUES IN MALWARE THREAT REPORTS

… Initially, we used the common crawl GloVe embedding, which was trained over 42 billion tokens, with a 300 dimensional vector space [22]. It was then used as the embedding layer of the BiLSTM model. The results (method GloVe Pretrained) can …


Automated text-level semantic markers of Alzheimer's disease

C Sanz, F Carrillo, A Slachevsky, G Forno… - Alzheimer's & Dementia …, 2022

Methods Combining statistical and machine learning analyses of natural speech data, we aimed to discriminate ADD patients from healthy controls (HCs) based on automated measures of domains typically affected in ADD: semantic granularity (coarseness …


Automatic and Manual Detection of Generated News: Case Study, Limitations and Challenges

J Bogaert, A Descampe, MC de Marneffe, FX Standaert - 2022

In this paper, we study the exploitation of language generation models for disinformation purposes from two viewpoints. Quantitatively, we argue that language models hardly deal with domain adaptation (ie, the ability to generate text on topics …


Automatic Generation and Detection of Motivational-Interviewing-Style Reflections for Smoking Cessation Therapeutic Conversations Using Transformer-based …

… The GPT-3 model [9] has 175 billion parameters and is trained on 499GB of text data from multiple datasets: Common Crawl from 2016-2019 [48], WebText, Books1 and Books2 (collection of books and movie script text data) [49], and English-language …


Automatic generation of medical reports from chest X-rays in Czech

This thesis deals with the problem of automatic generation of medical reports in the Czech language based on the input chest X-ray images using deep neural networks. The first part deals with the analysis of the problem itself including a comparison of …


Automatic Grammar Correction of Commas in Czech Written Texts: Comparative Study

J Machura, A Frémund, J Švec - International Conference on Text, Speech, and …, 2022

… We used our own pre-trained model from a collection of web data processed in our web mining tool [reference-hidden], Common Crawl data2 and texts collected from the Czech part of Wikipedia. We followed all the training steps, mentioned in [15] …


Automatic Pull Request Title Generation

T Zhang, IC Irsan, F Thung, DG Han, D Lo, L Jiang - arXiv preprint arXiv:2206.10430, 2022

… Unlike BART, T5 was pre-trained with the Colossal Clean Crawled Corpus (C4) dataset, which consists of 750GB of English text from the public Common Crawl web scrape. T5 was reported to achieve state-of-the-art results on many benchmarks …


Automatic question answering for multiple stakeholders, the epidemic question answering dataset

TR Goodwin, D Demner-Fushman, K Lo, LL Wang… - Scientific Data, 2022

… In the end-to-end collection, we also included 265 Reddit threads as well as a subset of the CommonCrawl News Crawl (CCNC) from … e2e/ccns_trec/ contains 114,645 JSON files, each corresponding to an HTML-parsed website included in the …


AUTOMATIC ROMANIAN TEXT GENERATION USING GPT-2

… from Transformers (BLEURT) – a BLEURT checkpoint was used – a self-contained folder that contained a regression model which was tested on several languages, but should work for the 100+ languages of multilingual C4 (a cleaned version of …


Automatic Term Extraction with Joint Multilingual Learning

IN Karaman, I Cicekli, G Ercan - 2022 7th International Conference on Computer …, 2022

Automatic term extraction using deep learning achieves promising results if sufficient training data exists. Unfortunately, some languages may lack these resources in some domains causing poor performance due to under-fitting. In this study, we …


Automatic Text Summarization for Moroccan Arabic Dialect Using an Artificial Intelligence Approach

I Benelallam - … Intelligence: 7th International Conference, CBI 2022 …

A major advantage of artificial intelligence is its ability to automatically perform tasks at a human-like level quickly; this is needed in many fields, and more particularly in Automatic Text Summarization (ATS). Several advances related to this technique …


Automatic text summarization of konkani texts using pre-trained word embeddings and deep learning.

J D'Silva, U Sharma - International Journal of Electrical & Computer …, 2022

… The models are trained on documents from Common Crawl and Wikipedia to obtain embeddings for words in the languages offered [12], [13]. With Konkani, the number of Konkani language articles is significantly low on Wikipedia compared to …


Automating Feedback to Improve Teachers' Effective Use of Instructional Discourse in K-12 Mathematics Classrooms

… In our model, we use the vectors trained on Common Crawl with 840 billion tokens and 300 dimensions. For a given sentence, we identify the GloVe representation for each word in the sentence and compute a mean vector. …


Automating Idea Unit Segmentation and Alignment for Assessing Reading Comprehension via Summary Protocol Analysis

M Gecchele, H Yamada, T Tokunaga, Y Sawaki…

In this paper, we approach summary evaluation from an applied linguistics (AL) point of view. We provide computational tools to AL researchers to simplify the process of Idea Unit (IU) segmentation. The IU is a segmentation unit that can …


Autonomous schema markups based on intelligent computing for search engine optimization

BUD Abbasi, I Fatima, H Mukhtar, S Khan, A Alhumam… - PeerJ Computer Science, 2022

With advances in artificial intelligence and semantic technology, search engines are integrating semantics to address complex search queries to improve the results. This requires identification of well-known concepts or entities and their relationship from …


Balancing between holistic and cumulative sentiment classification

P Agathangelou, I Katakis - Online Social Networks and Media, 2022

Sentiment analysis is a fast-accelerating discipline that develops algorithms for knowledge discovery from opinionated content. The challenges however, when it comes to analyzing user reviews are plenty. Bad-quality, informal use of language …


Balancing novelty and appropriateness leads to creative associations in children

C Rastelli, A Greco, N De Pisapia, C Finocchiaro - 2022

Creative cognition is conceived as the process whereby something novel and appropriate is generated. However, the contribution of novelty and appropriateness to creativity is far from being understood, especially during developmental age. Here …


BanglaNLG: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla

A Bhattacharjee, T Hasan, WU Ahmad, R Shahriyar - arXiv preprint arXiv:2205.11081, 2022

This work presents BanglaNLG, a comprehensive benchmark for evaluating natural language generation (NLG) models in Bangla, a widely spoken yet low-resource language in the web domain. We aggregate three challenging conditional text …


Based on billions of words on the internet,< scp> people</scp>=< scp> men</scp

Recent advances have made it possible to precisely measure the extent to which any two words are used in similar contexts. In turn, this measure of similarity in linguistic context also captures the extent to which the concepts being denoted are …


Being Polite: Modeling Politeness Variation in a Personalized Dialog Agent

M Firdaus, A Shandilya, A Ekbal, P Bhattacharyya - IEEE Transactions on …, 2022

… The word embedding dimension is set as 300 for which we use FastText Embeddings [52] pretrained on the common crawl. We apply 0.5 dropout rate [53] to the layer before softmax. All the models are trained on a single NVIDIA 1080Ti GPU. …


Benchmarking Library Recognition in Tweets

T Zhang, DP Chandrasekaran, F Thung, D Lo - 2022

… News dataset); OpenWebText [2] (an open-source recreation of the WebText corpus), and Stories [40] (a subset of CommonCrawl data that … Compared to BERT, XLNet also involved more pre-training data, such as Common Crawl [1]. XLNet …


Benchmarking Transformers-based models on French Spoken Language Understanding tasks

O Cattan, S Ghannay, C Servan, S Rosset - INTERSPEECH 2022, 2022

… Most models are pre-trained on the content of either Wikipedia, the web pages gathered by Common Crawl (CC) or a combination of diverse corpora. This represents a few gigabytes to several hundred or even several terabytes of text. The …


Benefits from Variational Regularization in Language Models

C Ferner, S Wegenkittl - Machine Learning and Knowledge Extraction, 2022

… WMT19 contains data from news commentaries, Wiki titles, Europarl, ParaCrawl, and Common Crawl corpora. The data are tokenized using … Both BERT and MiniBERT were trained on a much larger collection of datasets (Wikipedia …


Bengali POS Tagging Using Bi-LSTM with Word Embedding and Character-Level Embedding

Part-of-speech tagging (POS) is an important and very fundamental process in natural language processing (NLP). POS tagging is required as a preprocessing task in many types of linguistic research such as named entity recognition (NER) …


Bernice: A Multilingual Pre-trained Encoder for Twitter

A DeLucia, S Wu, A Mueller, C Aguirre, M Dredze…

… XLM-T and TwHIN-BERT* models use the XLM-R tokenizer, which was trained on out-of-domain CommonCrawl data and has the same vocabulary size as Bernice (250K tokens), but covers 100 languages versus 66.While trained on tweets, BERTweet is …


BERT based Multiple Parallel Co-attention Model for Visual Question Answering

M Dias, H Aloj, N Ninan, D Koshti - 2022 6th International Conference on Intelligent …, 2022

Humans can easily interpret visual and textual content whereas for a machine this is a challenging task. Visual question answering is a well-known problem in the field of computer vision and NLP where an image and a question related to the image are …


Bertifying sinhala-a comprehensive analysis of pre-trained language models for sinhala text classification

V Dhananjaya, P Demotte, S Ranathunga, S Jayasena - Proceedings of the 13th …, 2022

This research provides the first comprehensive analysis of the performance of pre-trained language models for Sinhala text classification. We test on a set of different Sinhala text classification tasks and our analysis shows that out of the pre-trained …


BERTuit: Understanding Spanish language in Twitter through a native transformer

J Huertas-Tato, A Martin, D Camacho - arXiv preprint arXiv:2204.03465, 2022

The appearance of complex attention-based language models such as BERT, Roberta or GPT-3 has allowed to address highly complex tasks in a plethora of scenarios. However, when applied to specific domains, these models encounter …


Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources

While the NLP community is generally aware of resource disparities among languages, we lack research that quantifies the extent and types of such disparity. Prior surveys estimating the availability of resources based on the number of …


Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

B Patra, S Singhal, S Huang, Z Chi, L Dong, F Wei… - arXiv preprint arXiv …, 2022

… 2020) which contains texts in 100 languages collected from Common Crawl. As mentioned in (§3.1), we explore the utility of the CCMatrix and the … Furthermore, the raw data was compiled from static Common Crawl snapshots from January …


Bilingual attention based neural machine translation

L Kang, S He, M Wang, F Long, J Su - Applied Intelligence, 2022

… Our contrast models mainly include (1) the winning system in WMT2014 [3], a phrase-based system of which language models were trained on the Common Crawl corpus; (2) GNMT [16] that is the most competitive RNN-based NMT model to …


Bilingual Tabular Inference: A Case Study on Indic Languages

C Agarwal, V Gupta, A Kunchukuttan, M Shrivastava

Existing research on Tabular Natural Language Inference (TNLI) exclusively examines the task in a monolingual setting where the tabular premise and hypothesis are in the same language. However, due to the uneven distribution of …


Biomedical Semantic Textual Similarity: Evaluation of Sentence Representations Enhanced with Principal Component Reduction and Word Frequency Weighting

K Kantor, M Morzy - International Conference on Artificial Intelligence in …, 2022

… We examine 50-dimension and 300-dimension GloVe models built on Wikipedia corpus and 300-dimension models trained on the Common Crawl. We choose the BERT model [7] trained on BookCorpus and English Wikipedia for neural-based …


Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages

K Heffernan, O Çelebi, H Schwenk - arXiv preprint arXiv:2205.12654, 2022

… Our monolingual data comes mostly from Common Crawl and other public sources like ParaCrawl5, and some additional targeted … Our main source of monolingual data was Common Crawl, complemented and targeted crawling (see …


BJTU-WeChat's Systems for the WMT22 Chat Translation Task

Y Liang, F Meng, J Xu, Y Chen, J Zhou - arXiv preprint arXiv:2211.15009, 2022

… For monolingual data, we use the News Crawl, Common Crawl, and Extended Common Crawl. For synthetic data generation, we back-translate all the target monolingual data and forward-translate the source monolingual data. For the in-domain …


BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

TL Scao, A Fan, C Akiki, E Pavlick, S Ilić, D Hesslow… - arXiv preprint arXiv …, 2022

… The latter required the development of new tools to extract text from the HTML in the Common Crawl WARC files, which we made available on the main data preparation repository.We were able to find and extract usable text data from all …


Boosting named entity recognition in domain-specific and low-resource settings

Recent researches in natural language processing have leveraged attention-based models to produce state-of-the-art results in a wide variety of tasks. Using transfer learning, generic models like BERT can be fine-tuned for domain-specific tasks …


Bootstrapping Icelandic Knowledge Graph Data

… The model is trained on several sources of bitext data, including ccMatrix [13], ccAligned [3] and OPUS [15], as well as monolingual data from Common Crawl 2. The model uses a modified transformer architecture which reaches a BLEU-score of …


Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation

C Zan, L Ding, L Shen, Y Cao, W Liu, D Tao - arXiv preprint arXiv:2204.07834, 2022

For multilingual sequence-to-sequence pretrained language models (multilingual Seq2Seq PLMs), eg mBART, the self-supervised pretraining task is trained on a wide range of monolingual languages, eg 25 languages from commoncrawl, while …


Build-a-Bot: Teaching Conversational AI Using a Transformer-Based Intent Recognition and Question Answering Architecture

K Pearce, S Alghowinem, C Breazeal - arXiv preprint arXiv:2212.07542, 2022

As artificial intelligence (AI) becomes a prominent part of modern life, AI literacy is becoming important for all citizens, not just those in technology careers. Previous research in AI education materials has largely focused on the introduction of …


Building a Closed-Domain Question Answering System for a Low-Resource Language

P Tran, D Nguyen, HA Tran, T Nguyen, T Tran - Transactions on Asian and Low …, 2022

… The authors announced two models including XLM-Rbase with 250 million parameters and XLM-Rlarge with 560 million parameters trained over 2.5 terabytes of data in 100 languages (including Vietnamese) from Common Crawl. XLM-R …


Building a comprehensive NER model for Satellite Domain

P Maurya, O Jafari, B Thatte, C Ingram, P Nagarkar - SN Computer Science, 2022

… A model trained on the OntoNotes 5 and GloVe Common Crawl datasets. All of these models can recognize types of entities such as PERSON, NORP, FAC, ORG, GPE, LOC, PRODUCT, EVENT, WORK_OF_ART, LAW, LANGUAGE, DATE, TIME …


Building a Connected Web Observatory

… For the SUWO, we are harvesting a variety of Web feeds, which include real-time streams from social networking and user-generated content sites (Twitter, Facebook, YouTube), Web-crawled datasets using existing resources such as the ’CommonCrawl’1 …


Building a Multilingual Taxonomy of Olfactory Terms with Timestamps

S Menini, T Paccosi, SS Tekiroglu, S Tonelli - To

Olfactory references play a crucial role in our memory and, more generally, in our experiences, since researchers have shown that smell is the sense that is most directly connected with emotions. Nevertheless, only few works in NLP have tried to …


Building AI Enabled Note-taking Interfaces for Patient Encounters

Current clinical note-taking approaches cannot capture the entirety of information available from patient encounters and detract from patient-clinician interactions. To address those issues we investigate the feasibility and suitability of using artificial …


Building better machine learning models for rhetorical analyses: The use of rhetorical feature sets for training artificial neural network models

ZP Majdik, J Wynn - Technical Communication Quarterly, 2022

In this paper, we investigate two approaches to building artificial neural network models to compare their effectiveness for accurately classifying rhetorical structures across multiple (non-binary) classes in small textual datasets. We find that the most …


Building Comparable Corpora for Assessing Multi-Word Term Alignment

O Adjali, E Morin, P Zweigenbaum - Proceedings of the 13th Language Resources …, 2022

Recent work has demonstrated the importance of dealing with Multi-Word Terms (MWTs) in several Natural Language Processing applications. In particular, MWTs pose serious challenges for alignment and machine translation systems because of their …


Building Korean NER models for a manually annotated corpus from clinical notes using cross-lingual transfer learning

J Li, J Park, X Hu, J Wang, RW Park, H Xu

… text dataset, BERT-MULTILINGUALBASE was a multilingual BERT model that was pre-trained on 104 languages with the largest Wikipedias dataset, and XLM-ROBERTABASE was a multilingual language model that was pre-trained on 100 languages with 2TB …


Building Machine Translation System for Software Product Descriptions Using Domain-specific Sub-corpora Extraction

P Lohar, S Madden, E O'Connor, M Popović… - Proceedings of the 15th …, 2022

Building Machine Translation systems for a specific domain requires a sufficiently large and good quality parallel corpus in that domain. However, this is a bit challenging task due to the lack of parallel data in many domains such as economics …


Building Sentiment Lexicons for Mainland Scandinavian Languages Using Machine Translation and Sentence Embeddings

This paper presents a simple but effective method to build sentiment lexicons for the three Mainland Scandinavian languages: Danish, Norwegian and Swedish. This method benefits from the English Sentiwordnet and a thesaurus in one of the target …


Building Siamese Attention-Augmented Recurrent Convolutional Neural Networks for Document Similarity Scoring

S Han, L Shi, R Richie, FR Tsui - Information Sciences, 2022

Automatically measuring document similarity is imperative in natural language processing, with applications ranging from recommendation to duplicate document detection. State-of-the-art approach in document similarity commonly involves deep …


Business Document Information Extraction: Towards Practical Benchmarks

M Skalický, Š Šimsa, M Uřičář, M Šulc - arXiv preprint arXiv:2206.11229, 2022

… [93] was not published, while pre-training on it was crucial for the proposed method, and the C4 dataset [64] is shared in the form of code that extracts it directly from Common Crawl. Publicly available datasets for KI(L)E from business …


C-OSINT: COVID-19 Open Source artificial INTelligence framework

L Ranaldi, A Nourbakhsh, F Fallucchid, FM Zanzotto - 2022

With the emergence of COVID-19 disease worldwide, a market of the products related to this disease formed across the Internet. By the time these goods were in short supply, many uncontrolled Dark Web Marketplaces (DWM) were active in …


Can Cross-Domain Term Extraction Benefit from Cross-lingual Transfer?

HTH Tran, M Martinc, A Doucet, S Pollak - International Conference on Discovery …, 2022

Automatic term extraction (ATE) is a natural language processing task that eases the effort of manually identifying terms from domain-specific corpora by providing a list of candidate terms. In this paper, we experiment with XLM-RoBERTa to evaluate the …


Can Foundation Models Help Us Achieve Perfect Secrecy?

S Arora, C Ré - arXiv preprint arXiv:2205.13722, 2022

… 4A particular challenge was obtaining an FM for Reddit, since most FMs are trained on generic Common Crawl — we thus use Grover FMs, which are pretrained only on news articles and find the performance of Grover models directly match or …


Can linguistic features extracted from geo-referenced tweets help building function classification in remote sensing?

M Häberle, EJ Hoffmann, XX Zhu - ISPRS Journal of Photogrammetry and Remote …, 2022

… To add more heterogeneous language, they additionally utilized CommonCrawl 13 to train the word embedding using fastText, and words were left in upper and lower case to add some semantics and context to words (cf. Section 3.4). For our …


Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?

P Schramowski, C Tauchmann, K Kersting - arXiv preprint arXiv:2202.06675, 2022

… The most recent and successful models, such as GPT-3 [5], CLIP [36], DALL-E [38] and other similar models, are trained on data scraped from the web, eg using CommonCrawl. The information they acquire from this data is largely uncontrolled …


Capturing Failures of Large Language Models via Human Cognitive Biases

E Jones, J Steinhardt - arXiv preprint arXiv:2202.12299, 2022

Large language models generate complex, open-ended outputs: instead of outputting a single class, they can write summaries, generate dialogue, and produce working code. In order to study the reliability of these open-ended systems, we must …


Catching a Phish: Frontiers of Deep Learning-Based Anticipating Detection Engines

H Salah, H Zuhair - International Conference of Reliable Information and …, 2022

In recent years, cyber-security gains high attention in the light of ethical hacking and social engineering attacks like phishing that riskily overshadow the development of social networking, e commerce, and information technology. Thus, mitigation of such …


CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarks

X He, Q Xu, Y Zeng, L Lyu, F Wu, J Li, R Jia - arXiv preprint arXiv:2209.08773, 2022

… Since the training data of the victim model is unknown to the malicious users, we randomly select 5M sentences from common crawl data as the benign corpus. Then we obtain the word distribution for the watermarked and benign corpora, respectively …


CausalDialogue: Modeling Utterance-level Causality in Conversations

YL Tuan, A Albalak, W Xu, M Saxon, C Pryor, L Getoor… - arXiv preprint arXiv …, 2022

Despite their widespread adoption, neural conversation models have yet to exhibit natural chat capabilities with humans. In this research, we examine user utterances as causes and generated responses as effects, recognizing that changes in a cause …


Cedille: A large autoregressive French language model

M Müller, F Laurent - arXiv preprint arXiv:2202.03371, 2022

… Experiments conducted on GPT-3 (which has been trained on 570GB of text data from Common Crawl) show that the model may generate toxic sentences even when prompted with non-toxic text [11]. Although applying filtering of training data using …


Challenges and limits of an open source approach to Artificial Intelligence

… It takes the GPT model to a whole new level as it is trained on a massive number of parameters (ie 175 billions, which is over 10x the size of its predecessor) on an open source dataset called 'Common Crawl,' and other texts from OpenAI such as …


Challenges in Measuring Bias via Open-Ended Language Generation

AF Akyürek, MY Kocyigit, S Paik, D Wijaya - arXiv preprint arXiv:2205.11601, 2022

Researchers have devised numerous ways to quantify social biases vested in pretrained language models. As some language models are capable of generating coherent completions given a set of textual prompts, several prompting datasets …


Changing Language Ideological Concepts in the Human-Machine Era. Questions, Themes and Topics

D Dippold, I Erdocia, T Fester-Seeger, S Höhn…

It introduces general thoughts on language ideological reformulations in the age of digital technologies as well as exemplary empirical studies that display language ideological changes, conducted by individual members of the Working Group. In this …


Changing the Narrative Perspective: A New Language Processing Task and Machine Learning Approaches

In this dissertation I introduce the novel text processing task of changing the narrative perspective, where characters in a story are assigned a point of view that is di erent from the one originally used by the writer. The resulting shift in the narrative …


ChapterBreak: A Challenge Dataset for Long-Range Language Models

S Sun, K Thai, M Iyyer - arXiv preprint arXiv:2204.10878, 2022

While numerous architectures for long-range language models (LRLMs) have recently been proposed, a meaningful evaluation of their discourse-level language understanding capabilities has not yet followed. To this end, we introduce …


Characterization and Prediction of Mobile Tasks

Y Tian, K Zhou, D Pelleg - ACM Transactions on Information Systems (TOIS), 2022

Mobile devices have become an increasingly ubiquitous part of our everyday life. We use mobile services to perform a broad range of tasks (eg, booking travel or conducting remote office work), leading to often lengthy interactions with several …


Characterizing accident narratives with word embeddings: Improving accuracy, richness, and generalizability

DM Goldberg - Journal of Safety Research, 2021

Introduction: Ensuring occupational health and safety is an enormous concern for organizations, as accidents not only harm workers but also result in financial losses. Analysis of accident data has the potential to reveal insights that may improve …


Characterizing and Resolving Degeneracies in Neural Autoregressive Text Generation

… We use the subset of WMT’19 training set consisting of news commentary v12 and common crawl resulting in slightly more than 1M and 2M training sentence pairs for Ru→En and De↔En pairs, respectively. We ne-tuned single model checkpoints …


Choose Your QA Model Wisely: A Systematic Study of Generative and Extractive Readers for Question Answering

M Luo, K Hashimoto, S Yavuz, Z Liu, C Baral, Y Zhou - arXiv preprint arXiv …, 2022

While both extractive and generative readers have been successfully applied to the Question Answering (QA) task, little attention has been paid toward the systematic comparison of them. Characterizing the strengths and weaknesses of the two …


City of Disguise: A Query Obfuscation Game on the ClueWeb

We present City of Disguise, a retrieval game that tests how well searchers are able to reformulate some sensitive query in a ‘Taboo’-style setup but still retrieve good results. Given one of 200 sensitive information needs and a relevant example …


Classification of Construction Accident Court Cases Via Natural Language Processing in Hong Kong

RYM Li, HCY Li, B Tang, WC Au - Current State of Art in Artificial Intelligence and …, 2022

… The fastText system has a trained English language vector model derived from Wikipedia, Common Crawl and other sources. Every word is stored along with several different tokens, and the morphology and word representations are replaced …


Classification of German Jungian Extraversion and Introversion Texts with Assessment of Changes during the COVID-19 Pandemic

The corona pandemic and countermeasures such as social distancing and lockdowns have confronted individuals with new challenges for their mental health and well-being. It can be assumed that the Jungian psychology types of extraverts …


Classification of Indian Media Titles using Deep Learning Techniques

S Kumar, DD Rajesh, S Pranesh, VNH Kollipara… - International Journal of …, 2022

Automatic speech recognition is being used everywhere these days. An essential part of this is language identification. Our goal here is to identify the language of the media title, such as song names and movie titles, to help in speech recognition. The …


Classification of Layout vs Relational Tables on the Web: Machine Learning with Rendered Pages

W Haider, Y Yesilada - ACM Transactions on the Web (TWEB), 2022

Table mining on the web is an open problem, and none of the previously proposed techniques provides a complete solution. Most research focuses on the structure of the HTML document, but because of the nature and structure of the web, it is still a …


Classifying and labeling the relationships between cities with high levels of co-occurrence on the English Wikipedia

This study delves into three different approaches of document classification in order to successfully classify the type of relationship between European cities: ”LDA Topic modeling, Word embedding classification and Word frequency representation”. The …


Classifying Documents by Viewpoint Using Word2Vec and Support Vector Machines

J Harwell, Y Li - International Conference on Applications of Natural …, 2022

Ensuring viewpoint diversity in mass media is a historical challenge and recent political events, and the ever-increased use of the Internet, have made it an increasingly critical and contentious issue. This research explores the relationship …


Classifying multi-level product categories using dynamic masking and transformer models

O Ozyegen, H Jahanshahi, M Cevik, B Bulut, D Yigit… - Journal of Data, Information …, 2022

In an online shopping platform, a detailed categorization of the products greatly enhances user navigation. Online retailers also benefit from well-defined product categories as various sales and marketing operations such as special discounts and …


ClidSum: A Benchmark Dataset for Cross-Lingual Dialogue Summarization

J Wang, F Meng, Z Lu, D Zheng, Z Li, J Qu, J Zhou - arXiv preprint arXiv:2202.05599, 2022

We present ClidSum, a benchmark dataset for building cross-lingual summarization systems on dialogue documents. It consists of 67k+ dialogue documents from two subsets (ie, SAMSum and MediaSum) and 112k+ annotated summaries in different …


CLSE: Corpus of Linguistically Significant Entities

A Chuklin, J Zhao, M Kale - arXiv preprint arXiv:2211.02423, 2022

One of the biggest challenges of natural language generation (NLG) is the proper handling of named entities. Named entities are a common source of grammar mistakes such as wrong prepositions, wrong article handling, or incorrect entity …


ClueWeb22: 10 Billion Web Documents with Rich Information

A Overwijk, C Xiong, J Callan - Proceedings of the 45th International ACM SIGIR …, 2022

… One approach is to sift CommonCrawl data, eg, the C4 dataset used to pretrain T5 [10], which provides sufficient quantity, but the quality quickly becomes a concern. For example, the cleaned CommonCrawl reflects a quite weird distribution of the …


CM3: A Causal Masked Multimodal Model of the Internet

A Aghajanyan, B Huang, C Ross, V Karpukhin, H Xu… - arXiv preprint arXiv …, 2022

… on close to a Terabyte of multi-modal simplified HTML data from the common crawl. • We present the causally masked objective, a hybrid of … all of Common Crawl and instead opt into using a subset of the Common Crawl News (CC-NEWS) …


CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Data Limitation With Contrastive Learning

X Liu, Z Zhang, Y Wang, Y Lan, C Shen - arXiv preprint arXiv:2212.10341, 2022

… (2019a), in which HWTs are collected from RealNews, a large corpus of news articles from Common Crawl, and MGTs are generated by Grover-Mega, a transformer-based news generator with parameters sized 1.5B. …


Cognitive linguistics meets computational linguistics: Construction grammar, dialectology, and linguistic diversity

J Dunn - Data Analytics in Cognitive Linguistics: Methods and …, 2022

Computational linguistics and cognitive linguistics come together when we use data-driven methods to conduct linguistic experiments on corpora. This chapter uses usage-based construction grammar to model geographic variation in language. The basic …


Combining shallow and deep learning approaches against data scarcity in legal domains

F Sovrano, M Palmirani, F Vitali - Government Information Quarterly, 2022

We are recently witnessing a radical shift towards digitisation in many aspects of our daily life, including law, public administration and governance. This has sometimes been done with the aim of reducing costs and human errors by improving data …


Commonsense Validation for Arabic Sentences using Deep Learning

E Al-Bashabsheh, H Al-Khazaleh, O Elayan, R Duwairi - 2021 22nd International Arab …, 2021

… -based masked language model trained on a massive 2.5 TB of clean CommonCrawl data. The data includes 100 languages and obtained a decent performance as it … The XLM-Roberta was trained on 2.5 TB of recently produced …


Compact N-gram Language Models for Armenian

DS Karamyan, TS Karamyan - Mathematical Problems of Computer Science, 2022

Applications such as speech recognition and machine translation use language models to select the most likely translation among many hypotheses. For on-device applications, inference time and model size are just as important as performance. In …


Company recommendation model: Empowering the accounting system and publicly available data

R Štrimaitis, P Stefanovič, S Ramanauskaitė… - DAMSS 2022: 13th …, 2022

[eng] Business management requires constant decision-making. Usually, the selection of suitable partners for collaboration is very intuitive, experience-based skill or requires the usage of data analytics. The most important aspect to get an …


Comparative Analysis of Cross-lingual Contextualized Word Embeddings

Contextualized word embeddings have emerged as the most important tool for performing NLP tasks in a large variety of languages. In order to improve the crosslingual representation and transfer learning quality, contextualized embedding …


Comparative Analysis of Machine Learning Classifiers for Phishing Detection

MF Ab Razak, MI Jaya, F Ernawan, A Firdaus… - 2022 6th International …, 2022

… The source for this dataset is from the University Malaysia of Sarawak, compiled from PhishTank, OpenPhish, Alexa and Common Crawl. One method for detecting new phishing websites is to utilize heuristics such as the URL and CSS detection …


Comparative Analysis of Transformer Models on WikiHow Dataset

D Jadeja, A Khetri, A Mittal, DK Vishwakarma - 2022 International Conference on …, 2022

Because of the ever-increasing amount of information around us, it has become highly essential to consume it in the most efficient way as possible. Text Summarization is a way of generating a summary for a long piece of text while …