Formulating Automated Responses to Cognitive Distortions for CBT Interactions

I de Toledo Rodriguez, G Salton, R Ross - … of The Fourth International Conference on …, 2021

… 2020), trained on a cleaned version of the Common Crawl corpus (C4). The published model checkpoints have been pre-trained in a diverse variety of unsupervised and supervised tasks (language modelling, word embedding, machine translation …


FP8 Formats for Deep Learning

P Micikevicius, D Stosic, N Burgess, M Cornea… - arXiv preprint arXiv …, 2022

FP8 is a natural progression for accelerating deep learning training inference beyond the 16-bit formats common in modern processors. In this paper we propose an 8-bit floating point (FP8) binary interchange format consisting of two encodings …


FPSRS: A Fusion Approach for Paper Submission Recommendation System

ST Huynh, N Dang, DH Nguyen, PT Huynh, BT Nguyen - arXiv preprint arXiv …, 2022

… We remove single letters that likely do not have pretrained weights in wellknown pretrained word vectors like FastText Common Crawl4. 4. We remove words within stopwords downloaded from the Natural Language Toolkit (NLTK5) and additional …


French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English

A Névéol, Y Dupont, J Bezançon, K Fort - ACL 2022-60th Annual Meeting of the …, 2022

Warning: This paper contains explicit statements of offensive stereotypes which may be upsetting. Much work on biases in natural language processing has addressed biases linked to the social and cultural experience of English speaking individuals in …


From Bag-of-Words to Transformers: A Deep Dive into the Participation in the eRisk Early Risk Detection of Depression Tasks with Classical and New Approaches

H Damm, M Trotzek, S Koitka, H Schäfer… - … : The First Five Years of the …, 2022

… 12 For this work, a 50-dimensional GloVe model trained on Wikipedia and the Gigaword 5 news corpus as well as a 300-dimensional GloVe model trained on Common Crawl were chosen. In addition, three 300-dimensional pre-trained …


From Bag-of-Words to Transformers: A Deep Dive into the Participation in the eRisk Tasks with Classical and New Approaches

H Damm, M Trotzek, S Koitka, H Schäfer… - Early Detection of Mental …, 2022

… Footnote 12 For this work, a 50-dimensional GloVe model trained on Wikipedia and the Gigaword 5 news corpus as well as a 300-dimensional GloVe model trained on Common Crawl were chosen. In addition, three 300-dimensional pre-trained …


From tabular data to knowledge graphs: A survey of semantic table interpretation tasks and methods

J Liu, Y Chabot, R Troncy, VP Huynh, T Labbé… - Journal of Web Semantics, 2022

Tabular data often refers to data that is organized in a table with rows and columns. We observe that this data format is widely used on the Web and within enterprise data repositories. Tables potentially contain rich semantic information that still needs …


Fully Automated Detection of Formal Thought Disorder with Time-series Augmented Representations for Detection of Incoherent Speech (TARDIS)

W Xu, W Wang, J Portanova, A Chander, A Campbell… - Journal of Biomedical …, 2022

… For the current research, we used publicly available pre-trained vectors derived using the FastText [39] package 8 consisting of 2-million-word vectors trained on a corpus derived from Common Crawl 9 . Individual words are represented by their …


Fusion of Remote Sensing Images and Social Media Text Messages for Building Function Classification

Detailed data about urban land-use, like building functions, is useful to governments for resource planning. To detect building functions, satellite imagery could be too course. In this work, individual buildings from 42 cities are classified with multilingual …


FuzzyDedup: Secure Fuzzy Deduplication for Cloud Storage

T Jiang, X Yuan, Y Chen, K Cheng, L Wang, X Chen… - IEEE Transactions on …, 2022

Data deduplication is of critical importance to reduce the storage cost for clients and to relieve the unnecessary storage pressure for cloud servers. While various techniques have been proposed for secure deduplication of identical files/blocks …


Galactica: A Large Language Model for Science

R Taylor, M Kardas, G Cucurull, T Scialom, A Hartshorn… - arXiv preprint arXiv …, 2022

… While we do pick up some high-school level content through encyclopedias, textbooks and the filtered CommonCrawl, this amounts to a small quantity of tokens (a few billion). We leave the question of how to capture more of this base scientific …


Gauge against the machine: improving representations within sociotechnical instruments to enrich context and identify biases

The proliferation of digital data across all areas of society has transformed our ability to hypothesize, study, and understand social systems. From this richness of data we have seen the development of innovative instruments to study---and make decisions …


Gauging Engagement: Measuring Student Response to a Large-Scale College Advising Field Experiment

Interactive, text message-based advising programs have become an increasingly common strategy to support college access and success for underrepresented student populations. Despite the proliferation of these programs, we know relatively …


Gender and Racial Stereotype Detection in Legal Opinion Word Embeddings

… In addition, the instrument/weapons control display equivalent effect sizes between Caliskan’s Common Crawl corpus experiments and the legal … Gender Effects: While the effect sizes were comparable between the Common Crawl corpus …


Gender Bias in Machine Translation Systems

S Ullmann - Artificial Intelligence and Its Discontents, 2022

… The data was originally obtained from Common Crawl, a free online repository consisting of large amounts of internet data. Consequently, the data represents a variety of registers from formal to informal, slang and even archaic vocabulary due to …


Gender Bias in Meta-Embeddings

M Kaneko, D Bollegala, N Okazaki - arXiv preprint arXiv:2205.09867, 2022

Combining multiple source embeddings to create meta-embeddings is considered effective to obtain more accurate embeddings. Different methods have been proposed to develop meta-embeddings from a given set of source embeddings …


Generating Extended and Multilingual Summaries with Pre-trained Transformers

R Calizzano, M Ostendorff, Q Ruan, G Rehm

Almost all summarisation methods and datasets focus on a single language and short summaries. We introduce a new dataset called WikinewsSum for English, German, French, Spanish, Portuguese, Polish, and Italian summarisation tailored for …


Generating Full Length Wikipedia Biographies: The Impact of Gender Bias on the Retrieval-Based Generation of Women Biographies

A Fan, C Gardent - arXiv preprint arXiv:2204.05879, 2022

… Given a person’s name, one or more occupation(s), and CommonCrawl as a source of … name, one or more occupations, a section header, and CommonCrawl as a retrieval corpus. … We reject all CommonCrawl links from Wikipedia, to prevent …


Generating Music using AI

Network speakers are used in public spaces for public announcements and sometimes to play background music. To play music, a special commercial music licence is required, specifically made for use in public environments. However …


Generating Role-Playing Game Quest Descriptions With the GPT-2 Language Model

Recent advances in artificial intelligence research have brought forth text-generating language models with promising computational storytelling capabilities. This thesis leveraged one of the most successful Transformer models, GPT-2, to automatically …


Generating Scientific Definitions with Controllable Complexity

… We use scientific abstracts, rather than general audience text like Wikipedia or the Common Crawl, for two reasons. First, scientific terms are originally introduced and most commonly used in research papers, making them the most reliable source for …


Generating Word Embeddings for the Azerbaijani Language and Their Empirical Evaluation on Intrinsic and Extrinsic Evaluation Tasks

… al. has also partitioned their data corpora into smaller datasets, such as Common Crawl, Wikipedia, WebText and etc. The machine learning architecture they have built, is one of the largest language models ever built with 175 billion parameters …


The University of Nevada, Reno Getting along to get ahead: Predictors of success in an online competitive social environment A d Page 1 The University of Nevada, Reno Getting along to get ahead: Predictors of success in an online competitive …


Getting to the root of linguistic alignment: Testing the predictions of Interactive Alignment across developmental and biological variation in language skill

R Foushee, D Byrne, M Casillas, S Goldin-Meadow - 2022

Linguistic alignment—the contingent reuse of our interlocutors' language at all levels of linguistic structure—pervades human dialogue. Here, we design unique measures to capture the degree of linguistic alignment between interlocutors' …


Global Readiness of Language Technology for Healthcare: What would it Take to Combat the Next Pandemic?

I Mondal, K Ahuja, M Jain, JO Neil, K Bali… - arXiv preprint arXiv …, 2022

The COVID-19 pandemic has brought out both the best and worst of language technology (LT). On one hand, conversational agents for information dissemination and basic diagnosis have seen widespread use, and arguably, had an important …


Global-locality preserving projection for word embedding

B Wang, Y Sun, Y Chu, Z Yang, H Lin - … Journal of Machine Learning and Cybernetics, 2022

… -the Common Crawl corpus consisting of 840b tokens and a vocabulary containing 2.2 M words with 300-dimensional, the Common Crawl (… Here we use the pre-trained word embedding of the Common Crawl corpus (840b tokens, 2.2 M …


GPT-3 and Friends: Transformers in Natural Language Processing

In 2020, the British newspaper “The Guardian” published an editorial written in its entirety by an artificial intelligence called “Generative Pre-trained Transformer 3”. GPT-3 is one of several “Natural Language Processors”(NLPs), allowing a computer …


GPT-3 and InstructGPT: technological dystopianism, utopianism, and “Contextual” perspectives in AI ethics and industry

This paper examines the ethical solutions raised in response to OpenAI’s language model Generative Pre-trained Transformer-3 (GPT-3) a year and a half from its release. I argue that hype and fear about GPT-3, even within the Natural Language …


GPT-3 for Few-Shot Dialogue State Tracking

GPT-3 (Brown et al., 2020) has attracted considerable attention due to its superior performance across a wide range of Natural Language Processing (NLP) tasks, especially with its powerful and versatile in-context few-shot learning ability. That is …


GPT-NeoX-20B: An Open-Source Autoregressive Language Model

S Black, S Biderman, E Hallahan, Q Anthony, L Gao…

GPT-NeoX-20B is a 20 billion parameter autoregressive language model whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model …


GPTs at Factify 2022: Prompt Aided Fact-Verification

PK Sahu, S Aggarwal, T Gupta, G Das - arXiv preprint arXiv:2206.14913, 2022

S Aggarwal, P Sahu, T Gupta, G Das - Proceedings of De-Factify: Workshop on …, 2022

One of the most pressing societal issues is the fight against false news. The false claims, as difficult as they are to expose, create a lot of damage. To tackle the problem, fact verification becomes crucial and thus has been a topic of interest …


Grammatical Error Correction: A Survey of the State of the Art

C Bryant, Z Yuan, MR Qorib, H Cao, HT Ng, T Briscoe - arXiv preprint arXiv …, 2022

Grammatical Error Correction (GEC) is the task of automatically detecting and correcting errors in text. The task not only includes the correction of grammatical errors, such as missing prepositions and mismatched subject-verb agreement, but …


Granular Emotion Detection for Multi-Class Sentiment Analysis in Social Media

… XLNet was trained on many of the same or similar datasets as BERT and RoBERTa, including CommonCrawl, BooksCorpus, and English … A clean derivation of the CommonCrawl Corpus was used in pretraining XLM-R, with one …


GraphIC: A graph-based approach for identifying complaints from code-mixed product reviews

A Singh, S Saha - Expert Systems with Applications, 2022

… Wikipedia’s and Common Crawl’s publicly available corpora for 17 Indian languages were employed for monolingual data. PMINDIA and Dakshina corpora were used to obtain translated and transliterated data for parallel segments. For code-mixed …


GREENER: Graph Neural Networks for News Media Profiling

P Panayotov, U Shukla, HT Sencar, M Nabeel, P Nakov - arXiv preprint arXiv …, 2022

We study the problem of profiling news media on the Web with respect to their factuality of reporting and bias. This is an important but under-studied problem related to disinformation and "fake news" detection, but it addresses the issue at a …


GRILLBot: A flexible conversational agent for solving complex real-world tasks

C Gemmell, S Fischer, I Mackie, P Owoicho, F Rossetto…

… We use Common Crawl to download the raw HTML for target domains and develop website-specific wrappers to extract semi-structured information about each task, ie title, author, description, ingredients, images, steps, ratings, videos, etc. We …


GRILLBot: A multi-modal conversational agent for complex real-world tasks

C Gemmell, F Rossetto, I Mackie, P Owoicho, S Fischer… - Proceedings of the 23rd …, 2022

… We use Common Crawl to download the raw HTML for target domains and develop website-specific wrappers to extract semi-structured information about each task, ie title, author, description, ingredients, images, steps, ratings, videos, infoboxes, FAQs. …


GRILLBot: An Assistant for Real-World Tasks with Neural Semantic Parsing and Graph-Based Representations

C Gemmell, I Mackie, P Owoicho, F Rossetto, S Fischer… - arXiv preprint arXiv …, 2022


Grounded Keys-to-Text Generation: Towards Factual Open-Ended Generation

F Brahman, B Peng, M Galley, S Rao, B Dolan… - arXiv preprint arXiv …, 2022

… The documents are citations in the Wikipedia article obtained by CommonCrawl or web pages returned by Google Search. Each instance in our data has 40 grounding passages. Note that our dataset is distantly supervised, and these …


Grounding Language by Seeing, Hearing, and Interacting

As humans, our understanding of language is grounded in a rich mental model about "how the world works." As children, we learn this mental model gradually. We take in raw perceptual input about the world through all of our senses, and learn to …


Grounding Language Models on External Sources: A Comparison of Knowledge Graphs and Unstructured Text

… To get an idea of the size, WebText, for example, consists of 40GB of text from 8 million documents scraped from the internet, but an even larger example is the corpus used to train T5 [81], named C4, which consists of 750GB of text derived from …


GROWN+ UP: A Graph Representation Of a Webpage Network Utilizing Pre-training

B Yeoh, H Wang - arXiv preprint arXiv:2208.02252, 2022

… We can also pre-train on a more massive set of webpages from an identical time period as the target domain, as opposed to the relatively small CommonCrawl 2008 corpus that we used for all benchmarks in this work. Lastly, integrating image, audio …


HalBERT at PoliticEs 2022: Are Machine Learning Algorithms better for Author Profiling?

This paper presents an investigation of different machine and deep learning models for Author Profiling for the Shared Task PoliticEs: Spanish Author Profiling for Political Ideology at IberLEF 2022. We provide a broad comparative study to see …


Handling and Presenting Harmful Text

L Derczynski, HR Kirk, A Birhane, B Vidgen - arXiv preprint arXiv:2204.14256, 2022

… enormous internet-scraped datasets such as Common Crawl Corpus or WebText. As these unstructured corpora become larger, the risk of them containing harmful content increases, and the larger the dataset, the more difficult it is for humans to …


Hate Speech and Offensive Language Detection in Bengali

M Das, S Banerjee, P Saha, A Mukherjee - arXiv preprint arXiv:2210.03479, 2022

… 2020) is another form of Transformer model, pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. XLM-R was trained using a lot more data compared to m-BERT. Similar to BERT, it is a stack of transformer encoder …


Hate speech and offensive language detection in Dravidian languages using deep ensemble framework

PK Roy, S Bhawal, CN Subalalitha - Computer Speech & Language, 2022

… It is pretrained with 2.5 TB of CommonCrawl data in 100 languages. Due to the absence of language embedding it can result in much better performances in the case of code-switching and are much better compared to multilingual BERT …


HC4: A New Suite of Test Collections for Ad Hoc CLIR

D Lawrie, J Mayfield, D Oard, E Yang - arXiv preprint arXiv:2201.09992, 2022

… First, to develop a multilingual document collection that was easy to distribute, we chose the Common Crawl News Collection as the basis for the suite of collections. We applied automatic language identification to determine the language of each …


Health Talk: Understanding Practices of Popular Professional YouTubers

TT Phan, C Michoud, L Volpato, M del Rio Carral… - 2022

Practices related to health are circulated widely on YouTube. With a health psychology perspective, we present a study to understand health and wellbeing-related practices of a group of popular, professional YouTubers from the audio-visual …


HeLI-OTS, Off-the-shelf Language Identifier for Text

This paper introduces HeLI-OTS, an off-the-shelf text language identification tool using the HeLI language identification method. The HeLI-OTS language identifier is equipped with language models for 200 languages and licensed for academic as …


HICEM: A High-Coverage Emotion Model for Artificial Emotional Intelligence

B Wortman, JZ Wang - arXiv preprint arXiv:2206.07593, 2022

… Trained on Common Crawl and Wikipedia, this model provides a 300-dimensional embedding for each word. FastText was chosen over … We then once again proceeded with Facebook’s FastText models which have been trained on Common …


Hierarchical Agglomerative Graph Clustering in Poly-Logarithmic Depth

L Dhulipala, D Eisenstat, J Łącki, V Mirronki, J Shi - arXiv preprint arXiv:2206.11654, 2022

Obtaining scalable algorithms for hierarchical agglomerative clustering (HAC) is of significant interest due to the massive size of real-world datasets. At the same time, efficiently parallelizing HAC is difficult due to the seemingly sequential nature of the …


Hierarchical Graph Decomposition for Natural Language Generation

Central to natural language understanding and generation is the ability to accurately model the semantics of text. We can frame understanding as parsing a semantic form from a given sentence that represents all sentences of equivalent meaning …


Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding

L Liu, O Perez-Concha, A Nguyen, V Bennett, L Jorm - arXiv preprint arXiv …, 2022

International Classification of Diseases (ICD) coding plays an important role in systematically classifying morbidity and mortality data. In this study, we propose a hierarchical label-wise attention Transformer model (HiLAT) for the explainable …


Hierarchical template transformer for fine-grained sentiment controllable generation

L Yuan, J Wang, LC Yu, X Zhang - Information Processing & Management, 2022

Existing methods for text generation usually fed the overall sentiment polarity of a product as an input into the seq2seq model to generate a relatively fluent review. However, these methods cannot express more fine-grained sentiment polarity …


High-frequency words have higher frequencies in Turkish social sciences article

N Gürsakal, S Çelik, S Özdemir - Quality & Quantity, 2022

Abstract Words, sentences, and paragraphs are the basis of texts. When we consider texts as data and want to establish a relationship between qualitative and quantitative perspectives, we can do this with the word frequencies in a text. We aim …


High-Resource Methodological Bias in Low-Resource Investigations

M ter Hoeve, D Grangier, N Schluter - arXiv preprint arXiv:2211.07534, 2022

The central bottleneck for low-resource NLP is typically regarded to be the quantity of accessible data, overlooking the contribution of data quality. This is particularly seen in the development and evaluation of low-resource systems via down sampling …


Hints of Independence in a Pre-scripted World: On Controlled Usage of Open-domain Language Models for Chatbots in Highly Sensitive Domains

E Basar, I Hendrickx, E Krahmer, GJ de Bruijn, T Bosse - Proceedings of the 14th …, 2022

… Because the training approach is unsupervised, it is possible to include a massive amount of data from various online sources, such as Wikipedia and CommonCrawl, with minimal human labour required. Hence, these neural language …


Hints of Independence Independence Independence in a Pre-scripted e-scripted e-scripted World: On Controlled Usage of Open-domain Open-domain Open-domain …

EBE Basar, I Hendrickx, E Krahmer, GJ de Bruijn… - 2022

… Because the Because the Because the training approach is unsupervised, unsupervised, unsupervised, it is possible to include a massive amount of data from various online sources, such as Wikipedia and CommonCrawl, CommonCrawl …


Hipsters and the cool: A game theoretic analysis of identity expression, trends, and fads.

R Golman, EH Bugbee, A Jain, S Saraf - Psychological Review, 2021

Cultural trends and popularity cycles can be observed all around us, yet our theories of social influence and identity expression do not explain what perpetuates these complex, often unpredictable social dynamics. We propose a theory of social identity …


Homepage2Vec: Language-Agnostic Website Embedding and Classification

S Lugeon, T Piccardi, R West - Proceedings of the International AAAI Conference on …, 2022

… Following this intuition, we collected from Common Crawl,5 a large-scale sample of the Web, the 19 most frequent TLDs: .com, .org, .net, .info, … We consider the 30 most frequent metatags from a sample of the Common Crawl and represent their …


Homophobic and Hate Speech Detection Using Multilingual-BERT Model on Turkish Social Media

H Karayiğit, A Akdagli, Çİ ACI - Information Technology and Control, 2022

… The GloVe algorithm used in this study is trained on Common Crawl [24]. There are 253K words in the vocabulary and the dimension size is 300. Training data is web-crawled multilingual text with 2,736B … Available: https:// commoncrawl.org/2021 …


HOTTEST: Hate and offensive content identification in Tamil using transformers and enhanced STemming

R Rajalakshmi, S Selvaraj, P Vasudevan - Computer Speech & Language, 2022

… It is a transformer-based model which is pre-trained on about 2500 Gigabytes of filtered CommonCrawl data. This data consists of texts from about 100 languages. The model is trained in a self-supervised manner which makes the model more …


How can we detect Homophobia and Transphobia? Experiments in a multilingual code-mixed setting for social media governance

BR Chakravarthi, A Hande, R Ponnusamy… - International Journal of …, 2022

… XLM-R is pretrained over two terabytes of CommonCrawl data spanning over 100 languages to boost the performances in cross-lingual tasks. All these models use transformers as the building blocks of their architecture. The vital attribute is self-attention …


How Different is Stereotypical Bias in Different Languages?

Recent studies have demonstrated how to assess the stereotypical bias in pre-trained mono-lingual language models for the English language. In this master thesis, we extend this approach to multi-lingual models and multiple languages to show the …


How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?

A Araabi, C Monz, V Niculae - arXiv preprint arXiv:2208.05225, 2022

Neural Machine Translation (NMT) is an open vocabulary problem. As a result, dealing with the words not occurring during training (aka out-of-vocabulary (OOV) words) have long been a fundamental challenge for NMT systems. The predominant …


How good can an automatic translation of Pokémon names be?

L Talec-Bernarda - Proceedings of the Working Notes of CLEF, 2022

For those of you who are not familiar with the successful franchise of Pokémon, it revolves around imaginary creatures often representing animals mixed with objects, plants, etc. Their names reflect these characteristics and are, most of the time …


How Using a Foreign Language Influences Perceived Fairness

G Yu-Buck, A Mishra, H Mishra - ACR North American Advances, 2021

… We used pre-trained word embedding vectors that were trained with the fastText algorithm and the Common Crawl corpus. The association between the fairness construct and each review is the dependent variable. We followed the procedure of …


HTML Violations and Where to Find Them: A Longitudinal Analysis of Specification Violations in HTML

… To this end, we identify securityrelevant violations and leverage Common Crawl to check archived pages for these. Using this framework, we automatically analyze over 23K popular domains over time. This analysis reveals that while the number of …


Human Behavior in Domestic Environments: Prediction and Applications

… However, many deep learning algorithms work best under big data regimes, where the number of data samples is counted from the tens of thousands (eg the MNIST dataset) to 500 billion (the Common Crawl dataset used to train the GPT-3 …


Human-Guided Generation of Sketches and Prototypes

Sketching and prototyping are central to creative activities that improve and advance many aspects of human lives. They enable non-experts to express themselves through drawing, or help User Interface (UI) designers explore diverse alternatives …


HUMAN-INSPIRED COMPUTER VISION & REASONING

… We used 300-dimensional GloVe vectors for the word embedding layer, which are pre-trained on the Common Crawl corpus consisting of 840B tokens and a 2.2M words vocabulary list [6]. GloVe vectors act as a non-trainable base to learn …


HW-TSC's Submissions to the WMT 2022 General Machine Translation Shared Task

D Wei, Z Rao, Z Wu, S Li, Y Luo, Y Xie, X Chen… - Proceedings of the Seventh …, 2022

… It should be noted that in order to obtain better performance in the general domain, we mix the monolingual data from Common Crawl and News Crawl. … On En→Ru, after the Common Crawl monolingual is added, we observe 2.0 BLEU improvement …


Hybrid Architecture Based Intelligent Diagnosis Assistant for GP

R Wang, K Jayathunge, R Page, H Li, J Zhang, X Yang - Authorea Preprints, 2022

… As mentioned in the introduction part, there mainly four pre-trained word embedding models, because of the wide coverage of different semantic lexical, the ‘crawl-300d-2M.vec’[29] (2-millionword vectors trained on Common Crawl (600B tokens)) of the Google’s …


Hypernymy Detection for Low-resource Languages: A Study for Hindi, Bengali, and Amharic

A Jana, G Venkatesh, SM Yimam, C Biemann - Transactions on Asian and Low …, 2022

… Even though the focus of our study is not claiming superiority of the distributional semantic models over fastText, given that DT embeddings are prepared from a smaller corpus as compared to the corpus (Wikipedia, Common Crawl) on which …


HyperTree Proof Search for Neural Theorem Proving

G Lample, MA Lachaux, T Lavril, X Martinet, A Hayat… - arXiv preprint arXiv …, 2022

We propose an online training procedure for a transformer-based automated theorem prover. Our approach leverages a new search algorithm, HyperTree Proof Search (HTPS), inspired by the recent success of AlphaZero. Our model learns from …


iCAR: Bridging Image Classification and Image-text Alignment for Visual Recognition

Y Wei, Y Cao, Z Zhang, Z Yao, Z Xie, H Hu, B Guo - arXiv preprint arXiv:2204.10760, 2022

… It extracts image-caption pairs in random web pages crawled between 2014 and 2021 from the Common Crawl web data, and de-noises the dataset by scoring and filtering numerous raw pairs with a threshold using a CLIP ViT-B/32 [40] model …


iCompass at CheckThat! 2022: ARBERT and AraBERT for Arabic Checkworthy Tweet Identification

This paper provides a detailed overview of systems and its achieved results, which were produced as part of CLEF2022-Check-That! Lab Fighting the COVID-19 Infodemic and Fake News Detection. The task was carried out using transformers pre-trained …


Idea Mining From Online Reviews Using Transformation-Based Natural Language Processing Tasks

HF Mahdi, LK Gupta, T Choudhury, N Bansal - 2022 International Symposium on …, 2022

Product reviews from online communities are good sources to innovative ideas. Making use of this unstructured data from these reviews is a complicated process. The major task of identifying good ideas is ensuring the correct reviews are picked …


Identifying Implicitly Abusive Remarks about Identity Groups using a Linguistically Informed Approach

M Wiegand, E Eder, J Ruppenhofer - Proceedings of the 2022 Conference of the …, 2022

… regression classifier where each verb was represented by its (publicly available) word embedding induced on Common Crawl (Mikolov et al.… As features, we represented each verb by its word embedding from Common Crawl. (Using such a …


Identifying Opinion Based Questions in Developer Chat Communication

… We have used a general GloVe [108] that uses a common crawl with 42B tokens, 1.9M vocabulary, and 300− dimensional vectors. GloVe Twitter [108] was a common crawl with 27B tokens, 1.2M vocabulary, and 200− dimensional vectors. We have …


IDEOLOGY DETECTION USING TRANSFORMER-BASED MACHINE LEARNING MODELS

Ideology detection has been a challenging but essential problem that has been studied for a long time. Certain groups and organizations, such as politicians, rely on people’s political views to make wise or forward-looking decisions. In the previous …


IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension

RA Putri, A Oh - arXiv preprint arXiv:2210.13778, 2022

… These language models are mostly pre-trained on the common-crawl dataset, which may contain harmful biases or stereotypes. All datasets in this work are publicly available and distributed under CC BY-SA 4.0 license. Our data collection …


IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models

C Wang, X Liu, D Song - arXiv preprint arXiv:2210.14128, 2022

We introduce a new open information extraction (OIE) benchmark for pre-trained language models (LM). Recent studies have demonstrated that pre-trained LMs, such as BERT and GPT, may store linguistic and relational knowledge. In particular …


IgboBERT Models: Building and Training Transformer Models for the Igbo Language

C Chukwuneke, I Ezeani, P Rayson, M El-Haj

This work presents a standard Igbo named entity recognition (IgboNER) dataset as well as the results from training and fine-tuning state-of-the-art transformer IgboNER models. We discuss the process of our dataset creation-data collection and …


IIT Bombay's WMT22 Automatic Post-Editing Shared Task Submission

This paper describes IIT Bombay’s submission to the WMT-22 Automatic Post-Editing (APE) shared task for the English-Marathi (En-Mr) language pair. We follow the curriculum training strategy to train our APE system. First, we train an encoder-decoder …


Image Captioning for Thai Cultures

S Watcharabutsarakham, S Marukatat… - 2022 17th International Joint …, 2022

Before each trip, tourists generally gather information or photos from different places. This work aims at providing additional information about touristic sites in Thailand via automatic image captioning. Image captioning is the process of generating a …


Imbalanced data as risk factor of discriminating automated decisions

… experiment on the search engine Common Crawl [17]. The authors compared three techniques of machine learning for occupational classification with almost 400.000 collected biographies. In all cases, even without explicitly using gender …


Impact of convolutional neural network and FastText embedding on text classification

M Umer, Z Imtiaz, M Ahmad, M Nappi, C Medaglia… - Multimedia Tools and …, 2022

… It contains 2 million common crawl words with 300 −dimensions, providing 600 billion word-vectors. It uses hand-crafted n-grams as features in addition to single words. Text classification is performed very effectively and efficiently because of its …


Impact of preprocessing and word embedding on extreme multi-label patent classification tasks

G Jung, J Shin, S Lee - Applied Intelligence, 2022

… The Word2Vec pre-trained model learned approximately 3 million words using GoogleNews as a dataset, whereas the GloVe pre-trained model learned approximately 2.2 million words using the Common Crawl dataset [30, 31]. Because …


Impact of Pretraining Term Frequencies on Few-Shot Reasoning

Y Razeghi, RL Logan IV, M Gardner, S Singh - arXiv preprint arXiv:2202.07206, 2022

Pretrained Language Models (LMs) have demonstrated ability to perform numerical reasoning by extrapolating from a few examples in few-shot settings. However, the extent to which this extrapolation relies on robust reasoning is unclear. In this paper …


Impact of Sentence Representation Matching in Neural Machine Translation

H Jung, K Kim, JH Shin, SH Na, S Jung, S Woo - Applied Sciences, 2022

… To build a training set, we merged the Europarl parallel corpus and the Common-crawl corpus released from WMT-14 (http://www.statmt.org/wmt14/, accessed on 18 January 2022). We applied tokenizing and lowercasing and limited the length of …


Impact of Tokenization on Language Models: An Analysis for Turkish

C Toraman, EH Yilmaz, F Şahinuç, O Ozcelik - arXiv preprint arXiv:2204.08832, 2022

… OSCAR is a multilingual corpus that is obtained by filtering of the Common Crawl corpus that maintain an open repository of publicly available web pages. We use the split of this corpus prepared for Turkish. However, we observe that this split includes …


Implementación de algoritmos de reducción de sesgo en WEFE

Se ha demostrado en los últimos años que los modelos de word embeddings logran captar los sesgos presentes en los corpus de los que son entrenados, aprendiendo relaciones con estereotipos de género, raciales, entre otros. A modo de abordar el …


Implicit Social Cognition

Among the central features of the mind that drive social behavior are the attitudes and beliefs that humans bring to their engagement with other humans. Accordingly, ever since the inception of the field, research in social psychology has been guided …


Improved Zero-Shot Audio Tagging & Classification with Patchout Spectrogram Transformers

P Primus, G Widmer - arXiv preprint arXiv:2208.11402, 2022

… We choose Word2Vec and Glove models that are pre-trained on the Google News and the Common Crawl dataset, respectively. Both produce embedding vectors of size 300. To embed textual labels made out of multiple words (eg, fire truck) …


Improving Address Sequence Tagger Using Geographical Context

For humans, parsing an address into its components such as street and city is easy but time-consuming. Computers can have a harder time with such tasks. There are existing tools that use machine learning which has proven to be effective but still …


Improving BERTScore for Machine Translation Evaluation Through Contrastive Learning

… Scaled to one hundred languages, XLM-RoBERTa was pre-trained on 2.5TB of filtered CommonCrawl data (CC-100). However, it is … Furthermore, quality of data also poses challenges, as CommonCrawl or parallel localization data from projects …


Improving Bilingual Lexicon Induction with Cross-Encoder Reranking

Y Li, F Liu, I Vulić, A Korhonen - arXiv preprint arXiv:2210.16953, 2022

… Panlex-BLI does not provide processed WEs, so we follow the original instructions from the authors and adopt fastText WEs trained on Common Crawl + Wikipedia.The WEs are trimmed to the most frequent 200k words for each language …


Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles

S Ye, Y Xie, D Chen, Y Xu, L Yuan, C Zhu, J Liao - arXiv preprint arXiv:2211.16504, 2022

This paper focuses on analyzing and improving the commonsense ability of recent popular vision-language (VL) models. Despite the great success, we observe that existing VL-models still lack commonsense knowledge/reasoning ability (eg, "Lemons …


Improving Domain-specific Cross-lingual Embeddings with Automatically Generated Bilingual Dictionaries

P Singh, AR Terryn, E Lefever - Computational Linguistics in the Netherlands Journal, 2022

… To investigate this, we trained a set of FastText embeddings in English, initialised using the same Common Crawl and Wikipedia … Since the PubMed data would be vastly different from the Common Crawl and Wikipedia corpora, we can assume this …


Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Data

Y Zhang, C Xu, B Hu, C Zhang, T Xiao, J Zhu - arXiv preprint arXiv:2212.01778, 2022

We present a method for introducing a text encoder into pre-trained end-to-end speech translation systems. It enhances the ability of adapting one modality (ie, source-language speech) to another (ie, source-language text). Thus, the speech …


Improving Language Model Predictions via Prompts Enriched with Knowledge Graphs

R Brate, MH Dang, F Hoppe, Y He, A Meroño-Peñuela… - Workshop on Deep …, 2022

… These entities are embedded in HTML pages on the web using Microformat, Microdata and RDFa from the Common Crawl web corpus, the largest and most up-to-date data web corpus available to the public. As more and more websites embed …


Improving Neural Question Answering with Retrieval and Generation

Text-based Question Answering (QA) is a subject of interest both for its practical applications, and as a test-bed to measure the key Artificial Intelligence competencies of Natural Language Processing (NLP) and the representation and …


Improving open text matching in a communication serious game

… The ”clips/mfaq” model I am using for this application is pretrained on the ”MFAQ” [21] dataset, which is a multilingual corpus of Frequently Asked Questions parsed from the Common CrawlCommon Crawl is an open-source database of web crawling …


Improving searchability of datasets

Data is one of the most important digital assets in the world thanks to its business and social value. As is becoming increasingly available online, in order to use it effectively, we need tools that allow us to retrieve the most relevant datasets that …


Improving Sentiment Analysis By Emotion Lexicon Approach on Vietnamese Texts

AL Doan, ST Luu - arXiv preprint arXiv:2210.02063, 2022

The sentiment analysis task has various applications in practice. In the sentiment analysis task, words and phrases that represent positive and negative emotions are important. Finding out the words that represent the emotion from the text can …


Improving Short Text Classification With Augmented Data Using GPT-3

S Balkus, D Yan - arXiv preprint arXiv:2205.10981, 2022

GPT-3 is a large-scale natural language model developed by OpenAI that can perform many different tasks, including topic classification. Although researchers claim that it requires only a small number of in-context examples to learn a task, in …


Improving Taxonomy-based Categorization with Categorical Graph Neural Networks

T Du, K Chang, P Liu, R Zhang - 2021 IEEE International Conference on Big Data …, 2021

… Then each query was converted to a 300-dimensional embedding by averaging each word’s FastText embedding [23] (pretrained on the common-crawl dataset). Each FastText word embedding was L2 normalized before use. The query …


Improving the Accessibility of Arabic Electronic Theses and Dissertations (ETDs) with Metadata and Classification

Much research work has been done to extract data from scientific papers, journals, and articles. However, Electronic Theses and Dissertations (ETDs) remain an unexplored genre of data in the research fields of natural language processing and …


Improving the Robustness of Summarization Models by Detecting and Removing Input Noise

K Krishna, Y Zhao, J Ren, B Lakshminarayanan, J Luo… - arXiv preprint arXiv …, 2022

The evaluation of abstractive summarization models typically uses test data that is identically distributed as training data. In real-world practice, documents to be summarized may contain input noise caused by text extraction artifacts or data …


Improving Word Translation via Two-Stage Contrastive Learning

Y Li, F Liu, N Collier, A Korhonen, I Vulić - arXiv preprint arXiv:2203.08307, 2022

Word translation or bilingual lexicon induction (BLI) is a key cross-lingual task, aiming to bridge the lexical gap between different languages. In this work, we propose a robust and effective two-stage contrastive learning framework for the BLI …


In Situ Answer Sentence Selection at Web-scale

Z Zhang, T Vu, A Moschitti - arXiv preprint arXiv:2201.05984, 2022

… Question Sampling We sample the questions from a large collection of information inquiries collected from Web data, eg, CommonCrawl. … The candidates are extracted from 30MM passages, retrieved from a large index having …


In-context Examples Selection for Machine Translation

S Agrawal, C Zhou, M Lewis, L Zettlemoyer… - arXiv preprint arXiv …, 2022

… As XGLM is trained on monolingual Common Crawl snapshots, translation in any domain and language could be considered an out-of-domain task from the model’s perspective. However, we hypothesize that translation in specific domains like …


INAUGURAL ARTICLE by a Recently Elected Academy Member: When danger strikes: A linguistic tool for tracking America's collective response to threats

VK Choi, S Shrestha, X Pan, MJ Gelfand - Proceedings of the National Academy of …, 2022

In today’s vast digital landscape, people are constantly exposed to threatening language, which attracts attention and activates the human brain’s fear circuitry. However, to date, we have lacked the tools needed to identify threatening language …


IndicXNLI: Evaluating Multilingual Inference for Indian Languages

D Aggarwal, V Gupta, A Kunchukuttan - arXiv preprint arXiv:2204.08776, 2022

… XLM-RoBERTa is pre-trained using the common crawl monolingual data. mBERT (cased/uncased) includes pre-training on nine of eleven Indic lan… These model use the Indic enrich monolingual corpora: Common Crawl, Oscar and IndicCorp and …


IndicXTREME: A Multi-Task Benchmark For Evaluating Indic Languages

S Doddapaneni, R Aralikatte, G Ramesh, S Goyal… - arXiv preprint arXiv …, 2022

In this work, we introduce IndicXTREME, a benchmark consisting of nine diverse tasks covering 18 languages from the Indic sub-continent belonging to four different families. Across languages and tasks, IndicXTREME contains a total of 103 …


Indonesian Hoax News Classification with Multilingual Transformer Model and BERTopic

LB Hutama, D Suhartono - Informatica, 2022

… The most prominent development of the XLM-R model lies in the size of the dataset used where XLM-R is trained using 2.5TB of Common Crawl data in 100 languages, making the XLM-R model superior to previous multi-language models …


InfoDCL: A Distantly Supervised Contrastive Learning Framework for Social Meaning

C Zhang, M Abdul-Mageed, G Jawahar - arXiv preprint arXiv:2203.07648, 2022

Existing supervised contrastive learning frameworks suffer from two major drawbacks: (i) they depend on labeled data, which is limited for the majority of tasks in real-world, and (ii) they incorporate inter-class relationships based on instance-level information …


Inria-ALMAnaCH at the WMT 2022 shared task: Does Transcription Help Cross-Script Machine Translation?

JO Alabi, L Nishimwe, B Muller, C Rey, B Sagot… - Proceedings of the Seventh …, 2022

This paper describes the Inria ALMAnaCH team submission to the WMT 2022 general translation shared task. Participating in the language directions {cs,ru,uk}→en and cs↔uk, we experiment with the use of a dedicated Latin-script transcription …


Insights from heterogeneous data through transitive semantic relationships and text analytics

Many organisations are finding that the volume of information they need to analyse to make effective decisions is increasing. An important element in effective decision making is the ability to prioritise information quickly and accurately from a variety of …


Instance-Specific Feature Propagation for Referring Segmentation

C Liu, X Jiang, H Ding - IEEE Transactions on Multimedia, 2022

… We follow prior work [12], [36] and adopt the GloVe word embeddings [32] on Common Crawl 840B tokens. We use Adam with the base learning rate of 0.001 for optimization. Images are resized to 416 × 416 before sending to the network, and …


Integrating character-level and word-level representation for affect in Arabic tweets

AI Alharbi, P Smith, M Lee - Data & Knowledge Engineering, 2022

Affect tasks, which range from sentiment polarity classification to finer grained sentiment strength and emotional intensity detection, have become of increasing interest due to the vast amount of user-generated content and advanced learning …


Integrating Product Data using Deep Learning

R Peeters, C Bizer - Proceedings of the 7th bwHPC Symposium, 2022

… A successful approach for dealing with the heterogeneity of product offers is to combine deep learning-based matching techniques with large amounts of training data which can be extracted from Web corpora such as the Common Crawl. Training …


Intelligent Techniques to Accelerate Everyday Text Communication

People with some form of speech-or motor-impairments usually use a high-tech augmentative and alternative communication (AAC) device to communicate with other people in writing or in face-to-face conversations. Their text entry rate on these …


Interactive capsule network for implicit sentiment analysis

Y Qian, J Wang, D Li, X Zhang - Applied Intelligence, 2022

Existing sentiment analysis models mainly rely on evident emotive words within phrases. When the apparent emotional words within phrases are eliminated, the performance of these models will inevitably decrease. The implicit communication of …


Interactive Question Answering

Artificial intelligence has long aspired to create systems that can perform real-world activities and converse with humans using natural language. In the pursuit of this goal, Interactive Question Answering was proposed. Specifically for language …


International Journal of Knowledge Processing Studies

The speed and ease of production and loading of content on the web, the exponential increase in the volume and short life cycle of data in this space, have made the processes of collection, refinement, organization, storage, management …


Internet-augmented language models through few-shot prompting for open-domain question answering

A Lazaridou, E Gribovskaya, W Stokowiec, N Grigorev - arXiv preprint arXiv …, 2022

In this work, we aim to capitalize on the unique few-shot capabilities offered by large-scale language models to overcome some of their challenges with respect to grounding to factual and up-to-date information. Motivated by semi-parametric language models …


Interpreting Arabic Transformer Models

A Abdelali, N Durrani, F Dalvi, H Sajjad - arXiv preprint arXiv:2201.07434, 2022

Arabic is a Semitic language which is widely spoken with many dialects. Given the success of pre-trained language models, many transformer models trained on Arabic and its dialects have surfaced. While these models have been compared with …


Interpreting Logical Metonymy through Dense Paraphrasing

B Ye, J Tu, E Jezek, J Pustejovsky - Proceedings of the Annual Meeting of the …, 2022

… The extensive usage of C4 and the nature of it being sourced from the public Common Crawl web scrape ensure its huge volume and various language styles compared to other public datasets, which provides a “real world” text distribution for our task. …


Interpreting mental state decoding with deep learning models

AW Thomas, C Ré, RA Poldrack - Trends in Cognitive Sciences, 2022

In mental state decoding, researchers aim to identify the set of mental states (eg, experiencing happiness or fear) that can be reliably identified from the activity patterns of a brain region (or network). Deep learning (DL) models are highly …


Introducing EM-FT for Manipuri-English Neural Machine Translation

This paper introduces pretrained word embeddings for Manipuri, a low-resourced Indian language. The pretrained word embeddings based on fastText is capable of handling the highly agglutinative language Manipuri (mni). We then perform …


Introduction to Language Models

SM Jain - Introduction to Transformers for NLP, 2022

… It was trained using the largest corpus a model has ever been trained on, which is known as the Common Crawl , and it has 175 billion different parameters. This is made possible in part by the semi-supervised training method of a language model …


Intrusion detection based on keystroke biometrics and siamese neural networks

A Budžys, O Kurasova, V Medvedev - DAMSS: 13th conference on data analysis …, 2022

DAMSS-2022 is the 13th International Conference on Data Analysis Methods for Software Systems, held in Druskininkai, Lithuania. Every year at the same place and time. The exception was in 2020, when the world was gripped by the Covid-19 …


Investigating language relationships in multilingual sentence encoders through the lens of linguistic typology

R Choenni, E Shutova - Computational Linguistics, 2022

Multilingual sentence encoders have seen much success in cross-lingual model transfer for downstream NLP tasks. The success of this transfer is, however, dependent on the model’s ability to encode the patterns of cross-lingual similarity …


Investigating Opinions on Public Policies in Digital Media: Setting up a Supervised Machine Learning Tool for Stance Classification

C Viehmann, T Beck, M Maurer, O Quiring, I Gurevych - Communication Methods and …, 2022

Supervised machine learning (SML) provides us with tools to efficiently scrutinize large corpora of communication texts. Yet, setting up such a tool involves plenty of decisions starting with the data needed for training, the selection of an algorithm …


Investigating the Country of Origin and the Role of the. eu TLD in External Trade of European Union Member States

A Giannakoulopoulos, M Pergantis, L Limniati… - Future Internet, 2022

… that queried the Common Crawl Index servers for Web pages and collected all the unique domains found. Common Crawl is a non-profit foundation providing access to web information through an open database of web crawl data [28]. The …


Investigating the Efficient Use of Word Embedding with Neural-Topic Models for Interpretable Topics from Short Texts

R Murakami, B Chakraborty - Sensors, 2022

With the rapid proliferation of social networking sites (SNS), automatic topic extraction from various text messages posted on SNS are becoming an important source of information for understanding current social trends or needs. Latent …


Investigating the Quality of Static Anchor Embeddings from Transformers for Under-Resourced Languages

P Singh, O De Clercq, E Lefever - The 1st Annual Meeting of the ELRA/ISCA Special …, 2022

… This problem could be solved by adding more monolingual data (from Common Crawl, for example) for the anchor extraction step. We also … Since most methodologies use the Wikipedia and/or the Common Crawl corpus as initial pre-training …


Investigations of Performance and Bias in Human-AI Teamwork in Hiring

A Peng, B Nushi, E Kiciman, K Inkpen, E Kamar - arXiv preprint arXiv:2202.11812, 2022

In AI-assisted decision-making, effective hybrid (human-AI) teamwork is not solely dependent on AI performance alone, but also on its impact on human decision-making. While prior work studies the effects of model accuracy on humans, we endeavour …


Is It Out Yet? Automatic Future Product Releases Extraction from Web Data

… As each monthly snapshot of the Common Crawl data may include hundreds of millions of web pages, we have decided to use a simple heuristics to select web pages which discuss future product releases. Our approach is based on manually curating …


Is this Question Real? Dataset Collection on Perceived Intentions and Implicit Attack Detection

MS Mirzaei, K Meshgi, S Sekine - Proceedings of the ACM Web Conference 2022, 2022

… We used GloVe [38] word embedding (300d version on 840B Common Crawl data) for obtaining vector representations for words and Xavier initialization for the parameters. The batch size is set to 32, and the loss is calculated using binary cross-entropy …


Is Using an AI to Simplify a Scientific Text Really Worth It

TB Talec-Bernard - Proceedings of the Working Notes of CLEF, 2022

Nowadays, we encounter Artificial Intelligence (AI) everywhere and although it can help us save a lot of time when performing certain tasks such as translating or correcting a text, it can also lead to a loss in quality. When it comes to automatically …


Isomorphic Cross-lingual Embeddings for Low-Resource Languages

S Sannigrahi, J Read - arXiv preprint arXiv:2203.14632, 2022

… 2018) which uses Wikipedia dumps and Common Crawl for all languages. In addition to this, we use available parallel data between the following related language pairs respectively: English-Hindi (hi) for Nepali, English-Estonian (et) for …


IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces

K Marchisio, N Verma, K Duh, P Koehn - arXiv preprint arXiv:2210.05098, 2022

The ability to extract high-quality translation dictionaries from monolingual word embedding spaces depends critically on the geometric similarity of the spaces -- their degree of "isomorphism." We address the root-cause of faulty cross-lingual …


IT5: Large-scale Text-to-text Pretraining for Italian Language Understanding and Generation

G Sarti, M Nissim - arXiv preprint arXiv:2203.03759, 2022

… 2020), a large collection including roughly 750GB of webscraped English texts sourced from the CommonCrawl. This was subsequently cleaned employing heuristics aimed at removing templated fillers, text deduplication, Javascript code …


Jan Chołoniewski, B. Eng., M. Sc.

W pracy analizujemy i modelujemy działalność internetowych serwisów informacyjnych, łącząc ilościowe podejście medioznawcze z metodami fizyki statystycznej. Głównymi celami prezentowanych badań są:(i) charakterystyka …


Jointly Multi-Similarity Loss for Deep Metric Learning

L Zhang, S Shen, L Li, H Wang, X Li, J Lang - 2021 IEEE International Conference on …, 2021

… In particular, all of the input sentences are split and initialized in the word representation layer with the 300-dimensional GloVe word vectors pre-trained from the 840B Common Crawl corpus [24], and the embeddings of the out-of-vocabulary …


JParaCrawl v3. 0: A Large-scale English-Japanese Parallel Corpus

M Morishita, K Chousa, J Suzuki, M Nagata - arXiv preprint arXiv:2202.12607, 2022

… To list such parallel websites, we analyzed all the Common Crawl text archive data released from March 2019 to August 20213. We identified the language in the archive by CLD24 and listed 100,000 large websites that roughly have the same …


Just Rank: Rethinking Evaluation with Word and Sentence Similarities

B Wang, CCJ Kuo, H Li - arXiv preprint arXiv:2203.02679, 2022

Word and sentence embeddings are useful feature representations in natural language processing. However, intrinsic evaluation for embeddings lags far behind, and there has been no significant update since the past decade. Word and sentence …


Just-In-Time Obsolete Comment Detection and Update

Z Liu, X Xia, D Lo, M Yan, S Li - IEEE Transactions on Software Engineering, 2021

Comments are valuable resources for the development, comprehension and maintenance of software. However, while changing code, developers sometimes neglect the evolution of the corresponding comments, resulting in obsolete …


k 𝑘 k italic_k-Nearest Neighbor Augmented Neural Networks for Text Classification

In recent years, many deep-learning based models are proposed for text classification. This kind of models well fits the training set from the statistical point of view. However, it lacks the capacity of utilizing instance-level information from …


Keyword Extraction in Scientific Documents

SX Rao, P Piriyatamwong, P Ghoshal, S Nasirian… - arXiv preprint arXiv …, 2022

… As an additional hint to participants, document embeddings for each item in the train and test sets, as well as word embeddings for the entire corpus, were generated from a fastText model2 trained on the English Common Crawl dataset (cc.en.300.bin)3. …


Knowledge Complementarity between Seed Investors and Startup Founders: Lessons from Accelerators

Which entrepreneurs benefit most from accelerator programs? In this paper, we argue that the value-added by startup accelerators depends critically on whether the knowledge these supporting institutions provide is complementary to or a substitute …


Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks

C Peris, L Tan, T Gueudre, T Gojayev, V Wei, G Oz - arXiv preprint arXiv:2210.04834, 2022

… We then sample 50K utterances each, from the Wikipedia and Common Crawl data sets of the corresponding language. We create a tf-… CommonCrawl have relatively larger improvement in SemER, when compared to domains that are more …


Knowledge extraction from fictional texts

Knowledge extraction from text is a key task in natural language processing, which involves many sub-tasks, such as taxonomy induction, named entity recognition and typing, relation extraction, knowledge canonicalization and so on. By constructing …


Knowledge graph expansion using question answering by leveraging language models

… C4 is based on the Common Crawl Web Archive9, a massive collection of web-extracted text, with around 20 TB of text data added each month. The researchers took the massive collection of web-extracted text data with the actuality of April 2019. The …


KNOWLEDGE REPRESENTATIONS+ SCHEMATA


Korean and Multilingual Language Models Study for Cross-Lingual Post-Training (XPT)

S Son, C Park, J Lee, M Shim, C Lee, K Park, H Lim - Journal of the Korea …, 2022

It has been proven through many previous researches that the pretrained language model with a large corpus helps improve performance in various natural language processing tasks. However, there is a limit to building a large-capacity corpus for …


Kv2vec: A Distributed Representation Method for Key-value Pairs from Metadata Attributes

C Niu, W Zhang, S Byna, Y Chen - 2022 IEEE High Performance Extreme Computing …, 2022

… Common Crawl web corpus has the least error rate. Therefore, we choose to use the pre-trained dictionary generated from the word2vec model with Common Crawl … the pre-trained and retrofitted dictionary generated by the word2vec model on the …


L3Cube-MahaCorpus and MahaBERT: Marathi Monolingual Corpus, Marathi BERT Language Models, and Resources

R Joshi - arXiv preprint arXiv:2202.01159, 2022

We present L3Cube-MahaCorpus a Marathi monolingual data set scraped from different internet sources. We expand the existing Marathi monolingual corpus with 24.8M sentences and 289M tokens. We further present, MahaBERT, MahaAlBERT …


L3Cube-MahaHate: A Tweet-based Marathi Hate Speech Detection Dataset and BERT models

A Velankar, H Patil, A Gore, S Salunke, R Joshi - arXiv preprint arXiv:2203.13778, 2022

Social media platforms are used by a large number of people prominently to express their thoughts and opinions. However, these platforms have contributed to a substantial amount of hateful and abusive content as well. Therefore, it is important …


L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathi

A Joshi, A Kajale, J Gadre, S Deode, R Joshi - arXiv preprint arXiv:2211.11187, 2022

Sentence representation from vanilla BERT models does not work well on sentence similarity tasks. Sentence-BERT models specifically trained on STS or NLI datasets are shown to provide state-of-the-art performance. However, building these models …


Label Semantic Aware Pre-training for Few-shot Text Classification

A Mueller, J Krone, S Romeo, S Mansour, E Mansimov… - arXiv preprint arXiv …, 2022

In text classification tasks, useful information is encoded in the label names. Label semantic aware systems have leveraged this information for improved text classification performance during fine-tuning and prediction. However, use of label-semantics …


Landing AI on Networks: An equipment vendor viewpoint on Autonomous Driving Networks

… Similarly, recently hyped advances [21] in NLP relied on hundred billions of text tokens corpus such as Common Crawl [47]. Even in lesser massmediatized fields such as computational biology, it is fairly well recognized that astonishing advances [48] …


Language Agnostic Code-Mixing Data Augmentation by Predicting Linguistic Patterns

SS Li, K Murray - arXiv preprint arXiv:2211.07628, 2022

In this work, we focus on intrasentential code-mixing and propose several different Synthetic Code-Mixing (SCM) data augmentation methods that outperform the baseline on downstream sentiment analysis tasks across various amounts of …


Language Agnostic Multilingual Information Retrieval with Contrastive Learning

X Hu, X Chen, P Qi, D Kong, K Liu, WY Wang, Z Huang - arXiv preprint arXiv …, 2022

Multilingual information retrieval is challenging due to the lack of training datasets for many low-resource languages. We present an effective method by leveraging parallel and non-parallel corpora to improve the pretrained multilingual language …


Language and Gender

J Berger, R Boghrati, S Rathee, R Pogacar, A Mecit - ACR North American Advances, 2021

… We use Common Crawl and 11 million product reviews from Amazon and Yelp, to ask whether algorithms learn to associate women with … Common Crawl is a snapshot of the contents of the World Wide Web, a text repository of 2.45 billion web …


Language Chameleon: Transformation analysis between languages using Cross-lingual Post-training based on Pre-trained language models

S Son, C Park, J Lee, M Shim, C Lee, Y Jang, J Seo… - arXiv preprint arXiv …, 2022

… XLM-R XLM-R is a cross-lingual model (XLM), which is trained with commoncrawl corpus6 containing a hundred types of languages. Unlike previous XLM, it is based on RoBERTa. It has 550M … 6https://commoncrawl.org/ 7https://namu.wiki/ 8https://corpus.korean.go.kr/ …


Language Detoxification with Attribute-Discriminative Latent Space

JM Kwak, M Kim, SJ Hwang - arXiv preprint arXiv:2210.10329, 2022

Transformer-based Language Models (LMs) achieve remarkable performances on a variety of NLU tasks, but are also prone to generating toxic texts such as insults, threats, and profanities which limit their adaptations to the real-world applications …


Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion

K Shuster, M Komeili, L Adolphs, S Roller, A Szlam… - arXiv preprint arXiv …, 2022

… Knowledge Module Task To construct our knowledge task, we also start with Common Crawl, splitting it into sentences. We construct a Lucene5 search over Common Crawl, and then, for a given target sentence of a document, we find the …


Language Pre-Training and Auxiliary Tasks for Vision and Language Navigation

The Vision and Language Navigation task came to life from the idea that we can build a robot or an autonomous system that can be instructed in human language and that will navigate using the instructions given. For example, we tell the agent to “Go …


Language-Agnostic Website Embedding and Classification

S Lugeon, T Piccardi, R West - arXiv preprint arXiv:2201.03677, 2022

… Following this intuition, we collected from Common Crawl4, a large-scale sample of the Web, the 19 most frequent TLDs: .com, .org, .net, .info, … We consider the 30 most frequent metatags from a sample of the Common Crawl and represent their …


Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentation

Z Ding, T Hui, J Huang, X Wei, J Han, S Liu - … of the IEEE/CVF Conference on …, 2022

… tic inputs, we adopt LSTM [9] to extract language features from GloVe word embeddings [32] pretrained on Common Crawl with 840B tokens. The maximum length of the input sentence is 25. We set the frame interval δ = 6 for calculating the …


Large Language Models Struggle to Learn Long-Tail Knowledge

N Kandpal, H Deng, A Roberts, E Wallace, C Raffel - arXiv preprint arXiv:2211.08411, 2022

The internet contains a wealth of knowledge -- from the birthdays of historical figures to tutorials on how to code -- all of which may be learned by language models. However, there is a huge variability in the number of times a given piece of …


Large pre-trained language models contain human-like biases of what is right and wrong to do

P Schramowski, C Turan, N Andersen, CA Rothkopf… - Nature Machine Intelligence, 2022

Artificial writing is permeating our lives due to recent advances in large-scale, transformer-based language models (LMs) such as BERT, GPT-2 and GPT-3. Using them as pre-trained models and fine-tuning them for specific tasks, researchers have …


Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All

E Guetta, A Shmidman, S Shmidman, CS Shmidman… - arXiv preprint arXiv …, 2022

… Due to the lack of large public-domain textual corpora available for Modern Hebrew, we had to rely upon Wikipedia, Twitter, and the OSCAR common crawl Thus, other genres are unrepresented or underrepresented, including scientific …


Large-scale Bilingual Language-Image Contrastive Learning

B Ko, G Gu - arXiv preprint arXiv:2203.14463, 2022

This paper is a technical report to share our experience and findings building a Korean and English bilingual multimodal model. While many of the multimodal datasets focus on English and multilingual multimodal research uses machine-translated …


Large-scale product classification for efficient matching in procurement systems

We consider the problem of recommending relevant suppliers given detailed request context in a procurement setting. The fundamental recommendation in procurement systems is that a single query has potentially hundreds of relevant …


Large-scale product classification to recommend suppliers in procurement systems


Large-scale Semi-supervised Learning of Neural NLP Models

Deep neural network (DNN) based approaches have achieved remarkable performance in multiple research fields, such as natural language processing, computer vision, and speech processing. However, it is widely acknowledged that …


Large-scale study of web accessibility metrics

B Martins, C Duarte - Universal Access in the Information Society, 2022

… The pages were obtained from CommonCrawl Footnote 5 . CommonCrawl is an open corpus of web crawl data, including metadata and source of billions of web pages since 2013. We used the most recent crawls to obtain the URLs of the pages …


Laying the foundations for benchmarking open data automatically

… Common Crawl And Unlocking Web Archives For Research. Retrieved 2019-01-13 from https://www.forbes.com/sites/kalevleetaru/2017/09… Common Crawl And Unlocking Web Archives For Research. Retrieved 2019-01-13 from https://www.forbes.com/sites/kalevleetaru/2017/09…


Learner Profiling: Demographics Identification Based on NLP, Machine Learning, and MOOCs Metadata

Massive Open Online Courses (MOOCs) have become universal learning resources, and the COVID- 19 pandemic is rendering these platforms even more necessary. Many types of research are ongoing to improve the learning resources provided to …


Learning GUI Completions with User-defined Constraints

L Brückner, LA Leiva, A Oulasvirta - ACM Transactions on Interactive Intelligent …, 2022

A key objective in the design of graphical user interfaces (GUIs) is to ensure consistency across screens of the same product. However, designing a compliant layout is time-consuming and can distract designers from creative thinking. This …


Learning High-Order Semantic Representation for Intent Classification and Slot Filling on Low-Resource Language via Hypergraph.

X Qi, Y Gao, R Wang, M Zhao, S Cui, M Mortazavi - Mathematical Problems in …, 2022

… ey pretrained it on the Common Crawl and Wikipedia using fastText. is Indonesian word vector model is trained using CBOW with position-weights, in dimension 300 (ie, Wemb ∈ R1 × 300), with character n-grams of length 5, a window of size 5 and 10 …


Learning interpretable word embeddings via bidirectional alignment of dimensions with semantic concepts

LK Şenel, F Şahinuç, V Yücesoy, H Schütze, T Çukur… - Information Processing & …, 2022

We propose bidirectional imparting or BiImp, a generalized method for aligning embedding dimensions with concepts during the embedding learning phase. While preserving the semantic structure of the embedding space, BiImp makes dimensions …


Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings

D Bollegala - arXiv preprint arXiv:2204.12386, 2022

… Note that this is significantly smaller compared to vocabulary sizes and training corpora used in publicly available word embeddings (eg GloVe embeddings are trained on 42B tokens from Common Crawl corpus covering …


Learning on Streaming Graphs with Experience Replay

M Perini, G Ramponi, P Carbone, V Kalavri - 2022

… Text features have been generated using a 300-dimensional GloVe CommonCrawl word vector. Posts are linked with an edge if they have been commented by the same user and each edge bears the timestamp of the first comment. The …


Learning Sample Importance for Cross-Scenario Video Temporal Grounding

P Bao, Y Mu - arXiv preprint arXiv:2201.02848, 2022

The task of temporal grounding aims to locate video moment in an untrimmed video, with a given sentence query. This paper for the first time investigates some superficial biases that are specific to the temporal grounding task, and proposes a …


Learning the Morphological and Syntactic Grammars for Named Entity Recognition

M Sun, Q Yang, H Wang, M Pasquine, IA Hameed - Information, 2022

In some languages, Named Entity Recognition (NER) is severely hindered by complex linguistic structures, such as inflection, that will confuse the data-driven models when perceiving the word’s actual meaning. This work tries to alleviate …


Learning the Morphological and Syntactic Grammars for Named Entity Recognition. Information 2022, 13, 49

M Sun, Q Yang, H Wang, M Pasquine, IA Hameed - 2022


Learning to Enrich Query Representation with Pseudo-Relevance Feedback for Cross-lingual Retrieval

R Chandradevan, E Yang, M Yarmohammadi… - Proceedings of the 45th …, 2022

… To help picking the best performing model across the training steps and tune the free parameters, we used CLIR Common Crawl Collection (HC4) [13] as the development dataset. In this dataset, the exact same English topics are queried over …


Learning to Follow Instructions in Text-Based Games

M Tuli, AC Li, P Vaezipoor, TQ Klassen, S Sanner… - arXiv preprint arXiv …, 2022

Text-based games present a unique class of sequential decision making problem in which agents interact with a partially observable, simulated environment via actions and observations conveyed through natural language. Such observations typically …


Learning to translate by learning to communicate

CM Downey, LZ Liu, X Zhou, S Steinert-Threlkeld - arXiv preprint arXiv:2207.07025, 2022

We formulate and test a technique to use Emergent Communication (EC) with a pretrained multilingual model to improve on modern Unsupervised NMT systems, especially for low-resource languages. It has been argued that the currently …


Learning Trustworthy Web Sources to Derive Correct Answers and Reduce Health Misinformation in Search

D Zhang, A Vakili Tahami, M Abualsaud, M Smucker - 2022

When searching the web for answers to health questions, people can make incorrect decisions that have a negative effect on their lives if the search results contain misinformation. To reduce health misinformation in search results, we need to be …


Learning Two-Step Hybrid Policy for Graph-Based Interpretable Reinforcement Learning

T Mu, K Lin, F Niu, G Thattai - arXiv preprint arXiv:2201.08520, 2022

We present a two-step hybrid reinforcement learning (RL) policy that is designed to generate interpretable and robust hierarchical policies on the RL problem with graph-based input. Unlike prior deep reinforcement learning policies parameterized by an end-to-end …


Learnings from Federated Learning in the Real world

C Dupuy, TG Roosta, L Long, C Chung, R Gupta… - arXiv preprint arXiv …, 2022

Federated Learning (FL) applied to real world data may suffer from several idiosyncrasies. One such idiosyncrasy is the data distribution across devices. Data across devices could be distributed such that there are some "heavy devices" with …


Less is More: Task-aware Layer-wise Distillation for Language Model Compression

C Liang, S Zuo, Q Zhang, P He, W Chen, T Zhao - arXiv preprint arXiv:2210.01351, 2022

Layer-wise distillation is a powerful tool to compress large models (ie teacher models) into small ones (ie, student models). The student distills knowledge from the teacher by mimicking the hidden representations of the teacher at every intermediate layer …


Lessons learned from the evaluation of Spanish Language Models

R Agerri, E Agirre - arXiv preprint arXiv:2212.08390, 2022

Given the impact of language models on the field of Natural Language Processing, a number of Spanish encoder-only masked language models (aka BERTs) have been trained and released. These models were developed either within large projects …


Leveraging Natural Supervision for Language Representation Learning and Generation

M Chen - arXiv preprint arXiv:2207.10617, 2022

Recent breakthroughs in Natural Language Processing (NLP) have been driven by language models trained on a massive amount of plain text. While powerful, deriving supervision from textual resources is still an open question. For example, language …


LeviRANK: Limited Query Expansion with Voting Integration for Document Retrieval and Ranking

A Rana, P Golchha, R Juntunen, A Coajă… - Working Notes Papers of the …, 2022

In order to make informed decisions in personal life, the information available on the internet is often overwhelming, and thus comparative decision-making is particularly challenging. Given the plethora of online resources, it often results in wastage of …


Lexical Semantics with Large Language Models: A Case Study of English break

Large neural language models (LLMs) can be powerful tools for research in lexical semantics. We illustrate this potential using the English verb break, which has numerous senses and appears in a wide range of syntactic frames. We show that …


Lightweight diacritics restoration for V4 languages

… In the case of HunWeb2, we used the "2017-2018" part of the Common Crawl subcorpus. Not splitting up the documents ensures that the train and dev data do not correlate too much, as we have found that there can be a lot of repeated sentences …


Limitations and Challenges of Unsupervised Cross-lingual Pre-training

MQ Zaragoza, F Casacuberta - Proceedings of the 15th biennial conference of the …, 2022

Cross-lingual alignment methods for monolingual language representations have received notable attention in recent years. However, their use in machine translation pre-training remains scarce. This work tries to shed light on the effects of some of the …


Linguistic and Emotion-based Identification of Tweets with Fake News: A Case Study

Since the popularization of the term in 2016, we have observed a proliferation of so called “fake news” content, assisted by the widespread use of social media platforms. The dissemination of fake content brings with it serious political, economical, and …


Linguistic Approaches to the Analysis of Online Terrorist Threats

This chapter focuses on online terrorist threats. For many linguists, the best training for a forensic linguist is a course in descriptive and applied linguistics because each case will normally require a different selection of tools from the linguistic toolbox …


Linguistic resources for Romanian. Quality assessed machine-translated and post-edited Romanian corpus and opportunities for interdisciplinary research

G BULGARU - REVUE INTERNATIONALE D'ÉTUDES EN LANGUES …

Despite the great progress over the last years, Romanian is still a low-resourced language. In this paper, we set to make a short overview of the language technology field and the linguistic resources available for Romanian. We also sketch out a short …


LinkingPark: An automatic semantic table interpretation system

S Chen, A Karaoglu, C Negreanu, T Ma, JG Yao… - Journal of Web Semantics, 2022

In this paper, we present LinkingPark, an automatic semantic annotation system for tabular data to knowledge graph matching. LinkingPark is designed as a modular framework which can handle Cell-Entity Annotation (CEA), Column-Type Annotation …


liveopp/keras-quora-question-pairs

GPK Tool, JFMAM Jun, JASON Dec, JSMTW Thu, F Sat

… Max bag-of-embeddings (this work) GloVe Common Crawl (840B tokens, 300D) 0.83 "Multi-Perspective-LSTM" [5] GloVe Common Crawl (840B tokens, 300D) 0.83 … "Max-out Window Encoding" with depth 2 [7] GloVe Common Crawl pruned to 1M …


LLM. int8 (): 8-bit Matrix Multiplication for Transformers at Scale

T Dettmers, M Lewis, Y Belkada, L Zettlemoyer - arXiv preprint arXiv:2208.07339, 2022

Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for …


lo-fi: distributed fine-tuning without communication

M Wortsman, S Gururangan, S Li, A Farhadi, L Schmidt… - arXiv preprint arXiv …, 2022

… -tuning OPT language models (up to 1.3B parameters) on Common Crawl. By removing the communication requirement, lo-fi reduces resource … Finally, we test lo-fi beyond computer vision by fine-tuning OPT-125M and OPT-1.3B [72] on …


Location-based Twitter Filtering for the Creation of Low-Resource Language Datasets in Indonesian Local Languages

M Amien, C Feng, H Huang - arXiv preprint arXiv:2206.07238, 2022

Twitter contains an abundance of linguistic data from the real world. We examine Twitter for user-generated content in low-resource languages such as local Indonesian. For NLP to work in Indonesian, it must consider local dialects …


Longitudinal analysis of sentiment and emotion in news media headlines using automated labelling with Transformer language models

D Rozado, R Hughes, J Halberstadt - Plos one, 2022

This work describes a chronological (2000–2019) analysis of sentiment and emotion in 23 million headlines from 47 news media outlets popular in the United States. We use Transformer language models fine-tuned for detection of sentiment (positive …


Low-level factors increase gaze-guidance under cognitive load: A comparison of image-salience and semantic-salience models

… GloVe is trained on large web-based datasets, we used the Common Crawl dataset comprised of 840 billion tokens and 2.2 million words. GloVe categorizes words along feature dimensions to create a similarity web of various terms based on …


Low-Resource Authorship Style Transfer with In-Context Learning

A Patel, N Andrews, C Callison-Burch - arXiv preprint arXiv:2212.08986, 2022

… Other models were trained on variants of Common Crawl data. To the best of our knowledge, using various search … Common Crawl, as a breadth-first crawler, only contains a small amount of Reddit data, often a few thousand pages per month …


LPT: Language-agnostic Prompt Tuning

Y Ji, Y Wang, Z Tang - Sci China Inf Sci

… The pre-trained XLM-RoBERTa-base model contains 270M parameters, which is trained on 2.5 TB CommonCrawl data in 100 languages. Its vocabulary contains 250k tokens. The architecture of XLMRoBERTa-base is the same as RoBERTa-base. …


Machine Comprehension System in Tamil and English based on BERT

G Srivatsun, S Thivaharan, BK KS, S Sudharsan - 2022 3rd International Conference …, 2022

… XLM-R [6] used more than 2 terabytes of data filtered by Common Crawl to train a Transformer-based masked language model on 100 languages. It also gave a complete empirical study of the essential aspects that are necessary to improve cross-lingual …


Machine Learning and Deep Learning Based Phishing Websites Detection: The Current Gaps and Next Directions

K Adane, B Beyene - Review of Computer Engineering Research, 2022

There are many phishing websites detection techniques in literature, namely white-listing, black-listing, visual-similarity, heuristic-based, and others. However, detecting zero-hour or newly designed phishing website attacks is an inherent property of machine …


Machine learning and deep learning-based Natural Language Processing for auto-vetting the appropriateness of Lumbar Spine Magnetic Resonance Imaging …

A Alanazi, A Cradock, J Ryan, L Rainford - Informatics in Medicine Unlocked, 2022

Manual vetting of radiology referrals is an essential daily task to ensure the appropriateness of the received referrals. Such tasks require sufficient clinical experience and may challenge the radiology staff. With the emerging of artificial …


Machine Learning approaches for Topic and Sentiment Analysis in multilingual opinions and low-resource languages: From English to Guarani

This dissertation has focused on the study of machine learning techniques for sentiment analysis and topic modeling in texts from social media. It puts a special emphasis on approaches and methods for handling low-resource languages, ie …


Machine Reading at Scale: A Search Engine for Scientific and Academic Research

N Sousa, N Oliveira, I Praça - Systems, 2022

The Internet, much like our universe, is ever-expanding. Information, in the most varied formats, is continuously added to the point of information overload. Consequently, the ability to navigate this ocean of data is crucial in our day-to-day …


Machine Reading at Scale: A Search Engine for Scientific and Academic Research. Systems 2022, 10, 43


Machine Translation Performance Prediction System: Optimal Prediction for Optimal Translation

Abstract Machine translation performance prediction (MTPP) system (MTPPS) is an automatic, accurate, language and natural language processing (NLP) output independent prediction model. MTPPS is optimal by the capability to predict …


Machine's Conceptual Development Using FNet

Masked language modeling (MLM) is a well-known technique in Natural languageprocessing (NLP) to train a model on randomly masked tokens and use the trainedmodel to predict the masked words. FNet is a recently developed Fourier …


Making AI work for skills-based training: A case study

R Robson, E Kelsey, A Goel, L Egerton, S Nasir…

Motivated by the need to upskill workers rapidly and equitably in response to changes in work and the workplace, a team funded by the US National Science Foundation Convergence Accelerator program has developed and piloted an …


MalGen: Malware Generation with Specific Behaviors to Improve Machine Learning-based Detectors.

M Smith, A Carbajal, E Domschot, NT Johnson… - 2022

In recent years, infections and damage caused by malware have increased at exponential rates. At the same time, machine learning (ML) techniques have shown tremendous promise in many domains, often out performing human efforts by …


MANER: Mask Augmented Named Entity Recognition for Extreme Low-Resource Languages

S Sonkar, Z Wang, RG Baraniuk - arXiv preprint arXiv:2212.09723, 2022

This paper investigates the problem of Named Entity Recognition (NER) for extreme low-resource languages with only a few hundred tagged data samples. NER is a fundamental task in Natural Language Processing (NLP). A critical driver …


Manifold's English-Chinese System at WMT22 General MT Task

C Jin, T Shi, Z Xue, X Lin - Proceedings of the Seventh Conference on Machine …, 2022

Manifold’s English-Chinese System at WMT22 is an ensemble of 4 models trained by different configurations with scheduled sampling-based fine-tuning. The four configurations are DeepBig (XenC), DeepLarger (XenC), DeepBig-TalkingHeads (XenC) …


Many Languages, One Deep Learning Model Multilingual understanding is easier than you think!

How do we enable natural language-based deep learning systems to understand data in many languages? One naive approach would be to simply train a separate deep learning model on each of the desired languages. But, this approach is pretty …


Marathi Social Media Opinion Mining using XLM-R

N Rathod, N Mistry, D Talati, M Parikh, A Kore… - … International Conference on …, 2022

… XLM-R is a transformer-based multilingual masked model pre-trained on CommonCrawl data in 100 languages with base having 250M parameters, which obtains state-of-the-art results in the tasks of question answering, sequence labeling …


Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability

Y Fujinuma, J Boyd-Graber, K Kann - arXiv preprint arXiv:2203.10753, 2022

Pretrained multilingual models enable zero-shot learning even for unseen languages, and that performance can be further improved via adaptation prior to finetuning. However, it is unclear how the number of pretraining languages …


Matching Tweets With Applicable Fact-Checks Across Languages

A Kazemi, Z Li, V Pérez-Rosas, SA Hale, R Mihalcea - arXiv preprint arXiv …, 2022

… The model is trained on more than 2TBs CommonCrawl data and supports one hundred languages. We also rely on recent language … language modeling trained on 17 billion monolingual sentences from CommonCrawl and 6 billion translated …


Measuring document similarity with weighted averages of word embeddings.

B Seegmiller, D Papanikolaou, LDW Schmidt - Explorations in Economic History, 2022

We detail a methodology for estimating the textual similarity between two documents while accounting for the possibility that two different words can have a similar meaning. We illustrate the method’s usefulness in facilitating comparisons between …


Measuring Gender Bias in Word Embeddings of Gendered Languages Requires Disentangling Grammatical Gender Signals

SO Sabbaghi, A Caliskan - arXiv preprint arXiv:2206.01691, 2022

Does the grammatical gender of a language interfere when measuring the semantic gender information captured by its word embeddings? A number of anomalous gender bias measurements in the embeddings of gendered languages suggest this …


Measuring harmful and toxic representations in Scandinavian language models

… Combination of Danish texts from Common Crawl, Wikipedia, debate forums, and OpenSubtitles. DanishRoBERTa Danish subset of mC4 (from the Common Crawl). …


Measuring Harmful Representations in Scandinavian Language Models

S Touileb, D Nozza - arXiv preprint arXiv:2211.11678, 2022

… trained on subsets of the Common Crawl corpus. The remaining four models were trained on datasets comprising languagespecific subsets from the Common Crawl. As previous works have shown that this corpus contains various types of …


MEASURING NETWORK INTERFERENCE AND MITIGATING IT WITH DNS ENCRYPTION

The Internet has emerged as one of the most important tools of communication. With around 4.5 billion active users as of July 2020, it provides people the opportunity to access a vast treasure trove of information and express their opinions online. How-ever …


Measuring Similarity by Linguistic Features rather than Frequency

… As said above, we used BERT – with the Italian model taken from UWAC corpus, Umbertocommoncrawl - and examined the output of the first or projection layer5. In this way we intended to check the predicting ability of BERT on the masked word, by …


Measuring the Impact of (Psycho-) Linguistic and Readability Features and Their Spill Over Effects on the Prediction of Eye Movement Patterns

D Wiechmann, Y Qiao, E Kerz, J Mattern - arXiv preprint arXiv:2203.08085, 2022

… BERT is trained on the BooksCorpus (800M words) and Wikipedia (2,500M words), whereas GPT-2 is trained on WebText, an 8-million documents subset of CommonCrawl amounting to 40 GB of text. We chose the BERT base model (cased) …


Memory in humans and deep language models: Linking hypotheses for model augmentation

O Raccah, P Chen, TL Willke, D Poeppel, VA Vo - arXiv preprint arXiv:2210.01869, 2022

The computational complexity of the self-attention mechanism in Transformer models significantly limits their ability to generalize over long temporal durations. Memory-augmentation, or the explicit storing of past information in external memory …


Mental Health Disorder Identification From Motivational Conversations

T Saha, SM Reddy, S Saha, P Bhattacharyya - IEEE Transactions on Computational …, 2022

… 1) Embedding Features: To extract textual features of an utterance U having nu number of words, the representation of each of the words, w1,...,wu, where wi ∈ Rdu , wi s are obtained from the GloVe embeddings trained on the Common Crawl …


Menu close

J Ogden, M Kurzmeier, F Clavert, M Currie - 2022

The aim of this guide is to provide a starting point for using web archives as a source for social science research. This guide provides a primer for how to approach web archives research design by first outlining what web archives are and how they differ …


Meta-Learning for Offensive Language Detection in Code-Mixed Texts

G Vadakkekara Suresh, BR Chakravarthi, JP McCrae - Forum for Information …, 2021

This research investigates the application of Model-Agnostic Meta-Learning (MAML) and ProtoMAML to identify offensive code-mixed text content on social media in Tamil-English and Malayalam-English code-mixed texts. We follow a two-step …


METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals

P Bajaj, C Xiong, G Ke, X Liu, D He, S Tiwary, TY Liu… - arXiv preprint arXiv …, 2022

We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model. Originated in ELECTRA, this training strategy has demonstrated sample-efficiency to pretrain …


mFACE: Multilingual Summarization with Factual Consistency Evaluation

R Aharoni, S Narayan, J Maynez, J Herzig, E Clark… - arXiv preprint arXiv …, 2022

Abstractive summarization has enjoyed renewed interest in recent years, thanks to pre-trained language models and the availability of large-scale datasets. Despite promising results, current models still suffer from generating factually inconsistent …


mGPT: Few-Shot Learners Go Multilingual

O Shliazhko, A Fenogenova, M Tikhonova, V Mikhailov… - arXiv preprint arXiv …, 2022

Recent studies report that autoregressive language models can successfully solve many NLP tasks via zeroand few-shot learning paradigms, which opens up new possibilities for using the pre-trained language models. This paper introduces two …


MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

L Fan, G Wang, Y Jiang, A Mandlekar, Y Yang, H Zhu… - arXiv preprint arXiv …, 2022

Autonomous agents have made great strides in specialist domains like Atari games and Go. However, they typically learn tabula rasa in isolated environments with limited and manually conceived objectives, thus failing to generalize across a wide …


Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training

K Marchisio, P Lewis, Y Chen, M Artetxe - arXiv preprint arXiv:2212.10503, 2022

… 2020) as our training corpus, which is a filtered version of CommonCrawl. We report the full list of languages along with their corpus size and linguistic details in Table 1. Each language is preprocessed individually using SentencePiece (Kudo …


minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models

K Misra - arXiv preprint arXiv:2203.13112, 2022

We present minicons, an open source library that provides a standard API for researchers interested in conducting behavioral and representational analyses of transformer-based language models (LMs). Specifically, minicons enables …


Misreporting and Fake News Detection Techniques on the Social Media Platform

Y Feng - Highlights in Science, Engineering and Technology, 2022

… RealNews is also a news article dataset scraped from Common Crawl with newspaper python library. The training sample includes the news dump from Common Crawl ranging from December 2016 to March 2019, and the articles in …


Misspelling Semantics in Thai

User-generated content is full of misspellings. Rather than being just random noise, we hypothesise that many misspellings contain hidden semantics that can be leveraged for language understanding tasks. This paper presents a fine-grained …


MLGL-MP: a Multi-Label Graph Learning framework enhanced by pathway interdependence for Metabolic Pathway prediction

BX Du, PC Zhao, B Zhu, SM Yiu, AK Nyamabo, H Yu… - Bioinformatics, 2022

Motivation During lead compound optimization, it is crucial to identify pathways where a drug-like compound is metabolized. Recently, machine learning-based methods have achieved inspiring progress to predict potential metabolic pathways …


Model and Data Transfer for Cross-Lingual Sequence Labelling in Zero-Resource Settings

I García-Ferrero, R Agerri, G Rigau - arXiv preprint arXiv:2210.12623, 2022

Zero-resource cross-lingual transfer approaches aim to apply supervised models from a source language to unlabelled target languages. In this paper we perform an in-depth study of the two main techniques employed so far for cross-lingual zero-resource …


Modeling Associative Reasoning Processes

U Furbach, C Schon, M Ragni - arXiv preprint arXiv:2201.00716, 2022

The human capability to reason about one domain by using knowledge of other domains has been researched for more than 50 years, but models that are formally sound and predict cognitive process are sparse. We propose a formally sound …


Modeling Intensification for Sign Language Generation: A Computational Approach

M İnan, Y Zhong, S Hassan, L Quandt, M Alikhani - arXiv preprint arXiv:2203.09679, 2022

End-to-end sign language generation models do not accurately represent the prosody in sign language. A lack of temporal and spatial variations leads to poor-quality generated presentations that confuse human interpreters. In this paper, we aim to …


Modeling Target-Side Morphology in Neural Machine Translation: A Comparison of Strategies

MWD Marco, M Huck, A Fraser - arXiv preprint arXiv:2203.13550, 2022

Morphologically rich languages pose difficulties to machine translation. Machine translation engines that rely on statistical learning from parallel training data, such as state-of-the-art neural systems, face challenges especially with rich morphology …


Modelling Commonsense Properties using Pre-Trained Bi-Encoders

A Gajbhiye, L Espinosa-Anke, S Schockaert - arXiv preprint arXiv:2210.02771, 2022

Grasping the commonsense properties of everyday concepts is an important prerequisite to language understanding. While contextualised language models are reportedly capable of predicting such commonsense properties with human-level …


MODERN INFORMATION TECHNOLOGIES AND IT-EDUCATION

… The experiments in this section are performed with a CommonCrawl data . The CommonCrawl is an open repository of web crawl data. Said … The vocabulary of the CommonCrawl text is plenty rich due to nearly all human languages present in …


MOE/RF: A Novel Phishing Detection Model based on Revised Multi-Objective Evolution Optimization Algorithm and Random Forest

E Zhu, Z Chen, J Cui, H Zhong - IEEE Transactions on Network and Service …, 2022

To effectively boost computer usage, machine learning models are used in several phishing detection systems to classify enormous phishing datasets. Based on phishing patterns, researchers prefer to extract a considerable number of features to …


Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Models

AH Zadeh, M Mahmoud, A Abdelhadi, A Moshovos - arXiv preprint arXiv:2203.12758, 2022

Increasingly larger and better Transformer models keep advancing state-of-the-art accuracy and capability for Natural Language Processing applications. These models demand more computational power, storage, and energy. Mokey reduces …


Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi

A Velankar, H Patil, R Joshi - arXiv preprint arXiv:2204.08669, 2022

Transformers are the most eminent architectures used for a vast range of Natural Language Processing tasks. These models are pre-trained over a large text corpus and are meant to serve state-of-the-art results over tasks like text classification. In …


MonoByte: A Pool of Monolingual Byte-level Language Models

H Abonizio, LR de Souza, R Lotufo, R Nogueira - arXiv preprint arXiv:2209.11035, 2022

… 2021b), the same corpus used to train mT5 and ByT5, which comprises 101 natural languages extracted from the Common Crawl web scrape. For each model, we selected only the documents written in the specific language, which was originally …


Moving Stuff Around

When training neural rankers with Large Language models, a practitioner generally leverages multiple GPUs, which increases the amount of available VRAM, allowing larger batches, and decreasing training time. However, managing how data moves …


MSCTD: A Multimodal Sentiment Chat Translation Dataset

Y Liang, F Meng, J Xu, Y Chen, J Zhou - arXiv preprint arXiv:2202.13645, 2022

Multimodal machine translation and textual chat translation have received considerable attention in recent years. Although the conversation in its natural form is usually multimodal, there still lacks work on multimodal machine translation in …


mSLAM: Massively multilingual joint pre-training for speech and text

A Bapna, C Cherry, Y Zhang, Y Jia, M Johnson… - arXiv preprint arXiv …, 2022

We present mSLAM, a multilingual Speech and LAnguage Model that learns cross-lingual cross-modal representations of speech and text by pre-training jointly on large amounts of unlabeled speech and text in multiple languages. mSLAM combines w2v-BERT …