Learning adaptive representations for entity recognition in the biomedical domain

I Lauriola, F Aiolli, A Lavelli, F Rinaldi - Journal of Biomedical Semantics, 2021

Named Entity Recognition is a common task in Natural Language Processing applications, whose purpose is to recognize named entities in textual documents. Several systems exist to solve this task in the biomedical domain …


Learning Mathematical Properties of Integers

M Ryskina, K Knight - arXiv preprint arXiv:2109.07230, 2021

… Ours OEIS 14M 133,004 GloVe–840B–300D Common Crawl 840B 64,729 … (2019), we choose three embedding models pre-trained on English text corpora: • GloVe–840B–300D: 300-dimensional GloVe vectors trained …


Learning policy scheduling for text augmentation

S Li, X Ao, F Pan, Q He - Neural Networks, 2021

Abstract When training deep learning models, data augmentation is an important technique to improve the performance and alleviate overfitting. In natural language processing (NLP), existing augmentation methods often use fixed strategies …


Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy

D Bollegala, Y Zhou - Proceedings of the 35th Pacific Asia Conference on …, 2021

Y Zhou, D Bollegala - arXiv preprint arXiv:2110.02204, 2021

… context t . 3 Experiments 3.1 Experimental Setup In our experiments, we use the pretained GloVe3 em- beddings (Common Crawl with 840B tokens and 2.2M vocabulary) as the static word embeddings g(u) with p = 300. We use …


Learning Sentiment Sentence Representation with Multiview Attention Model

Y Zhang, J Wang, X Zhang - Information Sciences, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Page 1. Learning Structured Neural Semantic Parsers Pengcheng Yin CMU LTI 21 014 August 16, 2021 Language Technologies Institute School of Computer Science Carnegie Mellon University 5000 Forbes Ave., Pittsburgh, PA 15123 www.lti.cs.cmu.edu …


Learning to Lemmatize in the Word Representation Space

… Data We validate the approach on Finnish language, using pretrained embeddings provided by the fastText library (Bojanowski et al., 2016). The embeddings were trained on Common Crawl and Wikipedia corpora and have dimensionality of d= 300 …


Learning to rank for Consumer Health Search

H Yang, X Liu, B Zheng, G Yang - CLEF 2021 Evaluation Labs and Workshop: Online …, 2021

… Totally, the collection consists of over 5 million medical webpages from selected domains acquired from the CommonCrawl and other resources [4]. 3.3. Topics Totally 55 topics are used in the CLEF 2021 CHS task, and they are based on realistic search scenarios …


Learning To Retrieve Prompts for In-Context Learning

O Rubin, J Herzig, J Berant - arXiv preprint arXiv:2112.08633, 2021

In-context learning is a recent paradigm in natural language understanding, where a large pre-trained language model (LM) observes a test instance and a few training examples as its input, and directly decodes the output without any update to its …


Page 1. Leveraging Longitudinal Data for Personalized Prediction and Word Representations by Charles Welch A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Computer …


Leveraging Transformers for Hate Speech Detection in Conversational Code-Mixed Tweets

ZM Farooqi, S Ghosh, RR Shah - arXiv preprint arXiv:2112.09986, 2021

… • XLM-Roberta (XLM-R) [42] is a transformer-based masked language model pretrained on Common Crawl data having about 100 languages. It was proposed by Facebook and happened to be one of the best-performing transformer models for …


Lexicon Development for COVID-19-related Concepts Using Open-source Word Embedding Sources: An Intrinsic and Extrinsic Evaluation

S Parikh, A Davoudi, S Yu, C Giraldo, E Schriver… - JMIR Medical Informatics, 2021

Background: Scientists are developing new computational methods and prediction models to better clinically understand COVID-19 prevalence, treatment efficacy, and patient outcomes. These efforts could be improved by leveraging …


Light gradient boosting machine-based phishing webpage detection model using phisher website features of mimic URLs

E Oram, PB Dash, B Naik, J Nayak, S Vimal, SK Nataraj - Pattern Recognition Letters, 2021


LILE: Look In-Depth before Looking Elsewhere--A Dual Attention Network using Transformers for Cross-Modal Information Retrieval in Histopathology Archives

The volume of available data has grown dramatically in recent years in many applications. Furthermore, the age of networks that used multiple modalities separately has practically ended. Therefore, enabling bidirectional cross-modality …


Limitations of Knowledge Distillation for Zero-shot Transfer Learning

… However, they focused on distillation during pretraining and evaluate their model only on XNLI from XTREME benchmarks. Since they used common crawl data for pretraining and a different architecture, their models are not directly comparable to ours. 3 Method …


LIMSI@ MLIA

… In addition, we used the Cochrane bilingual parallel corpus [4]3 and the Taus Corona Crisis corpus.4. For out-of-domain corpus, we used the parallel corpora provided by the WMT14 campaign, which included Gigafr-en …


Lingua Custodia's participation at the WMT 2021 Machine Translation using Terminologies shared task

M Ailem, J Liu, R Qader - arXiv preprint arXiv:2111.02120, 2021

This paper describes Lingua Custodia's submission to the WMT21 shared task on machine translation using terminologies. We consider three directions, namely English to French, Russian, and Chinese. We rely on a Transformer-based …


Linking place records using multi-view encoders

V Cousseau, L Barbosa - Neural Computing and Applications, 2021

… For address embeddings, we utilized case-sensitive embeddings trained with the FastText model [4] on top of a Wikipedia and Common Crawl corpus [22]. The reasoning behind the different embedding sources for names and …


LIORI at SemEval-2021 Task 2: Span Prediction and Binary Classification approaches to Word-in-Context Disambiguation

A Davletov, N Arefyev, D Gordeev, A Rey - Proceedings of the Fifteenth Workshop on …, 2021

… tokens. XLM-R was trained on a cleaned two-terabyte CommonCrawl Corpus in 100 languages. A new promising approach to language task modelling is treating any natural language task as a question answering problem …


LISN@ WMT 2021

J Xu, SA Rauf, MQ Pham, F Yvon - 6th Conference on Statistical Machine Translation, 2021

This paper describes LISN's submissions to two shared tasks at WMT'21. For the biomedical translation task, we have developed resource-heavy systems for the English-French language pair, using both out-ofdomain and in-domain corpora. The …


D Loureiro, AM Jorge, J Camacho-Collados - arXiv preprint arXiv:2105.12449, 2021

Page 1. LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond Daniel Loureiroa,∗, Alıpio Mário Jorgea, Jose Camacho-Colladosb aLIAAD - INESC TEC, Dept. of Computer Science, FCUP …


LNN-EL: A Neuro-Symbolic Approach to Short-text Entity Linking

H Jiang, S Gurajada, Q Lu, S Neelam, L Popa, P Sen… - arXiv preprint arXiv …, 2021

… We also employ a suite of pretrained or custom trained neural language models to compute the similarity of mi and eij. Pre-trained Embedding Models. These include SpaCy's semantic similarity6 function that uses Glove …


Log Parsing and Template Extraction Using Neueral Sequence-To-Sequence Models

Abstract System log messages are produced by software systems as their code executes. The contents of these log messages vary across systems and are defined by the software developers of the system. Log messages can be captured and …


Looking for Semantic Similarity: What a Vector Space Model of Semantics Can Tell Us About Attention in Real-world Scenes

… ing the semantic vectors from Word2vec (Mikolov, Chen, Corrado, & Dean, 2013) and GloVe 1.2 (Pennington, Socher, & Manning, 2014) that learn how words are associated with each other from large text corpora (ie, Google …


R Wolfe, A Caliskan - arXiv preprint arXiv:2110.00672, 2021

Page 1. Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models Robert Wolfe University of Washington rwolfe3@uw.edu Aylin Caliskan University of Washington aylin@uw.edu Abstract We use …


LSI_UNED at CLEF eHealth2021: Exploring the effects of transfer learning in negation detection and entity recognition in clinical texts

H Fabregat, A Duque, L Araujo, J Martinez-Romo - CLEF eHealth, 2021

… Word embeddings: Two different pre-trained word embeddings from different sources are used for text representation. On the one hand, we use general domain Spanish 100dimensional word embeddings in …


M6: A Chinese Multimodal Pretrainer

J Lin, R Men, A Yang, C Zhou, M Ding, Y Zhang… - arXiv preprint arXiv …, 2021

… Xu et al. [45] released a 100GB corpus named CLUECorpus2020, which is re- trieved from the multilingual Common Crawl dataset … Forum discussion Plain-text - 8.7 39.0 223.1 - 18.0 Common Crawl Plain-text - 40.3 108.7 370.7 - 83.3 …


M6: Multi-Modality-to-Multi-Modality Multitask Mega-transformer for Unified Pretraining

J Lin, R Men, A Yang, C Zhou, Y Zhang, P Wang… - 2021

… Xu et al. [45] released a 100𝐺𝐵 corpus named CLUECorpus2020, which is retrieved from the multilingual Common Crawl dataset … Forum discussion Plain-text - 8.7 39.0 223.1 - 18.0 Common Crawl Plain-text - 40.3 108.7 370.7 - 83.3 …


Machine Habitus: Toward a Sociology of Algorithms


V Agarwal, A Manjunatha, JA Smith, AV Gribok… - 2021

Page 1. INL/EXT-21-61984 Revision 0 Industry Funding Opportunity Announcement Award Machine Learning and Economic Models to Enable Risk-Informed Condition Based Maintenance of a Nuclear Plant Asset March …


Machine Learning and Security Studies

Page 1. Machine Learning and Security Studies by Spencer Dorsey Department of Political Science Duke University Date: Approved: Scott de Marchi, Advisor Peter Feaver, Advisor Laia Balcells Erik Wibbels Dissertation submitted …


Machine learning for financial transaction classification across companies using character-level word embeddings of text fields

RK Jørgensen, C Igel - Intelligent Systems in Accounting, Finance and …

Abstract An important initial step in accounting is mapping financial transfers to the corresponding accounts. We devised machine-learning-based systems that automate this process. They use word em...


Machine Learning Meets Natural Language Processing--The story so far

NI Galanis, P Vafiadis, KG Mirzaev, GA Papakostas - arXiv preprint arXiv:2104.10213, 2021

… in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4938–4947, 2020. 42. “Common Crawl”, Accessed: Mar. 24, 2021.[Online]. Available: https://commoncrawl.org/ 43. “Kaggle”, Accessed: Mar. 24, 2021.[Online] …


Machine Learning with Applications

N Kanakaris, N Giarelis, I Siachos, N Karacapilidis

NK Corrêa, S Falk, S Fatimah, A Sen, N De Oliveira

This paper employs techniques and algorithms from the fields of natural language processing, graph representation learning and word embeddings to assist project managers in the task of personnel selection. To do so, our approach initially …


Machine Translation Customization via Automatic Training Data Selection from the Web

… 5 Fig. 1: Customization process of MT Systems Corpus Sent. (MM) News Commentary v13 0.3 Rapid (press releases) 1.3 Common Crawl 1.9 Europarl v7 2.4 ParaCrawl (Zipporah) 40.6 ParaCrawl (BiCleaner) 27.7 …


A Jones, DT Wijaya - arXiv preprint arXiv:2103.06369, 2021

Page 1. Majority Voting with Bidirectional Pre-translation Improves Bitext Retrieval Alex Jones Dartmouth College alexander.g.jones.23@dartmouth.edu Derry Tanti Wijaya Boston University wijaya@bu.edu Abstract Obtaining …


Making personnel selection smarter through word embeddings: A graph-based approach

N Kanakaris, N Giarelis, I Siachos, N Karacapilidis - Machine Learning with …, 2021

… GoogleNews using Word2Vec and Common Crawl using GloVe are some of the most favored public pre-trained word embeddings. For an in-depth analysis as far as the differences and use-cases of various pre-trained word embeddings are …


Maps Search Misspelling Detection Leveraging Domain-Augmented Contextual Representations

Y Li - arXiv preprint arXiv:2108.06842, 2021

… The second two models are trained with 42B Glove (Common Crawl) vectors, with embedding dimension 300, hidden state di- mension 50 and the same vocabulary. Our motivation of using a Page 4. KDD WIT '21, Aug …


Marked Attribute Bias in Natural Language Inference

… Otherwise, we apply the method to the 2Taken as the GloVe embeddings trained on the Common Crawl corpus for 840B tokens; available at https://nlp.stanford.edu/projects/glove/. Results were not found to vary …


MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding

J Li, Y Xu, L Cui, F Wei - arXiv preprint arXiv:2110.08518, 2021

… Equipped with the existing webpage datasets Common Crawl (CC)3, we pre-train MarkupLM with large-scale unlabeled HTML data and evaluate the pre-trained models on web-based structural reading comprehension and information extraction tasks. …


MasakhaNER: Named Entity Recognition for African Languages

DI Adelani, J Abbott, G Neubig, D D'souza, J Kreutzer… - arXiv preprint arXiv …, 2021

… XLMR XLMR (Conneau et al., 2020) was trained on 100 languages including Amharic, Hausa, and Swahili. The major differences be tween XLMR and mBERT are (1) XLMR was trained on Common Crawl while mBERT was …


Masked Language Model Entity Matching for Cultural Heritage Data

D Piché, A Zouaq, M Gagnon, L Font - 2021

… CamemBERT [10] is trained on the French portion of the OSCAR corpus, a pre-filtered version of Common Crawl, while FlauBERT [8] trains on a combination of French sets, including WMT19, OPUS, Wikimedia and Project Gutenberg. 3 Data overview …


Mastering Transformers: Build state-of-the-art models from scratch with advanced natural language processing techniques

Page 1. Mastering Transformers Build state-of-the-art models from scratch with advanced natural language processing techniques 0001) il -0 1 0 -0 -O 0 - -0 1 -0 - -0 1 1. -0 -0 -0 -0 -0 -0 1 1 0 010 1 0 1 0 1 01100110 0001101010 …


N Arabzadeh, A Bigdeli, S Seyedsalehi, M Zihayat…

Page 1. Matches Made in Heaven: Toolkit and Large-Scale Datasets for Supervised Query Reformulation Negar Arabzadeh University of Waterloo narabzad@uwaterloo. ca Amin Bigdeli Ryerson University abigdeli@ryerson.ca …


RK Jørgensen, M Hartmann, X Dai, D Elliott - arXiv preprint arXiv:2109.06605, 2021

Page 1. MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model Rasmus Kær Jørgensen1,2 and Mareike Hartmann3∗ and Xiang Dai1 and Desmond Elliott1 1University of Copenhagen, Denmark 2PricewaterhouseCoopers …


Measuring associational thinking through word embeddings

C Periñán-Pascual - Artificial Intelligence Review, 2021

The development of a model to quantify semantic similarity and relatedness between words has been the major focus of many studies in various fields, eg p.


Measuring Gender Bias in Word Embeddings for Russian Language

… The effect size is calculated as Cohen's d [6]: mean𝑥𝑥∈𝑋𝑋 𝑐𝑐(𝑥𝑥𝑎 𝐴𝐴𝑎 𝐵𝐵) - mean𝑦𝑦∈𝑌𝑌 𝑐𝑐(𝑦𝑦𝑎 𝐴𝐴𝑎 𝐵𝐵) std - dev𝑤𝑤∈𝑋𝑋∪𝑌𝑌 𝑐𝑐(𝑤𝑤𝑎 𝐴𝐴𝑎 𝐵𝐵) (4) 3https://tatianashavrina.github.io/taiga\_site …


Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset

HR Kirk, Y Jun, P Rauba, G Wachtel, R Li, X Bai… - arXiv preprint arXiv …, 2021

… F T-SNE Text Embeddings The meme-level embeddings are calculated by (i) extracting a 300-dimensional embedding for each word in the meme, using fastText embeddings trained on Wikipedia and Common Crawl; (ii) …


Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese

Z Zhang, H Zhang, K Chen, Y Guo, J Hua, Y Wang… - arXiv preprint arXiv …, 2021

Although pre-trained models (PLMs) have achieved remarkable improvements in a wide range of NLP tasks, they are expensive in terms of time and resources. This calls for the study of training more efficient models with less computation but still …


MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning

M Xia, G Zheng, S Mukherjee, M Shokouhi, G Neubig… - arXiv preprint arXiv …, 2021

… The Hausdorff distance between the source and target representations drops from 0.57 to 0.20 with F1 score improvement from 74.07 to 78.15. Wikipedia and XLM-R (Conneau et al., 2020) is pre-trained on 100 languages with CommonCrawl Corpora …


Metric Learning in Multilingual Sentence Similarity Measurement for Document Alignment

C Rajitha, L Piyarathne, D Sachintha, S Ranathunga - arXiv preprint arXiv …, 2021

… applied to the training data. XLM-R is trained with common crawl data from 100 languages, and has shown to outperform mBERT on multiple Natural Language Processing tasks. 3 Metric Learning Unsupervised metrics such as …


MFAQ: a Multilingual FAQ Dataset

M De Bruyn, E Lotfi, J Buhmann, W Daelemans - arXiv preprint arXiv:2109.12870, 2021

… provides an open repository of the web.5 Common Crawl's complete web archive consists of petabytes of data collected over 10 years of web crawling (Ortiz Suárez et al., 2020). The repository is organized in monthly bucket …


Mind the Gap: Assessing Temporal Generalization in Neural Language Models

A Lazaridou, A Kuncoro, E Gribovskaya, D Agrawal… - Advances in Neural …, 2021

Abstract Our world is open-ended, non-stationary, and constantly evolving; thus what we talk about and how we talk about it change over time. This inherent dynamic nature of language contrasts with the current static language modelling …


Minimum Bayes Risk Decoding with Neural Metrics of Translation Quality

M Freitag, D Grangier, Q Tan, B Liang - arXiv preprint arXiv:2111.09388, 2021

This work applies Minimum Bayes Risk (MBR) decoding to optimize diverse automated metrics of translation quality. Automatic metrics in machine translation have made tremendous progress recently. In particular, neural metrics, fine-tuned on …


Mining emotion-aware sequential rules at user-level from micro-blogs

MP Skenduli, M Biba, C Loglisci, M Ceci, D Malerba - Journal of Intelligent Information …, 2021

… the fine-grained emotion analysis task. Based on the obtained results, we decide to use multi-lingual word embeddings, pre-trained on Common Crawl and Wikipedia corpora using fastText. These models were trained using …


Misinformation Retrieval

… 34 4.1 Excerpt from a correct document from the Common Crawl Collection with fact used to validate tweets in blue. . . . . 43 viii Page 9 … for a given topic. The corpus used for both tasks was the common crawl corpus consisting of news articles online …


Mitigating the Position Bias of Transformer Models in Passage Re-Ranking

S Hofstätter, A Lipani, S Althammer, M Zlabinger… - arXiv preprint arXiv:2101.06980

… For the Transformer layers in TK we evaluate 2 layers each with 16 attention heads with size 32 and a feed-forward dimension of 100. For kernel-pooling we set the number 3 github.com/microsoft/BlingFire 4 42B CommonCrawl …


MKPM: Multi keyword-pair matching for natural language sentences

X Lu, Y Deng, T Sun, Y Gao, J Feng, X Sun, R Sutcliffe - Applied Intelligence, 2021

Sentence matching is widely used in various natural language tasks, such as natural language inference, paraphrase identification and question answering. F.


MLLP-VRAIN Spanish ASR Systems for the Albayzin-RTVE 2020 Speech-To-Text Challenge

… El Periódico [28] 2677 46637 291 Common Crawl [29] 1719 41792 486 Internal: parliamentary data 1361 35170 126 … [27] “Eldiario.es,” https://www.eldiario.es/. [28] “ElPeriodico.com,” https://www.elperiodico …


mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

R Ri, I Yamada, Y Tsuruoka - arXiv preprint arXiv:2110.08151, 2021

… 2019) trained with CommonCrawl data from 100 languages, and the parameters in common (eg, the weights of the transformer encoder and the word embeddings) are initialized using the checkpoint from the Transformers library.The size of the …


Mobile Testing Framework Exploiting Machine Learning and NLP

… the NLP step. Bert-Base, Multilingual Cased for BERT feature vector, GloVe Common Crawl 300 dimensional model for Glove, and lastly Fasttext 300 dimensional pre-trained model for Fasttext vector was used. On the android …


Y Guo, Z Zhang, J Jiang, W Wu, C Zhang, B Cui, J Li

Page 1. Noname manuscript No. (will be inserted by the editor) Model Averaging in Distributed Machine Learning: A Case Study with Apache Spark Yunyan Guo · Zhipeng Zhang · Jiawei Jiang · Wentao Wu · Ce Zhang · Bin …


Modeling Endorsement for Multi-Document Abstractive Summarization

L Lebanoff, B Wang, Z Feng, F Liu - arXiv preprint arXiv:2110.07844, 2021

A crucial difference between singleand multi-document summarization is how salient content manifests itself in the document(s). While such content may appear at the beginning of a single document, essential information is frequently reiterated in a …


Modeling Global and Local Interactions for Online Conversation Recommendation

X Zeng, J Li, L Wang, KF Wong - ACM Transactions on Information Systems (TOIS), 2021

… For the parameters with respect to local interactions, we initialize the word embedding layer with 200-dimensional Glove embedding [66], where the Twitter version is used for our Twitter dataset, and the Common Crawl version is applied on …


Modern-Day Shakespeare: Training Set Experiments with a Generative Pre-Trained Transformer-Best Paper

Page 1. 1 Modern-Day Shakespeare: Training Set Experiments with a Generative Pre-Trained Transformer Applied Project Final Report By Roksolana Maria Sheverack Spring 2021 A paper submitted in partial …


Monitoring COVID-19 pandemic through the lens of social media using natural language processing and machine learning

Y Liu, C Whitfield, T Zhang, A Hauser, T Reynolds… - Health Information Science …, 2021

… The custom NER model that we trained was based on spaCy's multi-task, OntoNotes-trained Convolutional Neural Network which uses GloVe vectors that were trained using Common Crawl [37] corpus. Topic modeling. Topic …


Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpus

D Trotta, R Guarasci, E Leonardelli, S Tonelli - arXiv preprint arXiv:2109.12053, 2021

… Each sentence is represented as a sequence of word embeddings, obtained with the Italian model of FastText (Grave et al., 2018) trained on Common Crawl and Wikipedia with size 300.4 The network is implemented with Keras …


Moral rights

… offer other services, including: CC Search, which makes openly licensed material easier to discover and use. As it makes use of the Common Crawl dataset, creators do not have to apply to be included (“About CC Search”) …


More Data and New Tools. Advances in Parsing the Index Thomisticus Treebank

F Gamba, M Passarotti, P Ruffolo - Proceedings http://ceur-ws. org ISSN, 2021

Abstract This paper investigates the recent advances in parsing the Index Thomisticus Treebank, which encompasses Medieval Latin texts by Thomas Aquinas. The research focuses on two types of variables. On the one hand, it examines the …


More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answering

Y Bai, DZ Wang - arXiv preprint arXiv:2109.12264, 2021

… Long-form QA ELI5[16] 2019 EN Web. Common Crawl Reddit Reddit Free-form ROUG-1,2,L MASH-QA[17] 2020 EN Health WebMD QA section QA section Span of words F1, EM NLQuAD[18] 2021 EN News BBC …


Morph Call: Probing Morphosyntactic Content of Multilingual Transformers

V Mikhailov, O Serikov, E Artemova - arXiv preprint arXiv:2104.12847, 2021

… languages. D-BERT (Sanh et al., 2019) or DistilBERT is a 6-layer distilled version of M-BERT model. XLM-R (Conneau et al., 2019) was pre-trained over filtered CommonCrawl data in 100 languages (Wenzek et al., 2019). MiniLM …


TD Bispo, FAO Santos, HT Macedo, C Zanchettin - Inteligencia Artificial, 2021

Page 1. Inteligencia Artificial, 24(67), 1-17 doi: 10.4114/intartif.vol24iss67pp1-17 INTELIGENCIA ARTIFICIAL http://journal.iberamia.org/ Morphological Skip-Gram: Replacing FastText characters n-gram with morphological knowledge …


Multi-granular Legal Topic Classification on Greek Legislation

C Papaloukas, I Chalkidis, K Athinaios, DA Pantazi… - arXiv preprint arXiv …, 2021

… learning rate. Based on this study, Conneau et al. (2020) proposed the XLMRoBERTa model, which supports 100 different languages (Greek included) and is trained on 2.5TB of filtered Common Crawl data. GREEK-BERT …


Multi-Granularity Contrasting for Cross-Lingual Pre-Training

… Following Conneau et al. (2019), we re- construct Common-Crawl Corpus to obtain monolingual training datasets while the bilingual data is obtained from the OPUS website 5. We also conduct up/down-sample for all pre-training data with a smoothing parameter …


Multi-level Embeddings for Processing Arabic Social Media Contents

L Moudjari, F Benamara, K Akli-Astouati - Computer Speech & Language, 2021


Multi-Level Sentiment Analysis of Product Reviews Based on Grammar Rules

HD Nguyen, T Le, K Tran, ST Luu, SN Hoang, HT Phan - New Trends in Intelligent …, 2021

… BERT is a transfer learning method which is trained on large language corpus such as Wikipedia and Common Crawl in different languages, then applied to specific tasks by fine tuning the model [7]. The appearance of BERT makes …


Multi-task Learning Using a Combination of Contextualised and Static Word Embeddings for Arabic Sarcasm Detection and Sentiment Analysis

AI Alharbi, M Lee - Proceedings of the Sixth Arabic Natural Language …, 2021

… al., 2017). The Ara2Vec word embeddings is a well-known open-source tool made up of six Arabic models. Data was obtained from Wikipedia, Twitter and Common Crawl (webpage crawl data) to train Ara2Vec. Al- though it …


MultiEmo: Multilingual, Multilevel, Multidomain Sentiment Analysis Corpus of Consumer Reviews

… The first model, Unsupervised Cross-lingual Representation Learning at Scale (XLM-RoBERTa), is a large multillingual language model, trained on 2.5TB of filtered CommonCrawl data, using self-supervised training techniques to achieve state-of-the-art performance …


MultiEURLEX--A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer

I Chalkidis, M Fergadiotis, I Androutsopoulos - arXiv preprint arXiv:2109.00904, 2021

… It is pretrained on Common Crawl with a vocabulary of 250k sub-words shared across languages … (2021) released a multilingual variant of T5 (Raffel et al., 2020), an encoder-decoder TRANSFORMER-based model …


Multilingual and Zero-Shot is Closing in on Monolingual Web Register Classification

S Rönnqvist, V Skantsi, M Oinonen, V Laippala - NoDaLiDa 2021, 2021

… The English CORE is based on unrestricted search queries of extremely frequent n-grams, while the other datasets are randomly sampled from the 2017 CoNLL Shared Task datasets, originally drawn from Common Crawl (Ginter et al., 2017) …


N De Cao, L Wu, K Popat, M Artetxe, N Goyal… - arXiv preprint arXiv …, 2021

Page 1. Multilingual Autoregressive Entity Linking Nicola De Cao1,2, Ledell Wu1, Kashyap Popat1, Mikel Artetxe1, Naman Goyal1, Mikhail Plekhanov1, Luke Zettlemoyer1,3, Nicola Cancedda1, Sebastian Riedel1,4, Fabio Petroni1 …


Multilingual Counter Narrative Type Classification

YL Chung, M Guerini, R Agerri - arXiv preprint arXiv:2109.13664, 2021

… The XLM-R language model (Conneau et al., 2020), pre-trained on CommonCrawl for 100 languages, reports strong performance on several cross-lingual downstream tasks such as natural language inference or …


Multilingual discussion summarisation in social networks

Page 1. Санкт–Петербургский государственный университет Тарасов Никита Андреевич Выпускная квалификационная работа Суммаризация мультиязычных дискуссий в социальных сетях Уровень образования: магистратура …


Multilingual ELMo and the Effects of Corpus Sampling

V Ravishankar, A Kutuzov, L Øvrelid, E Velldal - NoDaLiDa 2021, 2021

… 3.2 Sampling Our initial starting point for collecting the language model training corpora were the CoNLL 2017 Wikipedia/Common Crawl dumps released as part of the shared task on Universal Dependencies parsing …


Multilingual Epidemic Event Extraction

S Mutuvi, E Boros, A Doucet, G Lejeune, A Jatowt… - International Conference on …, 2021

Abstract In this paper, we focus on epidemic event extraction in multilingual and low-resource settings. The task of extracting epidemic events is defined as the detection of disease names and locations in a document. We experiment with a multilingual …


Multilingual Learning for Mild Cognitive Impairment Screening from a Clinical Speech Task

H Lindsay, P Müller, I Kröger, J Tröger, N Linz, A König… - 2021

… al., 2020a). As semantic vectors are learned from large amounts of text corpora (usually Wikipedia and OSCAR common crawl), embedding quality heavily depends on the quantity of the available training data. While French …


Page 1. UNIVERSITY OF HELSINKI FACULTY OF ARTS DEPARTMENT OF DIGITAL HUMANITIES Master's Thesis Multilingual Named Entity Recognition through Data and Model Transfer Saara Palma-Suominen 014291850 …


Multilingual Neural Semantic Parsing for Low-Resourced Languages

M Xia, E Monti - arXiv preprint arXiv:2106.03469, 2021

… It is trained for 100 languages on the CommonCrawl corpus, which is several orders of magnitude larger than the Wikipedia dump, especially for low-resourced languages. We use the public XLM-R Base model3 (12-layer, 768-hidden, 12 heads) in our experiment …


Multilingual Sentiment Analysis and Toxicity Detection for Text Messages in Russian

D Bogoradnikova, O Makhnytkina, A Matveev… - 2021 29th Conference of …, 2021

… 179M pa- rameters, trained on cased texts in the top 104 languages with the largest Wikipedias), and xlm-roberta (270M parameters with 12-layers, 768-hidden-state, 3072 feed-forward hiddenstate, 8-heads, trained on 2.5 …


GI Winata - arXiv preprint arXiv:2104.06268, 2021

Page 1. Multilingual Transfer Learning for Code-Switched Language and Speech Neural Modeling by Genta Indra Winata A Thesis Submitted to The Hong Kong University of Science and Technology in Partial Fulfillment of the Requirements for …


Multilingual Translation from Denoising Pre-Training

Y Tang, C Tran, X Li, PJ Chen, N Goyal, V Chaudhary…

… al., 2019; Brown et al., 2020). Monolingual data is far more prevalent for low resource languages, particularly in resources such as Wikipedia or Commoncrawl, a version of the web. Recent work has explored monolingual denoising …


Multilingual Zero-Shot and Few-Shot Causality Detection

… BPE vocabulary with 100k subwords. Moreover, regarding training data, they build a CommonCrawl corpus for 100 languages, which is notably larger than the Wikipedia corpus used for mBERT. For pretraining XLM-R, they …


Multimodal and Multilingual Embeddings for Large-Scale Speech Mining

PA Duquenne, H Gong, H Schwenk - Advances in Neural Information Processing …, 2021

… Using a similarity metric in that multimodal embedding space, we perform mining of audio in German, French, Spanish and English from Librivox against billions of sentences from Common Crawl. This yielded more than twenty thousand hours of …


Multimodal datasets: misogyny, pornography, and malignant stereotypes

A Birhane, VU Prabhu, E Kahembwe - arXiv preprint arXiv:2110.01963, 2021

… project to train a GDPR-Article(9)-Sensitive-URL classifier which revealed that, of the 1 Billion URLs they audited in the Common Crawl project, 155 million URLs fell into the sensitive category. The Realtoxicityprompts work …


Multimodal Emotion Recognition Using Transfer Learning on Audio and Text Data

JJ Deng, CHC Leung, Y Li - … Conference on Computational Science and Its …, 2021

… As for text representation, we adopt fine-tuned Text-To-Text Transfer Transformer (T5) [23] model trained by a common crawl (C4) dataset to extract text embeddings. Operations of transfer learning have greatly accelerated model training and applied in specific domains …


Multimodal MRI cerebral correlates of verbal fluency switching and its impairment in women with depression

L Domain, M Guillery, N Linz, A König, JM Batail… - NeuroImage: Clinical, 2021

Background The search of biomarkers in the field of depression requires easy implementable tests that are biologically rooted. Qualitative analysis of verbal fluency tests (VFT) are good candidates, but its cerebral correlates are unknown …


Multimodal or Text? Retrieval or BERT? Benchmarking Classifiers for the Shared Task on Hateful Memes

… for English pre-trained on Common Crawl (Grave et al., 2018) (MNN:base). 2 The second variant em- ploys three BERT models, each fine-tuned on one of our tasks (see subsection 3.2), from which we extracted the CLS tokens …


Z Huang, D Liang, P Xu, B Xiang - arXiv preprint arXiv:2109.12788, 2021

Page 1. Multiplicative Position-aware Transformer Models for Language Understanding Zhiheng Huang AWS AI zhiheng@amazon.com Davis Liang AWS AI liadavis@amazon.com Peng Xu AWS AI pengx@amazon.com …


Multitask Learning for Complaint Identification and Sentiment Analysis

A Singh, S Saha, M Hasanuzzaman, K Dey - Cognitive Computation, 2021

In today's competitive business world, customer service is often at the heart of businesses that can help strengthen their brands. Resolution of cust.


MuRIL: Multilingual Representations for Indian Languages

S Khanuja, D Bansal, S Mehtani, S Khosla, A Dey… - arXiv preprint arXiv …, 2021

… Monolingual Data: We collect monolingual data for the 17 languages mentioned above from the Common Crawl OSCAR corpus1 and Wikipedia2 … Wikipedia Common Crawl Wikipedia Common Crawl PMINDIA …


My House, My Rules: Learning Tidying Preferences with Graph Neural Networks

I Kapelyukh, E Johns - arXiv preprint arXiv:2111.03112, 2021

… We load word embeddings from a FastText model [20], pre-trained on the Common Crawl dataset of over 600 billion tokens. Names of objects are provided to NeatNet either by the simulator or by an object detection system if deployed …


Nahuatl Neural Machine Translation Using Attention Based Architectures: A Comparative Analysis for RNNs and Transformers as a Mobile Application Service

SK Bello García, E Sánchez Lucero, E Bonilla Huerta… - … International Conference on …, 2021

… Trained with 175 billions parameters and more than a trillion words from Common Crawl’s [27] and Wikipedia, making GPT-3 the largest language model at the time. The architecture is composed by 96 stacked encoder-decoder layers, 96 multi-head …


M Ehrmann, A Hamdi, EL Pontes, M Romanello… - arXiv preprint arXiv …, 2021

Page 1. Named Entity Recognition and Classification on Historical Documents: A Survey MAUD EHRMANN, Ecole Polytechnique Fédérale de Lausanne AHMED HAMDI, University of La Rochelle ELVYS LINHARES PONTES …


Named entity recognition and linking augmented with large-scale structured data

P Rychlikowski, A Lancucki, A Kaczmarek, B Najdecki… - Proceedings of the 8th …, 2021

… possible. This line of reasoning applied especially to word-level embeddings. Ideally we wanted our systems to have, for every language, em- beddings trained on Wikipedia, Common Crawl1, and a collection of news articles. We …


Named Entity Recognition and Normalization in Biomedical Literature: A Practical Case in SARS-CoV-2 Literature

Page 1. Universidad Politécnica de Madrid Escuela Técnica Superior de Ingenieros Informáticos Máster Universitario en Inteligencia Artificial Trabajo Fin de Máster Named Entity Recognition and Normalization in Biomedical Literature: A …


Naming unrelated words predicts creativity

JA Olson, J Nahas, D Chmoulevitch, SJ Cropper… - Proceedings of the National …, 2021

Skip to main content. Main menu. Home; Articles: Current; Special Feature Articles - Most Recent; Special Features; Colloquia; Collected Articles; PNAS Classics; List of Issues. Front Matter: Front Matter Portal; Journal Club. News: For …


Natural and artificial intelligence; natural and artificial language

… desinteligência, Inteligência, inteligência-do-cinema, inteligibilidade, inteligXancia, deligência, inteligencia, inteligência (0.85) 300 columns, obtained on 2 billion words, common crawl: fasttext, words: inteligencia (0.77), Inteligência, ainteligência …


Natural language inference for Malayalam language using language agnostic sentence representation

S Renjit, S Idicula - PeerJ Computer Science, 2021

… It is available in many languages, including Malayalam. fastText provided pretrained word vector representations trained on Common Crawl and Wikipedia data in 294 languages, including Malayalam, which we utilized to create sentence representations …


Negation typology and general representation models for cross-lingual zero-shot negation scope resolution in Russian, French, and Spanish.

A Shaitarova, F Rinaldi - Proceedings of the 2021 Conference of the North …, 2021

… We build on this work and compare mBERT with a new multilingual general purpose representation model, XLM-R. Unlike mBERT, XLM-R was pre-trained on more than two terabytes of filtered data collected by CommonCrawl …


Z Wang, L Wang, T Wu, T Li, G Wu - arXiv preprint arXiv:2109.04872, 2021

Page 1. Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding Zhenzhi Wang, Limin Wang*, Tao Wu, Tianhao Li, Gangshan Wu State Key Laboratory for Novel Software Technology, Nanjing University …


Network embeddings from distributional thesauri for improving static word representations

A Jana, S Haldar, P Goyal - Expert Systems with Applications, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Page 1. Neural approaches to discourse coherence: modeling, evaluation and application Youmna Farag Murray Edwards College This dissertation is submitted in September 2020 for the degree of Doctor of Philosophy Page 2. Page 3. Declaration …


Z Tan, S Wang, Z Yang, G Chen, X Huang, M Sun… - arXiv preprint arXiv …, 2020

Page 1. Neural Machine Translation: A Review of Methods, Resources, and Tools Zhixing Tana,c,d, Shuo Wanga,c,d, Zonghan Yanga,c,d, Gang Chena,c,d, Xuancheng Huanga,c,d, Maosong Suna,c,d,e, Yang Liua,b,c,d,e,∗ …


Page 1. University of Massachusetts Amherst ScholarWorks@UMass Amherst Doctoral Dissertations Dissertations and Theses April 2021 Neural Methods for Answer Passage Retrieval over Sparse Collections Daniel Cohen …


Neural Representation Learning onTables

Page 1. From Tables to Knowledge (KDD'21): Jay Pujara, Pedro Szekely, Huan Sun, Muhao Chen Huan Sun The Ohio State University From Tables to Knowledge: Recent Advances in Table Understanding (Part III) Neural …


L Kumar, S Sarkar - arXiv preprint arXiv:2107.08291, 2021

Page 1. Neural Search: Learning Query and Product Representations in Fashion E-commerce Sagnik Sarkar* sagnik.sarkar@myntra.com Myntra Designs Pvt. Ltd. India Lakshya Kumar* lakshya.kumar@myntra.com Myntra Designs Pvt. Ltd. India …


Page 1. Neural Sequential Modeling and Applications Guokun Lai CMU-LTI-21-010 Language Technology Institute School of Computer Science Carnegie Mellon University 5000 Forbes Ave., Pittsburgh, PA 15213 www.lti.cs.cmu.edu …


S Meftah, N Semmar, Y Tamaazousti, H Essafi, F Sadat - arXiv preprint arXiv …, 2021

Page 1. Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Units Sara Meftah∗, Nasredine Semmar∗, Youssef Tamaazousti∗, Hassane Essafi∗, Fatiha Sadat+ ∗CEA-List, Université Paris …


Neural Text Generation with Artificial Negative Examples to Address Repeating and Dropping Errors

K Shirai, K Hashimoto, A Eriguchi, T Ninomiya, S Mori - Journal of Natural Language …, 2021

… Both Japanese and English sentences were preprocessed as recommended in WAT'15.5 For the De-En task, we used the datasets provided by WMT'166 as well as all parallel corpora (Europarl v7 …


Neural Text Generation with Artificial Negative Examples

K Shirai, K Hashimoto, A Eriguchi, T Ninomiya, S Mori - arXiv preprint arXiv …, 2020

… For the De-En task, we used the datasets provided by WMT'166 and used all parallel corpora (Europarl v7, Common Crawl corpus, and News Commentary v11) for training, NewsTest2013 for development, and NewsTest2014 for testing …


Page 1. Neural Transfer Learning for Domain Adaptation in Natural Language Processing Apprentissage par Transfert Neuronal pour l'Adaptation aux Domaines en Traitement Automatique de la Langue Thèse de doctorat de l'université Paris-Saclay …


S Wankmüller - arXiv preprint arXiv:2102.02111, 2021

Page 1. Neural Transfer Learning with Transformers for Social Science Text Analysis Sandra Wankmüller Ludwig-Maximilians-University Munich sandra.wankmueller@ gsi.lmu.de Abstract. During the last years, there have been …


NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

S Min, J Boyd-Graber, C Alberti, D Chen, E Choi… - arXiv preprint arXiv …, 2021

… T5-XL+SSM T5 [13] is a text-to-text Transformer language model utilizing an encoder-decoder architecture that was pre-trained using a “span corruption” objective on a cleaned and de-duplicated subset of Common Crawl. Roberts et al …


Page 1. 171 JIOS, VOL. 45. NO. 1 (2021), PP. 171-221 JIOS, VOL. 45, NO. 1 (2021) SUBMITTED 08/20; ACCEPTED 11/20 New Synonyms Extraction Model Based on a Novel Terms Weighting Scheme Ahmad Hussein Ababneh ahmad.ababneh@aum.edu.jo …


News Headline Summarization

A Kulkarni, A Shivananda, A Kulkarni - Natural Language Processing Projects, 2022

… This transformer model is trained on the Colossal Clean Common Crawl (C4) data set. The architecture is integrated to devise text2text model. And hence T5 can also be used for many text-to-text-based applications. …


NICT-2 Translation System at WAT-2021: Applying a Pretrained Multilingual Encoder-Decoder Model to Low-resource Language Pairs

… model can learn multiple languages. The published pretrained mBART model2 consists of a 12-layer encoder and decoder with a model dimension of 1,024 on 16 heads. This model was trained on 25 languages in the Common Crawl corpus (Wenzek et al., 2019) …


NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performance

A Talman, M Apidianaki, S Chatzikyriakidis… - arXiv preprint arXiv …, 2021

… 4The contexts for R1 and R2 consist of sentences re- trieved from Wikipedia. In R3 the contexts are retrieved from Wikipedia, News (Common Crawl), fiction, The Children's Book Test (CBT), formal spoken text and …


NLM at TREC 2020 Health Misinformation and Deep Learning Tracks

Y Mrabet, M Sarrouti, AB Abacha, S Gayen, T Goodwin…

… Figure 1 presents an overview of our data pipeline, approaches and workflow. 1 Common Crawl News: https://github.com/commoncrawl/news-crawl 2 https://github.com/optimaize/ language-detector Page 2. 2 Authors Suppressed Due to Excessive Length Fig …


No News is Good News: A Critique of the One Billion Word Benchmark

H Ngo, JGM Araújo, J Hui, N Frosst - arXiv preprint arXiv:2110.12609, 2021

… We train models solely on Common Crawl web scrapes partitioned by year, and demonstrate that they perform worse on this task over time due to distributional shift. Analysis of this corpus reveals that it contains several examples of harmful text, as …


Node. js based Document Store for Web Crawling

… and internet archiving such as the Common Crawl Project. While the Internet Archive focuses on preservation, Common Crawl focuses on making web crawled data publicly accessible for scientific analysis and studies [3]. Common Crawl …


NormFormer: Improved Transformer Pretraining with Extra Normalization

S Shleifer, J Weston, M Ott - arXiv preprint arXiv:2110.09456, 2021

During pretraining, the Pre-LayerNorm transformer suffers from a gradient magnitude mismatch: gradients at early layers are much larger than at later layers. These issues can be alleviated by our proposed NormFormer architecture, which …


Not a Mirror, but an Engine: Digital Methods for Contextual Analysis of “Social Big Data”

… In SNA, for example, the set of social relations is always at least two-dimensional 16 and is usually represented in the form of a node table (listing all the nodes 14 There are, eg, open science projects like …


Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMax

E Kamalloo, M Rezagholizadeh, P Passban, A Ghodsi - arXiv preprint arXiv …, 2021

… The aim of this stage is to find interpretable unannotated augmented samples that are semantically close to training data. For this purpose, we use one of the sentence repositories from SentAugment (Du et al., 2021), comprising …


Novelty Detection in Textual Data using Document Encoders and Self Attention.

… two-layer neural network. We try to minimize the negative log-likelihood of the next word for an entire corpus. GloVe vectors are trained on the common crawl dataset, consisting of 840B tokens and 2.2M different tokens. We use glove vectors with a spacy …


NPVec1: Word Embeddings for Nepali-Construction and Evaluation

… Common Crawl3 is a non-profit organization which collects data through web crawling and makes it publicly available. 3.1.3 Nepali Wikipedia Corpus We obtained Nepali Wikipedia corpus from Kaggle (Gaurav, 2020) …


NUIG at TIAD 2021: Cross-lingual Word Embeddings for Translation Inference

… concepts WordNet [13] graph analysis and cross-lingual word embeddings monolingual corpora of Common Crawl and Wikipedia [8] graph analysis relying on paths, synonyms, similarities and cardinality in the translation graph …


NukeLM: Pre-Trained and Fine-Tuned Language Models for the Nuclear and Energy Domains

L Burke, K Pazdernik, D Fortin, B Wilson, R Goychayev… - arXiv preprint arXiv …, 2021

… which tokens to predict in each batch rather than deciding offline, before training; it uses much larger batch sizes; it uses byte-level tokenization instead of character-level; and finally, it considers much more pre-training data …


Obtaining Better Static Word Embeddings Using Contextual Embedding Models

P Gupta, M Jaggi - arXiv preprint arXiv:2106.04302, 2021

… We also evaluate some pre-trained 300 dimensional GLOVE (Pennington et al., 2014) and FASTTEXT (Bojanowski et al., 2016) models in Table 3. The GLOVE model was trained on Common-Crawl corpus of 840 Billion …


of black-box classifiers'

… The embeddings used came from the 840b token Common Crawl pre-trained 300 dimensional GloVe vectors, limited to the vocabulary present in the SST dataset4. Ultimately, this model achieves 84% accuracy on the bi- nary SST classification task …


Offensive Language Detection in Turkish Tweets with Bert Models

A Özberk, İ Çiçekli - 2021 6th International Conference on Computer …, 2021

… XLM-RoBERTa that is a transformer-based cross-lingual model is pre-trained using a multilingual masNed language model objective on 2.5 TB of CommonCrawl data in 100 languages [15]. Our offensive language detection models use BERT …


A Hande, K Puranik, K Yasaswini, R Priyadharshini… - arXiv preprint arXiv …, 2021

Page 1. Offensive Language Identification in Low-resourced Code-mixed Dravidian languages using Pseudo-labeling Adeep Handea,, Karthik Puranika,, Konthala Yasaswinia, Ruba Priyadharshinib, Sajeetha Thavareesanc …


Offensive, aggressive, and hate speech analysis: From data-centric to human-centered approach

J Kocoń, A Figas, M Gruza, D Puchalska… - Information Processing & …, 2021


Omission of Information: Identifying Political Slant via an Analysis of Co-occurring Entities

J Ehrhardt, T Spinde, A Vardasbi, F Hamborg - 2021

… 3 https://commoncrawl.org/ Page 6. Omission of Information: Identifying Political Slant via an Analysis of … 85 We scraped a total of 76.221 articles from nine different news outlets equally distributed over the three slant …


OmniNet: Omnidirectional Representations from Transformers

Y Tay, M Dehghani, V Aribandi, J Gupta, P Pham, Z Qin… - arXiv preprint arXiv …, 2021

… We run experiments on large-scale autoregressive (unidirectional) language modeling. We use two large-scale datasets, language modeling one billion (LM1B) (Chelba et al., 2013) and the Colossal Cleaned Common …


On a Utilitarian Approach to Privacy Preserving Text Generation

Z Xu, A Aggarwal, O Feyisetan, N Teissier - arXiv preprint arXiv:2104.11838, 2021

… Due to the high dimensional nature of textual tasks and very large vocabulary sizes (eg 2.2M words for GLOVE common crawl (Pennington et al., 2014)), this can lead to adding a lot of noise for achieving the desired privacy …


On Anytime Learning at Macroscale

L Caccia, J Xu, M Ott, MA Ranzato, L Denoyer - arXiv preprint arXiv:2106.09563, 2021

… For instance, every once in a while new data from common crawl is gathered to train even larger language models, or every few weeks new images and associated labels are collected to train better visual object recognizers …


On the dangers of stochastic parrots: Can language models be too big

EM Bender, T Gebru, A McMillan-Major, S Shmitchell - Proceedings of FAccT, 2021

… show that models like GPT-3 trained with at least 570GB of data derived mostly from Common Crawl16 can generate sentences with high toxicity … 16https://commoncrawl.org/the-data/ 17GPT-3's training data is not openly …


S Wang, Z Tu, Z Tan, S Shi, M Sun, Y Liu - arXiv preprint arXiv:2106.03297, 2021

Page 1. On the Language Coverage Bias for Neural Machine Translation Shuo Wang1 Zhaopeng Tu2 Zhixing Tan1 Shuming Shi2 Maosong Sun1,3 Yang Liu1,3,4 1Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, Tsinghua …


On the Learning Dynamics of Semi-Supervised Training for ASR}}

E Wallington, B Kershenbaum, O Klejch, P Bell - Proc. Interspeech 2021, 2021

… Following initial analysis, we use these same varying-quality models to investigate iSST. 2.1. Language Models The LM training data consists of the Tagalog CommonCrawl data-set1 and four smaller Tagalog sets provided …


On the Multilingual Capabilities of Very Large-Scale English Language Models

J Armengol-Estapé, OG Bonet, M Melero - arXiv preprint arXiv:2108.13349, 2021

… models (Nozza et al., 2020). Even the multilingual ones2 such as mBERT (Devlin et al., 2018) or XLM-R (Conneau et al., 2019) employ large multilingual datasets based on Wikipedia or CommonCrawl. A very recent work trained …


Page 1. UCLA UCLA Electronic Theses and Dissertations Title On the Robustness of Neural Network: Attacks and Defenses Permalink https://escholarship.org/uc/item/3k2780bg Author Cheng, Minhao Publication …


On the Usability of Transformers-based models for a French Question-Answering task

O Cattan, C Servan, S Rosset - Recent Advances in Natural Language Processing …, 2021

… Transformer-based language models such as BERT (Devlin et al., 2019) are pre-trained on largescale data collections sourced from Wikipedia or Common Crawl (CC) with one or multiple training objectives (masked …


On the validity of pre-trained transformers for natural language processing in the software engineering domain

J von der Mosel, A Trautsch, S Herbold - arXiv preprint arXiv:2109.04738, 2021

… Therefore, most of the available models are pre-trained by large companies on vast amounts of general domain data such as the entire English Wikipedia or the Common Crawl News dataset [5]. While these models achieve …


On-Line Retrieval of Health Information Based on Language Complexity, Information Customization and Information Quality

M Alfano - … and Communication Technologies for Ageing Well and …, 2021

… initiative. The Web Data Commons (WDC)[30] contain all Microformat, Microdata and RDFa (Resource Description Framework in Attributes) data extracted from the open repository of Web crawl data named Common Crawl (CC) …


One to rule them all: Towards Joint Indic Language Hate Speech Detection

M Bhatia, TS Bhotia, A Agarwal, P Ramesh, S Gupta… - arXiv preprint arXiv …, 2021

… A summary for each model is as follows: • XLM-RoBERTa: The pre-training of XLM-RoBERTa is based on 100 languages, using around 2.5TB of preprocessed CommonCrawl dataset to train cross-language …


Open Dataset Archive

… data: to name an example, while large corpora of tabular data from Web tables have been made available via CommonCrawl [5], the … Common Crawl The Common Crawl Foundation is a also a non-profit organization which …


Open-Domain Conversational Search Assistant with Transformers

R Ferreira, M Leite, D Semedo, J Magalhaes - arXiv preprint arXiv:2101.08197, 2021

… Page 6. 6 R. Ferreira et al. – Text-to-Text Transfer Transformer (T5) [26] is a text-to-text model based on the encoder-decoder Transformer architecture, pre-trained on the large C4 corpus, which was derived from Common …


Opening up Military Innovation: Causal Effects of 'Bottom-Up'Reforms to US Defense Research

ST Howell, J Rathje, J Van Reenen, J Wong - Available at SSRN 3825034, 2021

Page 1. Opening up Military Innovation: Causal Effects of 'Bottom-Up' Reforms to US Defense Research Sabrina T. Howell, Jason Rathje, John Van Reenen and Jun Wong ∗ April 8, 2021 Abstract When investing in research …


OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts

Y Meng, S Wang, Q Han, X Sun, F Wu, R Yan, J Li - arXiv preprint arXiv:2012.15015, 2020

… and the open-sourced OCR model suffices to fulfill our need. With text-free images in hand, we pair them with texts. Texts are randomly selected from the CommonCrawl English corpus, then added to the images. Texts in images …


Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model

PE Kummervold, J De la Rosa, F Wetjen, SA Brygfjeld - arXiv preprint arXiv …, 2021

… For resource-rich languages such as Chinese, En- glish, French, and Spanish, collections of texts from open sources such as Wikipedia (2021a), variations of Common Crawl data (2021), and other open-source corpora such as the BooksCorpus (Zhu et al., 2015) are …


Opinion Classification via Word and Emoji Embedding Models with LSTM

I Rabbimov, S Kobilov, I Mporas - International Conference on Speech and Computer, 2021

… Twitter. 1,661. 300. The UzbekFastText1 word vector is the fastText model trained with crawled texts published on Wikipedia using Common Crawl [36]. UzbekFastText2 was presented in [37] and it is a 300-dimensional Uzbek …


Out-of-vocabulary but not meaningless: Evidence for semantic-priming effects in pseudoword processing

… The model was trained on Common Crawl (around 630 billion words) and Wikipedia (around 9 billion words) using the Continuous Bag of … The model was trained on Common Crawl and Italian Wikipedia (around 11 billion words) using …


Overcoming Data Challenges in Machine Translation

Page 1. OVERCOMING DATA CHALLENGES IN MACHINE TRANSLATION by Huda Khayrallah A dissertation submitted to The Johns Hopkins University in conformity with the requirements for the degree of Doctor of Philosophy. Baltimore, Maryland July, 2021 …


Page 1. San Jose State University SJSU ScholarWorks Master's Projects Master's Theses and Graduate Research Spring 5-25-2021 Overlapping Community Detection in Social Networks Akshar Panchal Follow this and additional …


Overview of Touché 2021: Argument Retrieval

A Bondarenko, L Gienapp, M Fröbe, M Beloucif… - 2021

… Other argument retrieval systems such as ArgumenText [3] and TARGER [4] take advantage of the large web document collection Common Crawl, but their ability to reliably retrieve arguments to support sides in a decision process is limited …


PAGnol: An Extra-Large French Generative Model

J Launay, EL Tommasone, B Pannier, F Boniface… - arXiv preprint arXiv …, 2021

… Having been pre-trained on Common Crawl-based corpora, our models are certainly not immune from toxic content generation. Gehman et al. (2020) explored two main ways to filter toxic outputs, one being based on the pursuing the pretraining on less …


PALI at SemEval-2021 Task 2: Fine-Tune XLM-RoBERTa for Word in Context Disambiguation

S Xie, J Ma, H Yang, L Jiang, Y Mo, J Shen - arXiv preprint arXiv:2104.10375, 2021

… 2017). Especially, the XLM-RoBERTa (Conneau et al., 2020) model is a newly released large cross-lingual language model based on RoBERTa and is trained on 2.5TB filtered CommonCrawl data in 100 languages. Different …


Z Zeng, S Wang, N Xu, W Mao - Proceedings of the 44th International ACM SIGIR …, 2021

Page 1. PAN: Prototype-based Adaptive Network for Robust Cross-modal Retrieval Zhixiong Zeng, Shuai Wang, Nan Xu, Wenji Mao∗ SKL-MCCS, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China …


PanGu-$\alpha $: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

W Zeng, X Ren, T Su, H Wang, Y Liao, Z Wang, X Jiang… - arXiv preprint arXiv …, 2021

… To build Chinese corpus with comprehensive coverage, we collect a large amount of data from a wide range of resources, including Common Crawl, e-Books, encyclopedias, news, and so on … Common Crawl 176.2B 62.81 …


M Reid, M Artetxe - arXiv preprint arXiv:2108.01887, 2021

Page 1. PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining Machel Reid The University of Tokyo machelreid@weblab.tu-tokyo.ac.jp Mikel Artetxe Facebook AI Research artetxe@fb.com Abstract …


Parallel feature weight decay algorithms for fast development of machine translation models

Parallel feature weight decay algorithms, parfwd, are engineered for languageand task-adaptive instance selection to build distinct machine translation (.


Parallelizing Legendre Memory Unit Training

N Chilkuri, C Eliasmith - arXiv preprint arXiv:2102.11417, 2021

… After showing that the training of this simplified variant can be 1https:// commoncrawl.org … Therefore, we construct parameter efficient models that em- ploy just the DN layer, with d = 1 and θ = maxlen. We use 300D Glove …


Paraphrase Generation as Unsupervised Machine Translation

C Fan, Y Tian, Y Meng, N Peng, X Sun, F Wu, J Li - arXiv preprint arXiv:2109.02950, 2021

… LDA and K-means algorithms are performed on part of the the CommonCrawl corpus containing 10 billion English tokens … An overview of deriving the UMT-Multi model is shown in Figure 1. Up to now, UMT-Multi is purely based on unlabeled commoncrawl corpus …


Pathogens Are Linked to Human Moral Systems Across Time and Space

M Atari, NK Reimer, J Graham, J Hoover, B Kennedy… - 2021

Infectious diseases have been an impending threat to the survival of individuals and groups throughout our evolutionary history. As a result, humans have developed psychological pathogen-avoidance mechanisms and groups have developed …


Pegasus@ Dravidian-CodeMix-HASOC2021: Analyzing Social Media Content for Detection of Offensive Text

PK Jada, K Yasaswini, K Puranik, A Sampath… - arXiv preprint arXiv …, 2021

… One of the reason for the unparalleled performance is that it was trained on a mammoth 2.5 TB of CommonCrawl data [25]. It was trained with MLM loss as it’s objective on 100 different languages, and it shares similar training routine as the …


PENGEMBANGAN APLIKASI INFORMATION GATHERING MENGGUNAKAN METODE HYBRID SCAN BERBASIS GRAPHICAL USER INTERFACE

M Riasetiawan, A Wisnuaji, D Hariyadi, T Febrianto - Cyber Security dan Forensik …, 2021

… F-Secure Riddler, Cert Spotter, Hacker Target, Certificate Search, DN Dumpster, AlienFault, Rapid DNS, URL Scan, Threat Crowd, Wayback Machine, dan Common Crawl … q="+self.url, 1 2 "https://riddler.io/search …


Penn-Helsinki Parsed Corpus of Early Modern English: First Parsing Results and Analysis

S Kulick, N Ryant, B Santorini - arXiv preprint arXiv:2112.08532, 2021

We present the first parsing results on the Penn-Helsinki Parsed Corpus of Early Modern English (PPCEME), a 1.9 million word treebank that is an important resource for research in syntactic change. We describe key features of PPCEME that …


Personalized sentiment classification of customer reviews via an interactive attributes attention model

Y Zhang, J Wang, X Zhang - Knowledge-Based Systems, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Phishing Detection Methods: A Review

Abstract The web has turned into a principal part of our conventional social and financial activities. The web isn't significant for singular clients just yet additionally for associations, since associations that offer web-based exchanging can accomplish …


PhoMT: A High-Quality and Large-Scale Benchmark Dataset for Vietnamese-English Machine Translation

L Doan, LT Nguyen, NL Tran, T Hoang, DQ Nguyen - arXiv preprint arXiv:2110.12199, 2021

We introduce a high-quality and large-scale Vietnamese-English parallel dataset of 3.02M sentence pairs, which is 2.9M pairs larger than the benchmark Vietnamese-English machine translation corpus IWSLT15. We conduct experiments comparing strong …


Planning with Entity Chains for Abstractive Summarization

S Narayan, Y Zhao, J Maynez, G Simoes, R McDonald - arXiv preprint arXiv …, 2021

… Pre-training Datasets. Following Zhang et al. (2019), our model pretraining also relied on two large web corpus which were processed to look like plain text: (i) C4 (Raffel et al., 2019) is composed of 350M Web-pages …


Planning with Learned Entity Prompts for Abstractive Summarization

S Narayan, Y Zhao, J Maynez, G Simões, V Nikolaev… - Transactions of the …, 2021

We introduce a simple but flexible mechanism to learn an intermediate plan to ground the generation of abstractive summaries. Specifically, we prepend (or prompt) target summaries with entity chains—ordered sequences of entities mentioned in the …


Polish Web resources described in the" Polish World" directory (1997). Characteristics of domains and their conservation state

M Wilkowski - Archiwa-Kancelarie-Zbiory, 2020

… Sep. 2008, p. 6. https://hal-bnf.archives-ouvertes.fr/hal-01098538 (accessed 19.09.2020). 18 Common Crawl, https://commoncrawl.org/ (accessed 12.09.2020). 19 HTTP Archive, https://httparchive.org/ (accessed 12.09.2020). Page 8. 126 Marcin Wilkowski …


Political Attacks in 280 Characters or Less: A New Tool for the Automated Classification of Campaign Negativity on Social Media

V Petkevic, A Nai - American Politics Research, 2021

Negativity in election campaign matters. To what extent can the content of social media posts provide a reliable indicator of candidates' campaign negativity? We introduce and critically assess an automated classification procedure that we trained …


Political Polarization in Online News Consumption

K Garimella, T Smith, R Weiss, R West - arXiv preprint arXiv:2104.06481, 2021

… The Common Crawl project1 regularly crawls the Web and releases datasets of the discovered webpages on a regular schedule, which are generally considered the most extensive publicly available sources …


PoolRank: Max/Min Pooling-based Ranking Loss for Listwise Learning & Ranking Balance

Z Chen, C Eickhoff - arXiv preprint arXiv:2108.03586, 2021

… Our GloVe word embeddings are trained on the Common Crawl dataset 2. We use an Adam optimizer [13] with 0.0001 learning rate to train all models … Most queries 1https://www.tensor ow.org/ 2https://commoncrawl …


Postal address extraction from the web: a comprehensive survey

M Kayed, S Dakrory, AA Ali - Artificial Intelligence Review, 2021

The Web is a source of information for Location-Based Service (LBS) applications. These applications lack postal addresses for the user's Point of In.


Practical Wavelet Tree Construction

P Dinklage, J Ellert, J Fischer, F Kurpicz, M Löbel - Journal of Experimental …, 2021

… alphabet. We denote the resulting file by RuWB. Suffix Array of Common Crawl We construct the suffix array [35] of CommonCrawl, which is a permutation of the positive integers smaller than the length of CommonCrawl. We …


Pre-trained Language Models for Tagalog with Multi-source Data

S Jiang, Y Fu, X Lin, N Lin - CCF International Conference on Natural Language …, 2021

… [19] to process, filter and classify Common Crawl, which is a non-profit organization that produces and maintains an open, freely available repository of crawled data from the web. The filtering step used to create OSCAR involves …


Pre-trained Language Models in Biomedical Domain: A Survey from Multiscale Perspective

B Wang, Q Xie, J Pei, P Tiwari, Z Li - arXiv preprint arXiv:2110.05006, 2021

Pre-trained language models have been the de facto paradigm for most natural language processing (NLP) tasks. In the biomedical domain, which also benefits from NLP techniques, various pre-trained language models were proposed by …


Pre-trained Models and Evaluation Data for the Myanmar Language

S Jiang, X Huang, X Cai, N Lin - International Conference on Neural Information …, 2021

… On the one hand, we utilized all the Myanmar data from the OSCAR corpus 3 , a humongous multilingual corpus whose texts all come from the Common Crawl corpus 4 . In addition, articles on CC-100 [16] were also used as part of our corpus …


Pre-Trained Transformer-Based Language Models for Sundanese

… Data The dataset used during pre-training consists of Sundanese subsets of multilingual common-crawl corpora of OSCAR [18], CC-100 [19, 20], and C4 [21]. They are of the size 141KB, 15MB, and 464MB, respectively. Seeing …


Pre-trained Word Embeddings for Malayalam Language: A Review

KR Rahmath, PCR Raj, PC Rafeeque - … on Artificial Intelligence and Smart Systems …, 2021

… pre-trained on 17 In- dian languages, and their transliterated counterparts including Malayalam [8]. The corpora used for pre-training process are Wikipedia, Common Crawl, PMINDIA and … This dataset has been augmented …


Pre-training Methods in Information Retrieval

Y Fan, X Xie, Y Cai, J Chen, X Ma, X Li, R Zhang, J Guo… - arXiv preprint arXiv …, 2021

The core of information retrieval (IR) is to identify relevant information from large-scale resources and return it as a ranked list to respond to user's information need. Recently, the resurgence of deep learning has greatly advanced this field and leads …


Predicting cross-linguistic adjective order with information gain

W Dyer, R Futrell, Z Liu, G Scontras - arXiv preprint arXiv:2012.15263, 2020

… 4.1 Data Our study relies on two types of source data, both extracted from the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal De- pendencies (Ginter et al., 2017; Zeman et al., 2017) a set of …


Page 1. Predicting Depression and Suicide Ideation in the Canadian Population Using Social Media Data by Ruba Skaik Thesis submitted to the University of Ottawa in partial Fulfillment of the requirements for the Ph.D. degree in Computer Science …


Predicting leadership perception with large-scale natural language data

S Bhatia, CY Olivola, N Bhatia, A Ameen - The Leadership Quarterly, 2021


Predicting Next Dialogue Action in Emotionally Loaded Conversation

D Deksne, R Skadiņš - Proceedings of the Future Technologies Conference, 2021

… the Latvian Wikipedia data 2 and trained with our internal monolingual corpora, as well as several BERT models: multilingual BERT [26], BERT-LV [27], Language-agnostic BERT Sentence Embedding (LaBSE) [28], and the multilingual XLM-R model trained …


Predicting the Performance of Multilingual NLP Models

A Srinivasan, S Sitaram, T Ganu, S Dandapat, K Bali… - arXiv preprint arXiv …, 2021

… The pretraining procedure makes use of only unlabelled data (raw text), which is easy to obtain for a large number of languages from various sources including CommonCrawl1 and Wikipedia dumps2. These models can then be used to solve a …


Preprint from: https://www. gipp. com/pub

T Spinde, L Rudnitckaia, K Sinha, F Hamborg

… We scraped articles from both news outlets, published in the last decade, from 2010 to 2020, from Common Crawl [4]. For preprocessing, we use Genism simple preprocessing and generate n-grams. 3.4 Linear mapping between vector spaces …


PREPRINT–ARTICLE IN PRESS: Kennedy, B., Atari, M., Mostafazadeh Davani, A., Hoover, J., Omrani, A., Graham, J., & Dehghani, M.(in press). Moral Concerns are …

… of posts, GloVe word embedding vectors (Pennington et al., 2014), which were trained on text from the Common Crawl4, were mapped to the individual words in the posts and subsequently averaged element-wise (ie, across the dimensions of the embedding). The …


Prescriptions in NLP with Transformer Models

… is optimized BERT. XLMR is trained with a filtered version of CommonCrawl data with 100 languages and it uses a subword tokenization, SentencePiece [20] with a unigram language model [19]. mT5 Multilingual TexttoText …


Pretrained models and evaluation data for the Khmer language

S Jiang, S Fu, N Lin, Y Fu - Tsinghua Science and Technology, 2021

… On the one hand, we use all the central Khmer data from the OSCAR corpus , a large multilingual corpus whose texts come from the Common Crawl corpus . Although Common Crawl comprises scraped data from the Internet and covers a …


Prevalence in News Media of Two Competing Hypotheses about COVID-19 Origins

… 2. Methods. The textual content of news and opinion articles from the outlets listed in Figure 1 is available in the outlets online domains and/or public cache repositories such as Google cache, The Internet Wayback Machine …


Prevalence of Prejudice-Denoting Words in News Media Discourse: A Chronological Analysis

D Rozado, M Al-Gharbi, J Halberstadt - Social Science Computer Review, 2021

This work analyzes the prevalence of words denoting prejudice in 27 million news and opinion articles written between 1970 and 2019 and published in 47 of the most popular news media outlets in the...


Preventing author profiling through zero-shot multilingual back-translation

DI Adelani, M Zhang, X Shen, A Davody, T Kleinbauer… - arXiv preprint arXiv …, 2021

… The languages chosen are German (DE), Spanish (ES), French (FR), Japanese (JA), Russian (RU), and Chinese (ZH) based on the large amount of resources they have on OPUS (Tiedemann, 2012) and Common Crawl corpora 2 …


Price Determinants of Airbnb Apartments

Page 1. Antti Piispanen Price Determinants of Airbnb Apartments An Approach with Deep Language Representations Master of Science Thesis Faculty of Information Technology and Communication Sciences Examiner …


Prioritizing Original News on Facebook

X Ni, S Bu, I Markov - arXiv preprint arXiv:2102.08465, 2021

… There are other open-source crawlers that serve the same purpose. Common Crawl3 is a well-maintained open repository of Web crawl data that can be accessed and analyzed by anyone. We limit news articles in the graph to …


Privacy and AI Ethics–Understanding the convergences and tensions for the responsible development of machine learning

I Cofone, O Abimana, B Bonan, E Grand-Pierre, A Qarri

Context. The democratization of mobile systems and the development of information technologies have been accompanied by a massive increase in the amount and the diversity of data collected about individuals, often referred to as Big Data …


Privacy-Preserving Learning of Human Activity Predictors in Smart Environments

S Zehtabian, S Khodadadeh, L Bölöni, D Turgut - arXiv preprint arXiv:2101.06564

… However, many deep learning algorithms work best under big data regimes, where the number of data samples are counted from the tens of thousands (eg the MNIST dataset) to 500 billion (the Common Crawl dataset used to train the GPT-3 model) …


PrivaSeer: A Privacy Policy Search Engine

… The URLs in the Common Crawl archive were first filtered based on a selection criteria that took advantage of the fact that most privacy policy URLs either have the word 'privacy' or the words 'data' and 'protection' in them … 6 https://commoncrawl.org …


Probabilistic Text Entry—Case Study 3

K Vertanen - Intelligent Computing for Interactive System Design …, 2021

Page 1. 7Probabilistic Text Entry—Case Study 3 Keith Vertanen The reading and writing of text is an activity that has long dominated our interac tions with computers. Early user interfaces often displayed only text and relied on input obtained via a desktop keyboard …


Probing Multilingual Language Models for Discourse

M Kurfalı, R Östling - arXiv preprint arXiv:2106.04832, 2021

… the XLM-tlm. Instead, it is a RoBERTa model (Liu et al., 2019), which is an optimized version of BERT, trained on 2.5 TB of cleaned CommonCrawl data covering 100 languages (Conneau et al., 2020). There are two released …


MA Hedderich, B Roth, K Kann, B Plank, A Ratner… - arXiv preprint arXiv …, 2021

Page 1. WeaSuL 2021 Proceedings of the First Workshop on Weakly Supervised Learning (WeaSuL) May 7, 2021 Co-located with ICLR (Online) arXiv:2107.03690 v1 [cs.LG] 8 Jul 2021 Page 2. Introduction Welcome to WeaSuL …


Processing large-scale Internet topology data to model Autonomous System Networks

… such interrelated data. The number of Facebook users grew from 500 million in 2010 to 1.5 billion in 2014. The Common Crawl web corpora [29] covered for 2012 contains 3.5 billion web pages and 128 billion hyperlinks between …


JM Han, J Rute, Y Wu, EW Ayers, S Polu - arXiv preprint arXiv:2102.06203, 2021

Page 1. Proof Artifact Co-training for Theorem Proving with Language Models Jesse Michael Han 1 Jason Rute 2 Yuhuai Wu 3 Edward W. Ayers 4 Stanislas Polu 5 Abstract Labeled data for imitation learning of theorem proving …


Propaganda in Press: Challenges of Automatic Detection

D Pušelj, T Škalec - Text Analysis and Retrieval 2020 Course Project …

… trained on cased Wikipedia text. RoBERTa has 125M parameters with 12 layers, 768 hidden states, 3072 feed-forward hidden states and 8 heads and is trained on CommonCrawl data in 100 languages. The XLNet is a Large …


ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation

W Qi, Y Gong, Y Yan, C Xu, B Yao, B Zhou, B Cheng… - arXiv preprint arXiv …, 2021

… models with downstream task finetuning scripts, including ProphetNet-En pre-trained with 160GB English raw text, ProphetNet-Zh pre-trained with 160GB Chinese raw text, ProphetNet-Multi with 101GB Wiki-100 corpus and …


Proposed Use of a Conversational Agent for Patient Empowerment

M Alfano, J Kellett, B Lenzitti, M Helfert - 2021

… Depression 1 https://schema.org/ 2 https://health-lifesci.schema.org/ 3 http:// webdatacommons.org/ 4 http://commoncrawl.org o Dermatitis o Tuberculosis 4.2 Creation of Repository of Schema.org and Health-lifesci Structured Data …


Protecting Anonymous Speech: A Generative Adversarial Network Methodology for Removing Stylistic Indicators in Text

R Balakrishnan, S Sloan, A Aswani - arXiv preprint arXiv:2110.09495, 2021

With Internet users constantly leaving a trail of text, whether through blogs, emails, or social media posts, the ability to write and protest anonymously is being eroded because artificial intelligence, when given a sample of previous work, can match text …


Psychographic traits identification based on political ideology: An author analysis study on Spanish politicians' tweets posted in 2020

JA García-Díaz, R Colomo-Palacios, R Valencia-García - Future Generation …, 2021

… The pre-trained model of Spanish have been trained from Common Crawl and Wikipedia but adjusted with the PoliCorpus 2020 during training. SE assemble a single vector of dimension 300 for a sequence of the individual word embeddings …


Putting words into the system's mouth: A targeted attack on neural machine translation using monolingual data poisoning

J Wang, C Xu, F Guzman, A El-Kishky, Y Tang… - arXiv preprint arXiv …, 2021

… While this type of attack might appear unrealistic, the nature of the largest collections of monolingual data like Common Crawl (Buck et al., 2014; Wenzek et al., 2020; El-Kishky et al., 2020) (which contains blogs and other …


QMUL-SDS at EXIST: Leveraging Pre-trained Semantics and Lexical Features for Multilingual Sexism Detection in Social Networks

… RoBERTa model (XLM-R) [7] has extended the way of pre-training MLM by scaling the amount of data by two orders of magnitude (from Wikipedia to Common Crawl) and training … FastText + TextCNN: we use the Fasttext …


Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

I Caswell, J Kreutzer, L Wang, A Wahab, D van Esch… - arXiv preprint arXiv …, 2021

… CC: CommonCrawl; TED-6: da, cr, sl, sk, lt, et; TED-45: 45-language subset of (Qi … et al., 2019, 2020) is a set of monolingual corpora extracted from Common Crawl snapshots, specifically from the plain text WET format distributed by Common Crawl which removes …


QuALITY: Question Answering with Long Input Texts, Yes!

RY Pang, A Parrish, N Joshi, N Nangia, J Phang… - arXiv preprint arXiv …, 2021

To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current …


Quantifying context with and without statistical language models

… Others, such as the Common Crawl (Smith et al., 2013) are a hodge-podge grabbed from across the entire internet, which may include forum posts, blogs, tweets, or computer code, in addition to news articles, fan fiction, and so on …


Quantum-inspired Hierarchical Attention Mechanism for Question Answering

P Guo - 2021 IEEE 33rd International Conference on Tools with …, 2021

… We initialize word embeddings with 300-dimensional Glove word vectors pretrained from the 840B Common Crawl corpus. The LSTM hidden size is set to 100 over WikiQA and TrecQA. AdamOptimizer is used for updating parameters with …


F Bianchi, J Tagliabue, B Yu - arXiv preprint arXiv:2104.02061, 2021

Page 1. Query2Prod2Vec Grounded Word Embeddings for eCommerce Federico Bianchi Bocconi University Milano, Italy f.bianchi@unibocconi.it Jacopo Tagliabue∗ Coveo Labs New York, USA jtagliabue@coveo.com …


Question Classification for the Travel Domain using Deep Contextualized Word Embedding Models

C Weerakoon, S Ranathunga - 2021 Moratuwa Engineering Research Conference …, 2021

… XLNet [5] has pre-trained data over 129 GB collected from BooksCorpus, Wikipedia, CommonCrawl, Giga5, and ClueWeb 2012-B. XLNet at its core uses the permutation technique to predict tokens in random order contrary to masked model of BERT …


Re-entry Prediction for Online Conversations via Self-Supervised Learning

L Wang, X Zeng, H Hu, KF Wong, D Jiang - arXiv preprint arXiv:2109.02020, 2021

… For the parameters in the main model, we first initialize the embedding layer with 200-dimensional Glove embedding (Pennington et al., 2014), whose Twitter version is used for the Twitter dataset and Common Crawl version …


Re-Evaluating GermEval17 Using German Pre-Trained Language Models

M Aßenmacher, A Corvonato, C Heumann - arXiv preprint arXiv:2102.12330, 2021

… DBMDZ combine Wikipedia, EU Bookshop (Skadin š et al., 2014), Open Subtitles (Lison and Tiedemann, 2016), CommonCrawl (Ortiz Suárez et al., 2019), ParaCrawl (Espl`a-Gomis et al., 2019) and News Crawl (Haddow …


Read & Improve: A Novel Reading Tutoring System

R Watson, E Kochmar - EDM 2021: Educational Data Mining, 2021

… To source news content, R&I monitors both RSS Feeds from news websites and the publicly available Common Crawl News (CC-NEWS) Dataset.3 A fully automated Indexing Pipeline (RIIP, herein) processes article URLs …


Reading Key Figures from Annual Reports

Page 1. Reading Numbers from Annual Reports Using OCR and NLP to Extract Key Figures from Documents Master's thesis in Engineering Mathematics and Computational Science Sara Nordin Hällgren DEPARTMENT OF MATHEMATICAL SCIENCES …


Recent advances of low-resource neural machine translation

R Haque, CH Liu, A Way - Machine Translation, 2021

… As for the low-resource pairs, monolingual data was provided by the Common Crawl (Buck et al. ) and English monolingual data is blog … 12.Buck C, Heafield K, van Ooyen B (2014) N-gram counts and language models from the common crawl. In …


Recent Automatic Post Editing Research

H Moon, C Park, S Eo, J Seo, H Lim - Journal of Digital Convergence, 2021

… 진행되었다. 사전학습에는 100개 언어에 대한 Common Crawl[21] 단일 언어 말 뭉치가 이용된다 … 도출하였다. 사전학습에는 25개 언어에 대한 Common Crawl 단 일 언어데이터를 활용하였고, BART[24]에서 제안된 자 기 지도 학습 방법론을 차용하였다 …


Recent Research on Phishing Detection Through Machine Learning Algorithm

A Selamat, O Krejcar - … Conference on Industrial, Engineering and Other …, 2021

… Phishing URLs, websites or emails are generally collected from PhishTank, Phishing Corpus, OpenPhish, SpamAssassin, etc. [12, 23, 31], whereas legitimate ones are selected from Alexa, Common Crawl, Ham, etc. [16, 24, 28, 31, 35]. Table 4 …


Recognizing and Splitting Conditional Sentences for Automation of Business Processes Management

NPA Vo, I Manotas, O Popescu, A Cerniauskas… - arXiv preprint arXiv …, 2021

… length. 4.3 XLM-R-based Model XLM-RoBERTa (XLM-R) is a state of the art multilingual masked language model trained on 2.5 TB of newly created clean CommonCrawl data in 100 languages (Conneau et al., 2019). It obtains …


Reconstructing Implicit Knowledge with Language Models

M Becker, S Liang, A Frank - Proceedings of Deep Learning Inside Out (DeeLIO) …, 2021

… GPT-2 is pre-trained on web pages from CommonCrawl, XLNet on CommonCrawl+ ClueWeb (Callan et al., 2009), and BART on the CNN/DM summarization dataset (Hermann et al., 2015). 3.3 Fine-tuning LMs Task-adapted Datasets for LM Fine-tuning …


Reducing Human Labor Cost in Deep Learning for Natural Language Processing

… cause it can be trained in a completely unsupervised manner with huge amount of unlabeled data, which are extremely cheap to fetch from internet nowadays. For example, the wellknown “Common Crawl Project”2 is …


Reference-based Weak Supervision for Answer Sentence Selection using Web Data

V Krishnamurthy, T Vu, A Moschitti - arXiv preprint arXiv:2104.08943, 2021

… Work was conducted while the author was an intern at Amazon Alexa AI. more than 100MM Web documents from Common Crawl's data. Given a question-reference pair, the question is used as query to retrieve a set of relevant documents from the index …


Refined Commonsense Knowledge from Large-Scale Web Contents

TP Nguyen, S Razniewski, J Romero, G Weikum - arXiv preprint arXiv:2112.04596, 2021

Commonsense knowledge (CSK) about concepts and their properties is useful for AI applications. Prior works like ConceptNet, COMET and others compiled large CSK collections, but are restricted in their expressiveness to subject-predicate-object (SPO) …


Relational Header Discovery using Similarity Search in a Table Corpus

H Harmouch, T Papenbrock, F Naumann - 2021 IEEE 37th International Conference …, 2021

… Because web tables contain many distinct values (eg, there are „185M unique values in web tables extracted from the Web Common Crawl [13]), such indexes would be huge and, hence, easily exhaust available memory capacities …


Reliability Prediction for Health-related Content: A Replicability Study

M Fernández-Pichel, D Losada, JC Pichel, D Elsweiler - 2021

… [31] to generate the assessment pool, instead of using a fixed-depth pooling strategy. After the pool was formed, human assessors from 12 http://commoncrawl.org Page 9. Reliability Prediction for Health-related Content 9 Schwarz et al …


Representations of Language Varieties Are Reliable Given Corpus Similarity Measures

J Dunn - arXiv preprint arXiv:2104.01294, 2021

… The experiments draw on existing geographic corpora from the web (ultimately derived from the Common Crawl data) and from tweets (Dunn and Adams … discussed in Section 4. Data from tweets (TW) is shown on the …


Reproducibility Study: On Heavy-User Bias in A/B Testing

… they obtained whole Wikipedia to compare it against itself. For the without-analysis they took a 10%-sample of the Common Crawl, which is an open repository for web crawl data. This results in 4.2 million articles consisting of …


Research on Recent Quality Estimation

S Eo, C Park, H Moon, J Seo, H Lim - Journal of the Korea Convergence Society, 2021

… XLM-R에서는 CommonCrawl[30] 데이터 중 100개 언어에 해당하는 CC100을, mBART에서 25개 언어에 해당하는 CC25를 사전 학습에 활용하였다 … 100개의 언어를 포함하는 2TB 이상의 CommonCrawl 데이터에 …


Residual Energy-Based Models for Text

A Bakhtin, Y Deng, S Gross, M Ott, MA Ranzato… - Journal of Machine …, 2021

… (2015); Kiros et al. (2015), which consists of fiction books in 16 different genres, totaling about half a billion words. • CCNews: We collect a de-duplicated subset of the English portion of the CommonCrawl news …


Retaining Named Entities for Headline Generation

B Singh, A Marathe, AA Rizvi, AR Joshi - Inventive Computation and Information …

… These vectors are Page 246. 226 B. Singh et al. trained using word co-occurrence statistics on the Common Crawl corpus. The word embeddings of more than 94% of the words of the dataset's vocabulary are present in the glove. 840B. 300d …


RetroGAN: A Cyclic Post-Specialization System for Improving Out-of-Knowledge and Rare Word Representations

P Colon-Hernandez, Y Xin, H Lieberman, C Havasi…

… In our tests we use the English Common Crawl FastText with sub-word information (FT-CC) and Numberbatch 19.08 (NB) to see how performance would be affected by using embeddings that were already retrofitted with a large KB …


M Naito, S Yokoi, G Kim, H Shimodaira - arXiv preprint arXiv:2105.08585, 2021

Page 1. Revisiting Additive Compositionality: AND, OR and NOT Operations with Word Embeddings Masahiro Naito1,3 Sho Yokoi 2,3 Geewook Kim4 Hidetoshi Shimodaira 1,3 1Kyoto University 2Tohoku University 3RIKEN 4 Naver …


Revisiting Document Representations for Large-Scale Zero-Shot Learning

J Kil, WL Chao - arXiv preprint arXiv:2104.10355, 2021

… al., 2009) that has more than 20,000 classes, ex- isting ZSL algorithms (Frome et al., 2013; Norouzi et al., 2013) mostly resort to word vectors of classes names (Mikolov et al., 2013; Pennington et al., 2014) that are automatically …


Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer

I Turc, K Lee, J Eisenstein, MW Chang, K Toutanova - arXiv preprint arXiv …, 2021

… with impressive crosslingual transfer capabilities. mT5 Multilingual T5 (mT5) (Xue et al., 2020) is an encoder-decoder model that was jointly trained on 101 languages from Common Crawl. We use its mT5-Base variant, whose …


Revisiting Tri-training of Dependency Parsers

J Wagner, J Foster - arXiv preprint arXiv:2109.08122, 2021

… 3.4 Unlabelled Data To match the training data of the word embeddings (Section 3.5), we use the Wikipedia and Common Crawl data of the CoNLL 2017 Shared Task in UD Parsing (Ginter et al., 2017; Zeman et al., 2017) as unlabelled data in tri-training …


Rewarding Open-Domain Conversational Agents Using Memory Networks

… To collect such pairs of sentences, we use the Common Crawl Monolingual datasets [5]. Hence the final rescaled reward that we use for our experiments is … url: https://commoncrawl.org/. [6] Jacob Devlin et al. “Bert: Pre-training of deep …


E Perez, D Kiela, K Cho - arXiv preprint arXiv:2103.03872, 2021

Page 1. Rissanen Data Analysis: Examining Dataset Characteristics via Description Length Ethan Perez1 Douwe Kiela2 Kyunghyun Cho1 3 1New York University, 2Facebook AI Research 3CIFAR Fellow in Learning in Machines & Brains perez@nyu.edu Abstract …


Robustness of Machine Translation for Low-Resource Languages

Abstract It is becoming increasingly common for researchers and practitioners to rely on methods within the field of Neural Machine Translation (NMT) that require the use of an extensive amount of auxiliary data. This is especially true for low-resource …


Robustness Testing of Language Understanding in Dialog Systems

J Liu, R Takanobu, J Wen, D Wan, W Nie, H Li, C Li… - arXiv preprint arXiv …, 2020

… system. We directly use the released models in the repository as DeepSpeech2's configuration, where the speech model is trained on Baidu Internal English Dataset, and the language model is trained on CommonCrawl Data …


RoGPT2: Romanian GPT2 for Text Generation

MA Niculescu, S Ruseti, M Dascalu - 2021 IEEE 33rd International Conference on …, 2021

… First, OSCAR [21] has the highest percentage of the total training corpus, is a multi-language corpus, each language being extracted via language classification, filtered, and cleaned from Common Crawl. Second, the latest version of the Wikipedia Dumps for …


Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages

TI Dhamecha, R Murthy V, S Bharadwaj… - arXiv preprint arXiv …, 2021

… 4.1 Data To train the language models, we obtained text data from various sources including: Wikipedia Dump6, WMT Common Crawl 7, WMT News CommonCrawl8, Urdu Charles University Corpus (Bojar et al., 2014; Jawaid …


ROMANIAN SPEECH RECOGNITION EXPERIMENTS FROM THE ROBIN PROJECT

AM AVRAM, V PĂIȘ, DAN TUFIȘ - ISSN 1843-911X

… model could benefit in terms of accuracy. 4 http://corolaws. racai. ro/ corola_sound_search/index. php 5 http://www. racai. ro/p/reterom/ 6 https:// commoncrawl. org/ Page 116. ANDREI-MARIUS AVRAM, VASILE PĂIȘ, DAN TUFIȘ …


RuSentEval: Linguistic Source, Encoder Force!

V Mikhailov, E Taktasheva, E Sigdel, E Artemova - arXiv preprint arXiv:2103.00573, 2021

… XLM-R (Conneau et al., 2020a) is trained on 'dy- namic' MLM task, over filtered CommonCrawl data in 100 languages (Wenzek et al., 2020). MiniLM (Wang et al., 2020) is a distilled transformer of BERT architecture, but uses the XLMRoBERTa tokenizer …


S&P500 INDEX DIRECTION PREDICTION USING TEXTUAL TWEETS AND THEIR CORRESPONDING SENTIMENT

… The model structure is a standard sort of encoder-decoder transformer. T5 model structure is shown in Fig. 3. T5 uses common crawl web extracted text and the authors applied a simple heuristic filtering. T5 removes any …


S-NLP at SemEval-2021 Task 5: An Analysis of Dual Networks for Sequence Tagging

… We discard subword part, take only word vector part of a FastText model (pre-trained on Common Crawl dataset) for word representation and utilize an external English Byte Pair Embedding for out-of-vocabulary functionality …


SAFFRON: tranSfer leArning For Food-disease RelatiOn extractioN

G Cenikj, T Eftimov, BK Seljak - Proceedings of the 20th Workshop on Biomedical …, 2021

… Wikipedia, which are used for the pretraining of BERT, RoBERTa is trained on data from 3 additional sources: the CommonCrawl News dataset (Nagel, 2016), the OpenWebText corpus (Gokaslan and Cohen, 2019) and Stories …


G Ramesh, S Doddapaneni, A Bheemaraj… - arXiv preprint arXiv …, 2021

Page 1. Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages Gowtham Ramesh1∗ Sumanth Doddapaneni1∗ Aravinth Bheemaraj2,5 Mayank Jobanputra3 …


SAU'S Submission for CCMT 2021 Quality Estimation Task

… XLM-R is a multi-language pre-training model proposed by Facebook, which uses 2.5TB CommonCrawl to filter data, and masked language model pre-trained on text in 100 languages, which obtains state-of-the-art performance on cross-lingual …


SAUCE: Truncated Sparse Document Signature Bit-Vectors for Fast Web-Scale Corpus Expansion

M Wahed, D Gruhl, A Alba, AL Gentile, P Ristoski… - arXiv preprint arXiv …, 2021

… This experiment facilitates evaluation on a limited seed corpus scenario. The document collection is a snapshot (approximately 200 million documents) of the Common Crawl web archive1, which is regularly updated and consists …


Say What? Collaborative Pop Lyric Generation Using Multitask Transfer Learning

N Ram, T Gummadi, R Bhethanabotla, RJ Savery… - Proceedings of the 9th …, 2021

… The authors opted for an encoder-decoder transformer based design, trained on a variant of the Common Crawl corpus named the Colossal Common Crawl Corpus (or C4 for short). The T5 shows a remarkable ability to tackle many different tasks with …


Scalable Graph Convolutional Variational Autoencoders

D Unyi, B Gyires-Tóth - 2021 IEEE 15th International Symposium on Applied …, 2021

… In Reddit, nodes are Reddit posts, and if the same user commented on two posts, a link is drawn between them; node features are GloVe CommonCrawl word vectors [31] based on the average embedding of the …


SCALE EFFICIENTLY: INSIGHTS FROM PRE-TRAINING AND FINE-TUNING TRANSFORMERS

AT Vaswani, D Yogatama, D Metzler, HW Chung… - 2022

Y Tay, M Dehghani, J Rao, W Fedus, S Abnar… - arXiv preprint arXiv …, 2021

Abstract: There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost which have both …


Scaling Laws for Transfer

D Hernandez, J Kaplan, T Henighan, S McCandlish - arXiv preprint arXiv:2102.01293, 2021

… Pre-trained text models were trained on a mix of WebText2 described in [KMH+20], Common Crawl5 [RSR+20], English Wikipedia, and publicly available Internet Books … 5https://commoncrawl.org/the-data/ 6https://www.gharchive.org/ 6 Page 7. 3 Results …


Scarecrow: A Framework for Scrutinizing Machine Text

Y Dou, M Forbes, R Koncel-Kedziorski, NA Smith… - arXiv preprint arXiv …, 2021

… GPT-3 DaVinci (Brown et al., 2020) The 175B parameter variant of GPT-3, which is trained on a version of the Common Crawl web scrape with additional filtering and deduplicating. These model choices allow us to study several …


SCOPA: Soft Code-Switching and Pairwise Alignment for Zero-Shot Cross-lingual Transfer

D Lee, J Lee, G Lee, B Chun, S Hwang - Proceedings of the 30th ACM International …, 2021

… To enhance such transfer, XLM-R [4] is pre-trained on 100 languages with CommonCrawl Corpora, supervised by translational objectives, to transfer from resource-rich languages (eg, English and Chinese) to resource-poor languages …


Screening Gender Transfer in Neural Machine Translation

G Wisniewski, L Zhu, N Bailler, F Yvon - … of the Fourth BlackboxNLP Workshop on …, 2021

… It includes the Europarl, NewsCommentary and CommonCrawl corpora, and altogether contains 4,813,682 sentences and nearly 141 million French running words. All the corpora were tokenized and segmented into sub-lexical units using …


Screening hardware and volume factors in distributed machine learning algorithms on spark

JB Rodrigues, GC Vasconcelos, PRM Maciel - Computing, 2021

This paper presents an approach to investigate distributed machine learning workloads on Spark. The work analyzes hardware and volume data factors regardin.


Selecting Parallel In-domain Sentences for Neural Machine Translation Using Monolingual Texts

JPR Sharami, D Shterionov, P Spronck - arXiv preprint arXiv:2112.06096, 2021

Continuously-growing data volumes lead to larger generic models. Specific use-cases are usually left out, since generic models tend to perform poorly in domain-specific cases. Our work addresses this gap with a method for selecting in-domain data from …


Selecting the best data filtering method for NMT training

F Bane, A Zaretskaya - Proceedings of Machine Translation Summit XVIII …, 2021

… Rule Based Classifier for Translation Memory Cleaning. Online. Smith, JR, Saint-Amand, H., Plamada, M., Koehn, P., Callison-Burch, C., and Lopez, A. (2013). Dirt cheap web-scale parallel text from the Common Crawl. In …


Page 1. Self-Attention Network for Text Representation Learning by Tao SHEN Thesis submitted in fulfilment of the requirements for the degree of Doctor of Philosophy under the supervision of Dist. Prof. Chengqi ZHANG, Prof. Guodong LONG and Dr. Jing JIANG …


Self-Contextualized Attention for Abusive Language Identification

H Jarquín-Vásquez, HJ Escalante, M Montes - … of the Ninth International Workshop on …, 2021

… For word representation we used pre-trained fastText embeddings (Mikolov et al., 2018), trained with subword information on Common Crawl, which have been recognized as useful for this task ac- cording to the study presented in (Corazza et al., 2020) …


Self-Supervised Representation Learning: Introduction, Advances and Challenges

L Ericsson, H Gouk, CC Loy, TM Hospedales - arXiv preprint arXiv:2110.09327, 2021

Self-supervised representation learning methods aim to provide powerful deep feature learning without the requirement of large annotated datasets, thus alleviating the annotation bottleneck that is one of the main barriers to practical deployment of …


Self-Training Pre-Trained Language Models for Zero-and Few-Shot Multi-Dialectal Arabic Sequence Labeling

M Khalifa, M Abdul-Mageed, K Shaalan - arXiv preprint arXiv:2101.04758, 2021

… XLM-R also uses Common Crawl for training, which is more likely to have dialectal data than the Arabic Wikipedia (used in mBERT), making it more suited to our work … mBERT, XLM-R was trained on much larger data …


Self-training vs Pre-trained Embeddings for Automatic Essay Scoring

X Zhou, L Yang, X Fan, G Ren, Y Yang, H Lin - China Conference on Information …, 2021

… In the training process, we use both the pre-trained embeddings and the randomly initialized self-trained embeddings, where the pre-trained word vector comes from Glove Common Crawl (6B token) and the dimension is …


Self–other moral bias: Evidence from implicit measures and the Word-Embedding Association Test

MH Li, PW Li, LL Rao - Personality and Individual Differences, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Semantic Detection of Targeted Attacks Using DOC2VEC Embedding

… This model was trained using Common Crawl and Wikipedia. CBOW was used to train these models. In this study, we used dimension 300, character n-grams of length 5, a window of size 5, and 10 negatives. …


Semantic maps and metrics for science Semantic maps and metrics for science using deep transformer encoders

B Chambers, J Evans - arXiv preprint arXiv:2104.05928, 2021

Page 1. Semantic maps and metrics for science using deep transformer encoders Brendan Chambers QuillBot Knowledge Lab at the University of Chicago James Evans Department of Sociology and Knowledge Lab, University of Chicago Santa Fe Institute …


J Han, S Sarica, F Shi, J Luo - Journal of Mechanical Design, 2021

Page 1. Journal of Mechanical Design 1 Semantic Networks for Engineering Design: State of the Art and Future Directions Ji Han1 School of Engineering University of Liverpool, the United Kingdom e-mail: ji.han@liverpool.ac.uk …


Semantic Search of Mobile Applications Using Word Embeddings

J Coelho, A Neto, M Tavares, C Coutinho, R Ribeiro… - 10th Symposium on …, 2021

… This way, character-level embeddings are considered, instead of word-level representations. In preliminary tests, we compared a CBOW [12] fastText model (FT1), pre-trained on English Common Crawl considering word 5-grams …


Semantic Similarity Based Evaluation for Abstractive News Summarization

FB Fikri, K Oflazer, B Yanıkoglu - GEM 2021, 2021

… et al., 2019). mT5 Multilingual T5 (mT5)(Xue et al., 2020) is a variant of T5 model (Raffel et al., 2020) that was pre-trained for 101 languages including Turkish on a new Common Crawl-based dataset. For Turkish summarization …