Distractor Analysis and Selection for Multiple-Choice Cloze Questions for Second-Language Learners

… form pair. For features that require embedding words, we use the 300-dimensional GloVe word embedPage 4. dings (Pennington et al., 2014) pretrained on the 42 billion token Common Crawl corpus. The GloVe embeddings …


Distributed frameworks for approximate data analytics

… analytical queries can be expensive, eg, the Google book Ngrams dataset contains 2.2 TB of text data [17], and CommonCrawl corpus petabytes of web pages [18]. The above challenge is exacerbated when it is desirable to run different types of …


Distributed Training of Graph Convolutional Networks using Subgraph Approximation

A Angerd, K Balasubramanian, M Annavaram - arXiv preprint arXiv:2012.04930, 2020

… The vertex label is the community a post belongs to. The features consist of an embedding of post information, created using GloVe CommonCrawl (Pennington et al., 2014). The first 20 days of posts are used for training, while the rest are used for testing and validation …


Distributional and Lexical Exploration of Semantics of Derivational Morphology

… using Wikipedia datasets. The second one is presented in Grave et al. (2018), covering 157 languages including Turkish. Their models used Common Crawl and Wikipedia datasets and trained on fastText. The models were …


Distributional Models in the Task of Hypernym Discovery

V Yadrintsev, A Ryzhova, I Sochenkov - Russian Conference on Artificial Intelligence, 2020

… Most likely, the largest text corpus was used for the first model, which includes Wikipedia and Common Crawl (we do not know the exact volume of crawl-data for the Russian, but roughly 24 terabytes of plain text was used …


Distributional semantic modeling: a revised technique to train term/word vector space models applying the ontology-related approach

O Palagin, V Velychko, K Malakhov, O Shchurov - arXiv preprint arXiv:2003.03350, 2020

… Ac- cessed: 2020-03-03. [42] Firefly documentation. https://rorodata.github. io/firefly/. Accessed: 2020-03-03. [43] Common crawl. http://commoncrawl org/. Accessed: 2020-03-03. [44] Google dataset search. https://datasetsearch …


Do Neural Language Models Show Preferences for Syntactic Formalisms?

A Kulmizev, V Ravishankar, M Abdou, J Nivre - arXiv preprint arXiv:2004.14096, 2020

… Che et al. (2018). These models are trained on 20 million words randomly sampled from the concatenation of WikiDump and CommonCrawl datasets for 44 different languages, including our 13 languages. Each model features …


Page 1. Masaryk University Faculty of Informatics Document Representations for Fast and Accurate Retrieval of Mathematical Information Rigorous Thesis Vít Novotný Advisor: Doc. RNDr. Petr Sojka, Ph. D. Brno, Fall 2019 Signature …


A Khormali, J Park, H Alasmary, A Anwar, D Mohaisen - arXiv preprint arXiv …, 2020

Page 1. Domain Name System Security and Privacy: A Contemporary Survey Aminollah Khormali, Jeman Park, Hisham Alasmary, Afsah Anwar, David Mohaisen University of Central Florida Abstract The domain name system …


Domain Specific Complex Sentence (DCSC) Semantic Similarity Dataset

D Chandrasekaran, V Mago - arXiv preprint arXiv:2010.12637, 2020

… trained. While BERT was trained on Book Corpus and Wikipedia corpus, RoBERTa model was trained on four different corpora namely Book Corpus, Common Crawl News dataset, OpenWebText dataset and the Stories dataset …


Domain-Specific Meta-Embedding with Latent Semantic Structures

Q Liu, J Lu, G Zhang, T Shen, Z Zhang, H Huang - Information Sciences, 2020

… For example, GloVe is trained on aggregated global word-word co-occurrence statistics from a corpus of over 840B tokens and fastText [2] pre-trained word representations for 157 languages on Common Crawl and the Wikipedia Corpora …


ZC CHEN-KRAUS, C COYE10, M EMERY… - LANGUAGE of, 2020

Page 441. DOMINANCE STYLE AND VOCAL COMMUNICATION IN NON-HUMAN PRIMATES K KATIE SLOCOMBE* 1, EITHNE KAVANAGH11,, SALLY STREET2, FELIX O. ANGWELA3, THORE J. BERGMAN4, MARYJKA …


S Gururangan, A Marasović, S Swayamdipta, K Lo… - arXiv preprint arXiv …, 2020

Page 1. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks Suchin Gururangan† Ana Marasovic†♦ Swabha Swayamdipta†♦ Kyle Lo† Iz Beltagy† Doug Downey† Noah A. Smith†♦ †Allen Institute for Artificial …


DRIVING INTENT EXPANSION VIA ANOMALY DETECTION IN A MODULAR CONVERSATIONAL SYSTEM

NR Mallinar, TK HO - US Patent App. 16/180,613, 2020

… In various embodiments, the dataset builder 365 uses one or more of word2vec, the enwiki 2015 document vectorizer, ppdb paragram sentence embeddings, common-crawl uncased GloVe word embeddings, and enwiki …


Drug-Drug Interaction Classification Using Attention Based Neural Networks

D Zaikis, I Vlahavas - 11th Hellenic Conference on Artificial Intelligence, 2020

… word in a given sentence. The large English statistical model was used, which is trained with GloVe vectors on the OntoNotes 5 Common Crawl corpus and has a POS syntax accuracy of 97.22 percent. The unique POS tags …


Dual Conditional Cross Entropy Scores and LASER Similarity Scores for the WMT20 Parallel Corpus Filtering Shared Task

… sentences. The Pashto language model was trained on a concatenation of the CommonCrawl and Wikipedia corpora, with the CommonCrawl oversampled by a factor of 64 to produce a dataset of 9,273,763 sentences. The …


ZY Dou, A Anastasopoulos, G Neubig - arXiv preprint arXiv:2004.03672, 2020

Page 1. Dynamic Data Selection and Weighting for Iterative Back-Translation Zi-Yi Dou, Antonios Anastasopoulos, Graham Neubig Language Technologies Institute, Carnegie Mellon University {zdou, aanastas, gneubig}@cs.cmu.edu Abstract …


DynE: Dynamic Ensemble Decoding for Multi-Document Summarization

C Hokamp, DG Ghalandari, NT Pham, J Glover - arXiv preprint arXiv:2006.08748, 2020

… Ghalandari et al. (2020) presented the WCEP dataset, which is a large-scale collection of clusters of news articles with a corresponding summary, constructed using the Wikipedia Current Events Portal, with additional articles gathered from CommonCrawl …


H Zhang, M Zhao, C Wei, D Mantini, Z Li, Q Liu - arXiv preprint arXiv:2009.11662, 2020

Page 1. EEGdenoiseNet: A benchmark dataset for deep learning solutions of EEG denoising Haoming Zhang1,†, Mingqi Zhao1,2,†, Chen Wei1, Dante Mantini2,3, Zherui Li1, Quanying Liu1,∗ September 25, 2020 1 Department …


Effect of Character and Word Features in Bidirectional LSTM-CRF for NER

C Ronran, S Lee - 2020 IEEE International Conference on Big Data and …, 2020

… CRF), respectively, using public word embedding, character features and word features [6,4]. We explore existing word embeddings that is distinct from the previous studies including (1) Glove 300 embedding of 42B, 840B …


Efficient and High-Quality Neural Machine Translation with OpenNMT

G Klein, D Zhang, C Chouteau, JM Crego, J Senellart - Proceedings of the Fourth …, 2020

… Europarl v9 1,838,568 Common Crawl corpus 2,399,123 News Commentary v14 338,285 Wiki Titles v1 1,305,141 Document-split Rapid 1,531,261 ParaCrawl v3 31,358,551 Total 38,770,929 news-crawl 2007-2018 …


Efficient strategies for hierarchical text classification: External knowledge and auxiliary tasks

KR Rojas, G Bustamante, MAS Cabezudo, A Oncevay - arXiv preprint arXiv …, 2020

… of tokens in the input document. We use pre-trained word embeddings from Common Crawl (Grave et al., 2018) for the weights of this layer, and we do not fine-tune them during training time. Encoder: It is a bidirectional GRU …


Efficient Transfer Learning for Quality Estimation with Bottleneck Adapter Layer

H Yang, M Wang, N Xie, Y Qin, Y Deng - Proceedings of the 22nd Annual Conference …, 2020

… BPE is used for tokenizing, where 32000 tokens are reserved. We use UN corpus and Common Crawl parallel corpus with the size of 1https://github.com/pytorch/fairseq Page 5. Total Params Training Params …


Efficiently Reusing Old Models Across Languages via Transfer Learning

T Kocmi, O Bojar - Proceedings of the 22nd Annual Conference of the …, 2020

… EN - Russian 12.6M News Commentary, Yandex, and UN Corpus WMT 2012 WMT 2018 EN - French 34.3M Commoncrawl, Europarl, Giga FREN, News commentary, UN corpus WMT 2013 WMT dis. 2015 Table 2: Corpora used for each language pair …


Embedding Compression with Isotropic Iterative Quantization

S Liao, J Chen, Y Wang, Q Qiu, B Yuan - arXiv preprint arXiv:2001.05314, 2020

… We perform experiments with the GloVe embedding (Pennington et al. 2014) and the HDC embedding (Sun et al. 2015). The GloVe embedding is trained from 42B tokens of Common Crawl data. The HDC Table 1: Experiment …


Embedding Compression with Right Triangle Similarity Transformations

H Song, D Zou, L Hu, J Yuan - International Conference on Artificial Neural Networks, 2020

… model. 4.1 Experimental Setup. Pre-trained Continuous Embeddings. We conduct experiments on GloVe [16] and fasttext [1]. GloVe embeddings have been trained on 42B tokens of Common Crawl data with 400k words. Fasttext …


EmoDet2: Emotion Detection in English Textual Dialogue using BERT and BiLSTM Models

H Al-Omari, MA Abdullah, S Shaikh - 2020 11th International Conference on …, 2020

… Moreover, we have encoded the words in the conversation using Word2vec, Glove Wiki, and Glove Common Crawl packages … The hyperparameters as follow: Dropout = 0.4, the text in Word2Vec and Glove Wiki are lowered …


Emotion Aided Dialogue Act Classification for Task-Independent Conversations in a Multi-modal Framework

T Saha, D Gupta, S Saha, P Bhattacharyya - Cognitive Computation

… To extract textual features, a convolutional neural network (CNN) [48]–based approach is used. Pretrained GloVe [49] embeddings trained on the CommonCrawl corpus of dimension 300 have been used to represent words as word vectors …


Employing distributional semantics to organize task-focused vocabulary learning

HS Ponnusamy, D Meurers - arXiv preprint arXiv:2011.11115, 2020

… graph, we start with a distributional semantic vector representation of each word, which we obtain from the pre-trained model of GloVe (Pennington et al., 2014) based on the co-occurrence statistics of the the words form a large …


Page 1. Empowering Architects and Designers: A Classification of What Functions to Accelerate in Storage Chen Zou chenzou@uchicago.edu University of Chicago Andrew A. Chien achien@cs.uchicago.edu University of Chicago …


End to end approach for i2b2 2012 challenge based on Cross-lingual models

… Joshi et al., 2019). Unlike mBERT who has been trained on Wikipedia, XLM-RoBERT uses the CommonCrawl(Conneau et al., 2019a) corpus for its training. In this section we explain step by step our approach. First we adapt …


End-to-End Simultaneous Translation System for IWSLT2020 Using Modality Agnostic Meta-Learning

HJ Han, MA Zaidi, SR Indurthi, NK Lakumarapu, B Lee… - Proceedings of the 17th …, 2020

… We evaluate our system on the MuST-C Dev set. Our parallel corpus of WMT19 consists of Europarl v9, ParaCrawl v3, Common Crawl, News Commentary v14, Wiki Titles v1 and Documentsplit Rapid for the German-English language pair …


Energy-Based Models for Text

A Bakhtin, Y Deng, S Gross, M Ott, MA Ranzato… - arXiv preprint arXiv …, 2020

… (2015); Kiros et al. (2015), which consists of fiction books in 16 different genres, totaling about half a billion words. • CCNews: We collect a de-duplicated subset of the English portion of the CommonCrawl news …


English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too

J Phang, PM Htut, Y Pruksachatkun, H Liu, C Vania… - arXiv preprint arXiv …, 2020

… 5XLM-R Large (Conneau et al., 2019) is a 550m-parameter variant of the RoBERTa masked language model (Liu et al., 2019b) trained on a cleaned version of CommonCrawl on 100 languages. 6Excluded in this draft due to implementation issues. Page 5 …


Enhanced-RCNN: An Efficient Method for Learning Sentence Similarity

S Peng, H Cui, N Xie, S Li, J Zhang, X Li - Proceedings of The Web Conference 2020, 2020

… Base model. Model parameter size time (s/batch) BERT-Base 102.2M 0.23 ± 0.20 Enhanced-RCNN 7.7M 0.02 ± 0.01 from the 840B Common Crawl corpus [21]. We set the hidden size as 192 for all BiGRU layers. Ant Financial …


Enhancing Word Embeddings with Knowledge Extracted from Lexical Resources

M Biesialska, B Rafieian, MR Costa-jussà - arXiv preprint arXiv:2005.10048, 2020

… Dinu et al., 2015; Artetxe et al., 2017, 2018). Moreover, GloVe vectors for English were trained on Common Crawl (Pennington et al., 2014). Linguistic Constraints. To perform semantic specialization of word vector spaces, we …


Entity-Switched Datasets: An Approach to Auditing the In-Domain Robustness of Named Entity Recognition Models

O Agarwal, Y Yang, BC Wallace, A Nenkova - arXiv preprint arXiv:2004.04123, 2020

… al., 2019). For the first two, we used 300-d cased GloVe (Pennington et al., 2014) vectors trained on Common Crawl.7 For BERT, we use the public large8 uncased9 model and apply the default fine-tuning strategy. We use …


Entrepreneurial Organizations and the Use of Strategic Silence

W Shi, M Weber - Proceedings of the 54th Hawaii International …

… number of competing apps for a particular keyword), chart rankings (current ranking position for a keyword), difficulty (the popularity of apps including reviews and ratings) and traffic (eg, autosuggestion when typing in the store …


Establishing a New State-of-the-Art for French Named Entity Recognition

PJO Suárez, Y Dupont, B Muller, L Romary, B Sagot - LREC 2020-12th Language …, 2020

… They use zero to three of the following vector representations: FastText non-contextual embeddings (Bojanowski et al., 2017), the FrELMo contextual language model ob- tained by training the ELMo architecture on the OSCAR …


Estimating educational outcomes from students' short texts on social media

… We obtained significantly better results with a model that used word-embeddings (see Methods). We also find that embeddings trained on the VK corpus outperform models trained on the Wikipedia and Common Crawl corpora (Table 1). Page 6 …


Estimating Mutual Information Between Dense Word Embeddings

V Zhelezniak, A Savkov, N Hammerla - Proceedings of the 58th Annual Meeting of …, 2020

… Our focus here is on fastText vectors (Bojanowski et al., 2017) trained on Common Crawl (600B tokens), as previous literature suggests that among unsupervised vectors fastText yields the best performance for all tasks and …


Page 1. Estimating the influence of auxiliary tasks for multi-task learning of sequence tagging tasks Fynn Schröder Language Technology Group Universität Hamburg Hamburg, Germany fschroeder@informatik.uni-hamburg.de …


Evaluating cross-lingual textual similarity on dictionary alignment problem

Y Sever, G Ercan - Language Resources and Evaluation, 2020

… 2018) provides embeddings for 157 different languages trained on Wikipedia and Common Crawl Footnote 3 . For efficiency concerns, each embedding set of the 7 languages are pruned to the the most frequent \(500\times 10^3\) words in each language …


Evaluating German Transformer Language Models with Syntactic Agreement Tests

K Zaczynska, N Feldhus, R Schwarzenberg…

… 1 The first model which we refer to as GBERTlarge is a community model provided by the Bavarian State Library.2 It was trained on multiple German corpora including a recent Wikipedia dump, EU Bookshop corpus …


Evaluating Multilingual BERT for Estonian

C Kittask, K Milintsevich, K Sirts - arXiv preprint arXiv:2010.00454, 2020

… 1Corresponding Author: Claudia Kittask; E-mail: claudiakittask@gmail.com Page 2. and cross-lingual RoBERTa (XLM-RoBERTa) [3], which was trained on much larger CommonCrawl corpora and also includes 100 languages …


Evaluating Sentence Representations for Biomedical Text: Methods and Experimental Results

NS Tawfik, MR Spruit - Journal of Biomedical Informatics, 2020

… 3.2. Embedding Methods. GloVe We use the pre-trained embeddings consisting of 2.2 million vocabulary words available at https://nlp.stanford.edu/projects/glove/ which were trained on the Common Crawl (840B tokens) dataset …


Evaluating Word Embeddings on Low-Resource Languages

N Stringham, M Izbicki - Proceedings of the First Workshop on Evaluation and …, 2020

… Grave et al. (2018) trained FastText embeddings on 157 languages using data from the Common Crawl project. But they were only able to explicitly evaluate 10 of these language models using the analogy task due to the …


Evaluation of related news recommendations using document similarity methods

M Pranjic, V Podpecan, M Robnik-Šikonja, S Pollak

… RoBERTa (Liu et al., 2019). It uses the sentence piece tokenizer and it is trained with the masked language model objective (MLM) on the CommonCrawl data in 100 languages, including Croatian. Similar to the mBERT, all …


Event Detection on Literature by Utilizing Word Embedding

J Chun, C Kim - International Conference on Database Systems for …, 2020

… On the contrary, Neural-based methods have the limitation that they ignore semantic relationships in a text. 2.3 Facebook Pre-trained Word Vectors. We chose pre-trained word vectors published by Facebook, trained on …


Evidence Integration for Multi-hop Reading Comprehension with Graph Neural Networks

L Song, Z Wang, M Yu, Y Zhang, R Florian, D Gildea - IEEE Transactions on …, 2020

Page 1. 1041-4347 (c) 2020 IEEE. Personal use is permitted, but republication/ redistribution requires IEEE permission. See http://www.ieee.org/ publications_standards/publications/rights/index.html for more information. This …


Examining the rhetorical capacities of neural language models

Z Zhu, C Pan, M Abdalla, F Rudzicz - arXiv preprint arXiv:2010.00153, 2020

… Non-contextualized word embeddings We consider two popular word embeddings here: • GloVe (Pennington et al., 2014) contains 2.2M vocabulary items and produces 300dimensional word vectors. The GloVe embedding …


Experience Grounds Language

Y Bisk, A Holtzman, J Thomason, J Andreas, Y Bengio… - arXiv preprint arXiv …, 2020

… (2013) trained on 1.6 billion tokens, while Pennington et al. (2014) scaled up to 840 billion tokens from Common Crawl. Recent approaches have made progress by substantially increasing the number of model parameters …


Experiencers, Stimuli, or Targets: Which Semantic Roles Enable Machine Learning to Infer the Emotions?

L Oberländer, K Reich, R Klinger - arXiv preprint arXiv:2011.01599, 2020

… 1For ET, 90% of the annotated experiencers are the authors of the tweets without corresponding span annotation. 2We use 42B tokens, pretrained on CommonCrawl (Pennington et al., 2014), https://nlp.stanford.edu …


Experiments on Paraphrase Identification Using Quora Question Pairs Dataset

A Chandra, R Stefanus - arXiv preprint arXiv:2006.02648, 2020

… matching result into a fix-length matching vector and continued to last layer of the model which is a fully connected layer. The paper use GloVe as a pretrained word vector from 840B Common Crawl corpus and apply it to Quora …


Explicit Alignment Objectives for Multilingual Bidirectional Encoders

J Hu, M Johnson, O Firat, A Siddhant, G Neubig - arXiv preprint arXiv:2010.07972, 2020

… Sentence Alignment Our first proposed objective encourages cross-lingual alignment of sentence 1 AMBER is trained on 26GB parallel data and 80GB monolingual Wikipedia data, while XLM-R-large is trained on 2.5TB …


Explicit Sentence Compression for Neural Machine Translation

Z Li, R Wang, K Chen, M Utiyama, E Sumita, Z Zhang… - arXiv preprint arXiv …, 2019

… for NMT evaluation. For the EN-DE translation task, 4.43 M bilingual sentence pairs from the WMT14 dataset were used as training data, including Common Crawl, News Commentary, and Europarl v7. The newstest2013 and …


Exploiting Categorization of Online News for Profiling City Areas

A Bondielli, P Ducange, F Marcelloni - 2020 IEEE Conference on Evolving and …, 2020

… FastText is created by Facebook and is based on Neural Networks. Pre-trained word vectors are available for 157 languages, trained on Common Crawl and Wikipedia. More specifically, we chose to use the pre-trained Italian model …


Exploiting Class Labels to Boost Performance on Embedding-based Text Classification

A Zubiaga - arXiv preprint arXiv:2006.02104, 2020

… 4.2 Word Embedding Models & Classifiers We tested four word embedding models: (1) Google's Word2Vec model (gw2v), (2) a Twitter Word2Vec model5 (tw2v) [10], (3) GloVe embeddings trained from Common Crawl (cglove) …


Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning

T Shen, Y Mao, P He, G Long, A Trischler, W Chen - arXiv preprint arXiv:2004.14224, 2020

Page 1. EXPLOITING STRUCTURED KNOWLEDGE IN TEXT VIA GRAPH-GUIDED REPRESENTATION LEARNING Tao Shen∗ University of Technology Sydney tao.shen@student.uts.edu.au Yi Mao, Pengcheng …


Exploring Different Methods for Solving Analogies with Portuguese Word Embeddings

T Sousa, H Gonçalo Oliveira, A Alves - 9th Symposium on Languages, Applications …, 2020

… from several sources. Such sources include raw text (ie, an ensemble of Google News word2vec, Common Crawl GloVe, Open Subtitles fastText) combined with the ConceptNet semantic network with retrofitting. 1 https://github …


Page 1. Exploring Event Extraction Across Languages Thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Computational Linguistics by Research by Suhan Prabhu 201525118 suhan.prabhuk@research.iiit.ac.in …


Exploring Neural Network Approaches in Automatic Personality Recognition of Filipino Twitter Users

… FastText differs by learning character-grams, as opposed to word-grams. We utilize the embeddings trained on Common Crawl and Wikipedia data2 – specifically, the embeddings for English and Tagalog (both 300 dimensions) …


Exploring Swedish & English fastText Embeddings with the Transformer

TP Adewumi, F Liwicki, M Liwicki - arXiv preprint arXiv:2007.16007, 2020

… We obtain better performance in both languages on the downstream task with far smaller training data, compared to recently released, common crawl versions and character n-grams appear useful for Swedish, a morphologically rich language …


Exploring the Dominance of the English Language on the Websites of EU Countries

A Giannakoulopoulos, M Pergantis, N Konstantinou… - Future Internet, 2020

… For this purpose, we used information obtained from Common Crawl, a “repository of web crawl data that is universally accessible and analyzable” [34]. Among the data Common Crawl offers is an index of every available webpage …


Page 1. Extended Overview of CLEF HIPE 2020: Named Entity Processing on Historical Newspapers Maud Ehrmann1[0000−0001−9900−2193], Matteo Romanello1[0000−0002− 1890−2577], Alex Flückiger2, and Simon Clematide2[0000−0003−1365−0662] …


Extended study on using pretrained language models and YiSi-1 for machine translation evaluation

C Lo - Proceedings of the Fifth Conference on Machine …, 2020

… The differencesbetweenXLM-RandBERTare1)XLM-Ris trained on the CommonCrawl corpus which is significantly larger than the Wikipedia training data used by BERT; 2) instead of a uniform data sampling rate used in BERT …


Extending Tables using a Web Table Corpus

… Lehmberg et al. [12] gathered a web table corpus containing 233 million tables from the 2015 version of the CommonCrawl2, using a table extraction method similar to that of [11]. The table corpus … and 2https://commoncrawl.org/about …


Extracting Family History of Patients From Clinical Narratives: Exploring an End-to-End Solution With Deep Learning Models

X Yang, H Zhang, X He, J Bian, Y Wu - JMIR Medical Informatics, 2020

… We screened 4 different word embeddings following a similar procedure reported in our previous study [46] and found that the Common Crawl embeddings—released by Facebook and trained using the fastText on the …


Extracting Training Data from Large Language Models

N Carlini, F Tramer, E Wallace, M Jagielski… - arXiv preprint arXiv …, 2020

… In particular, we select samples from a subset of Common Crawl6 to feed as context to the model.7 6http://commoncrawl.org/ 7It is possible there is some intersection between these two datasets, effectively allowing this strategy to “cheat” …


I Chung, B Kim, Y Choi, SJ Kwon, Y Jeon, B Park… - arXiv preprint arXiv …, 2020

Page 1. arXiv:2009.07453v1 [cs.LG] 16 Sep 2020 Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation Insoo Chung∗ Byeongwook Kim∗ Yoonjung Choi Se Jung Kwon Yongkweon …


Facebook AI's WMT20 News Translation Task Submission

PJ Chen, A Lee, C Wang, N Goyal, A Fan… - arXiv preprint arXiv …, 2020

… we use all the available monolingual data, eg NewsCrawl + CommonCrawl + Wikipedia dumps for Tamil, and CommonCrawl for Inuktitut … unconstrained track, we use Tamil monolingual data and Tamil-English mined bitext data …


Factors affecting sentence similarity and paraphrasing identification

M Alian, A Awajan - International Journal of Speech Technology, 2020

… Grave et al. (2018) contributed in a pre-trained word vector representation for 157 languages including Arabic. The word vectors have been trained on Wikipedia and the Common Crawl corpus using an extension of the FastText model with subword information …


Fairness in AI-based Recruitment and Career Pathway Optimization

Page 1. FAIRNESS IN AI-BASED RECRUITMENT AND CAREER PATHWAY OPTIMIZATION By Dena Freshta Mujtaba A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of …


Fake News Data Collection and Classification: Iterative Query Selection for Opaque Search Engines with Pseudo Relevance Feedback

A Elyashar, M Reuben, R Puzis - arXiv preprint arXiv:2012.12498, 2020

Page 1. FAKE NEWS DATA COLLECTION AND CLASSIFICATION: ITERATIVE QUERY SELECTION FOR OPAQUE SEARCH ENGINES WITH PSEUDO RELEVANCE FEEDBACK APREPRINT Aviad Elyashar, Maor Reuben …


Fake News Detection

IP Marín, D Arroyo - Conference on Complex, Intelligent, and Software …, 2020

… 4. https://spacy.io/ [Last accessed 26 Jan 2020]. 5. en_core_web_lg, pre-trained English statistical models. English multi-task CNN trained on OntoNotes, with GloVe vectors trained on Common Crawl. 6. https …


Fake news spreader detection using neural tweet aggregation

… As a preprocessing step we lowercased tweets and removed stop-words and punctuation. We did not use any special preprocessing. For the word embeddings we used fastText [1] trained on Common Crawl and Wikipedia with dimension set to 100 …


Fake News Spreader Identification in Twitter using Ensemble Modeling

A Hashemi, MR Zarei, MR Moosavi, M Taheri

… Page 4. The sources used in the English model for training data are OntoNotes 51 and GloVe Common Crawl2 and the Spanish model utilizes UD Spanish AnCora v2.53, WikiNER4, OSCAR (Common Crawl)5 and Wikipedia …


Fashion-IQ 2020 Challenge 2nd Place Team's Solution

M Shin, Y Cho, S Hong - arXiv preprint arXiv:2007.06404, 2020

… For training the LSTM and the GRU from scratch, we initialize the word embedding with the concatenation of three GloVe vectors2 learned from Wikipedia, Twitter, and Common Crawl that results in 900-dimensional input …


Fast entity linking in noisy text environments

SM Shah, MD Conover, PN Skomoroch, MT Hayes… - US Patent 10,733,383, 2020

… entry). In some embodiments, the candidate dictionary is determined by using the hyperlinks on a Wikipedia page, or a Common Crawl page to identify surface forms (the hyperlink anchor text) that point to a specific page. Each …


Fast Indexes for Gapped Pattern Matching

M Cáceres, SJ Puglisi, B Zhukova - International Conference on Current Trends in …, 2020

… Open image in new window Fig. 2. Fig. 2. Time to search a 2GiB subset of the Common Crawl web collection (commoncrawl.org). for 20 VLG patterns (\(k=2\), \(delta _i,\varDelta _i = \langle 100,110\rangle \)), composed of very …


FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining

… sizes, totaling over 61 GB text: • English Wikipedia1 and BooksCorpus (Zhu et al., 2015), which are the original training data used to train BERT (totaling 13GB, 3.31B words); • FinancialWeb (totaling 24GB, 6.38B words), which …


M Muraoka, T Nasukawa, KMA Salam - US Patent App. 16/287,326, 2020

US20200272696A1 - Finding of asymmetric relation between words - Google Patents. Finding of asymmetric relation between words. Download PDF Info. Publication number US20200272696A1. US20200272696A1 US16/287,326 …


Finding the needle in the haystack: Fine-tuning transformers to classify protest events in a sea ofnews articles, with Bayesian uncertainty measures

Page 1. Finding the needle in the haystack: Fine-tuning transformers to classify protest events in a sea of news articles, with Bayesian uncertainty measures Chris Ghai Master's Thesis, Spring 2020 Page 2. This master's thesis …


Findings of the 2020 conference on machine translation (wmt20)

L Barrault, M Biesialska, O Bojar, MR Costa-jussà… - Proceedings of the Fifth …, 2020

… Distinct words – 76,013 – 6,165 178,453 85,189 Common Crawl Parallel Corpus German ↔ English Czech ↔ English Russian ↔ English French ↔ German … Common Crawl Language Model Data English German Czech Russian Polish Sent …


Findings of the WMT 2020 Biomedical Translation Shared Task: Basque, Italian and Russian as New Additional Languages

R Bawden, G Di Nunzio, C Grozea, I Unanue, A Yepes… - 5th Conference on Machine …, 2020

Page 1. HAL Id: hal-02986356 https://hal.inria.fr/hal-02986356 Submitted on 2 Nov 2020 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not …


Findings of the WMT 2020 shared task on parallel corpus filtering and alignment

P Koehn, V Chaudhary, A El-Kishky, N Goyal, PJ Chen… - Proceedings of the Fifth …, 2020

… Noisy parallel documents and parallel sentences were sourced from the CCAligned2 dataset (El- Kishky et al., 2020a), a massive collection of cross-lingual web documents covering over 8k language pairs aligned from …


Fine-Grained Argument Unit Recognition and Classification

D Trautmann, J Daxenberger, C Stab, H Schütze…

… Table 1). This also increases comparability with related work. The topics are general enough to have good coverage in Common Crawl … 4http://commoncrawl.org/2016/02/ february-2016-crawlarchive-now-available/ 5https://www.elastic.co/products/elasticsearch …


Fine-grained entity type classification using GRU with self-attention

K Dhrisya, G Remya, A Mohan - International Journal of Information Technology, 2020

… The dictionary for the proposed model is built with GLoVe vectors of 300 dimensions. It is a pre-trained word embedding created by utilizing common Crawl 840B. These vectors on various corpora can be downloaded from Stanford GLoVe Website. Parameter settings …


FLERT: Document-Level Features for Named Entity Recognition

S Schweter, A Akbik - arXiv preprint arXiv:2011.06993, 2020

… Conneau et al. (2019). We use the xlm-roberta-large model in our experiments, trained on 2.5TB of data from a cleaned Common Crawl corpus (Wenzek et al., 2020) for 100 different languages Embeddings (+WE). For each setup …


Forum Duplicate Question Detection by Domain Adaptive Semantic Matching

… B. MODEL IMPLEMENTATION The word embedding was initialized with 300-dimensional GloVe [33] vectors which are pretrained in the 840B Common Crawl corpus. The Embedding was set to be trainable. The …


Four dimensions characterizing comprehensive trait judgments of faces

… and text classi cation using a neural network provided within the FastText library40; this neural network had been trained on Common Crawl data of 600 billion words to predict the identity of a word given a context. We then applied …


French Contextualized Word-Embeddings with a sip of CaBeRnet: a New French Balanced Reference Corpus

M Fabre, PJO Suárez, B Sagot, ÉV de la Clergerie - CMLC-8-8th Workshop on the …, 2020

M Popa-Fabre, PJO Suárez, B Sagot… - Proceedings of the 8th …, 2020

… al., 2019), we decided to include in our comparison a corpus of French text extracted from Common Crawl8. We … 8More information available at https://commoncrawl … OSCAR gathers a set of monolingual text extracted …


Frequency-dependent Regularization in Constituent Ordering Preferences

… a total of around 9 billion tokens. This corpus consists of web page data from both Common Crawl and Wikipedia and is automatically parsed with UDPipe (Straka & Straková, 2017). Within this corpus, each token is represented …


From Chest X-Rays to Radiology Reports: A Multimodal Machine Learning Approach

S Singh, S Karimi, K Ho-Shon, L Hamey - 2019 Digital Image Computing: Techniques …, 2019

… Also, on the text side, we use the Glove [30] word embeddings having a 300-dimensional embedding vector for each word, and have been trained on a generic text corpus named Common Crawl having 42B tokens, 1.9M vocab …


From Dataset Recycling to Multi-Property Extraction and Beyond

T Dwojak, M Pietruszka, Ł Borchmann, J Chłędowski… - arXiv preprint arXiv …, 2020

… T5. Recently proposed T5 model (Raffel et al., 2020) is a Transformer model pretrained on a cleaned version of CommonCrawl. T5 is famous for achieving excellent performance on the SuperGLUE benchmark (Wang et al., 2019) …


From Hero to Z\'eroe: A Benchmark of Low-Level Adversarial Attacks

S Eger, Y Benz - arXiv preprint arXiv:2010.05648, 2020

… The reason may be that our noises are not always natural, in the sense of having high support in large datasets such as CommonCrawl or Wikipedia, but they are still within the limits of cognitive abilities of ordinary humans …


Z Gu, S Zhou, L Niu, Z Zhao, L Zhang - arXiv preprint arXiv:2009.12232, 2020

Page 1. 1 From Pixel to Patch: Synthesize Context-aware Features for Zero-shot Semantic Segmentation Zhangxuan Gu, Siyuan Zhou, Li Niu*, Zihan Zhao, Liqing Zhang* Abstract—Zero-shot learning has been actively studied …


From Syntactic Structure to Semantic Relationship: Hypernym Extraction from Definitions by Recurrent Neural Networks Using the Part of Speech Information

Y Tan, X Wang, T Jia - International Semantic Web Conference, 2020

… In recent years, much research pay attention to extracting hypernyms from larger data resources via the high precise of pattern-based methods. [25] extract hypernymy relations from the CommonCrawl web corpus using lexico-syntactic patterns …


From Web Crawl to Clean Register-Annotated Corpora

V Laippala, S Rönnqvist, S Hellström, J Luotolahti… - … of the 12th Web as Corpus …, 2020

… crawl or extracting data from existing crawl-based datasets, such as Common Crawl1. As … CommonCrawl is a free and openly available web crawl maintained by the CommonCrawl … Lately the Common Crawl dataset has …


From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers

A Lauscher, V Ravishankar, I Vulić, G Glavaš - arXiv preprint arXiv:2005.00633, 2020

… It is trained on the CommonCrawl-100 data (Wenzek et al., 2019) of 100 languages … Interestingly, for both high7For XLM-R, we take the reported sizes of languagespecific portions of CommonCrawl-100 from Conneau et al …


From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers

A Lauscher, V Ravishankar, I Vulić, G Glavaš - … of the 2020 Conference on Empirical …, 2020

… sampling. XLM on RoBERTa (XLM-R). XLM-R (Conneau et al., 2020) is an instance of RoBERTa, robustly trained on a large multilingual CommonCrawl-100 (CC-100) corpus (Wenzek et al., 2019) covering 100 languages …


Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing

Z Dai, G Lai, Y Yang, QV Le - arXiv preprint arXiv:2006.03236, 2020

… ablation studies. 5 Page 6. • Large scale: Pretraining models for 500K steps with batch size 8K on the five datasets used by XLNet [3] and ELECTRA [5] (Wikipedia + Book Corpus + ClueWeb + Gigaword + Common Crawl). We will …


Gated Semantic Difference Based Sentence Semantic Equivalence Identification

X Liu, Q Chen, X Wu, Y Hua, J Chen, D Li, B Tang… - IEEE/ACM Transactions on …, 2020

… The word embeddings for the quora corpus are 300dimensional GloVe word vectors pre-trained from the 840B Common Crawl corpus [45] and for the LCQMC are 300-dimensional word vectors pre-trained from the Chinese 5[Online] …


Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer

J Zhao, S Mukherjee, S Hosseini, KW Chang… - arXiv preprint arXiv …, 2020

… is an OCCUPATION-TITLE” where name is recognized in each language by using the corresponding Named Entity Recognition model from spaCy.5 To control for the same time period for datasets across languages …


Gender Bias in Multilingual Embeddings

J Zhao, S Mukherjee, S Hosseini, KW Chang…

… an OCCUPATION-TITLE” where name is recognized in each language by using the corresponding Named Entity Recognition model from spaCy.5 To control for the same time period for datasets across languages …


Gender Detection on Social Networks using Ensemble Deep Learning

K Kowsari, M Heidarysafa, T Odukoya, P Potter… - arXiv preprint arXiv …, 2020

… 25d, 50d, 100d, and 200d vectors. This word embedding is trained over even bigger corpora, including Wikipedia and Common Crawl content. The objective function is as follows: f(wi − wj, ˜wk) = Pik Pjk (2) where wi is refer to …


Gender stereotype reinforcement: Measuring the gender bias conveyed by ranking algorithms

A Fabris, A Purpura, G Silvello, GA Susto - Information Processing & Management, 2020

… Corrado, Dean, 2013). Most frequently, they are learnt from large text corpora available online (such as Wikipedia, Google News and Common Crawl, capturing semantic relationships of words based on their usage. Recent work …


Gender stereotypes are reflected in the distributional structure of 25 languages

M Lewis, G Lupyan - Nature Human Behaviour, 2020

Cultural stereotypes such as the idea that men are more suited for paid work and women are more suited for taking care of the home and family, may contribute to gender imbalances in science, technology, engineering and …


Generalisation of Cyberbullying Detection

K Richard, L Marc-André - arXiv preprint arXiv:2009.01046, 2020

… We use FastText pre-trained on Common Crawl data featuring 300 dimensions and 2 million word vectors with subword information6 to convert the words into vector representations, of which we concatenate a 60-dimensional binary …


Generalize Sentence Representation with Self-Inference

… Our model is trained with the phrases in the parse trees and tested on the whole sentence. Experimental Settings We initialize word embeddings using the pretrained FastText common-crawl vectors (Mikolov et al. 2018) and freeze the weights during training …


Generating Categories for Sets of Entities

S Zhang, K Balog, J Callan - arXiv preprint arXiv:2008.08428, 2020

… entity linking for tables and table schema to predicate matching. Ritze et al. [31] propose an iterative method for matching tables to DBpedia. They develop a manually annotated dataset for matching between a Web table corpus …


Generating Diverse Conversation Responses by Creating and Ranking Multiple Candidates

YP Ruan, ZH Ling, X Zhu, Q Liu, JC Gu - Computer Speech & Language, 2020

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Generating Fact Checking Briefs

A Fan, A Piktus, F Petroni, G Wenzek, M Saeidi… - Proceedings of the 2020 …, 2020

… We take the top search hit as the evidence and retrieve the text from CommonCrawl4. Finally, the generated question and retrieved evidence document is provided to the question answering model to generate an answer. 4.1 Question Generation …


Generating fake websites: WikiGen

… [2015]. This is a relatively small vector file. An alternative considered was the Common Crawl (840B tokens) vectors from Stanford NLP's GloVe [Pennington et al., 2014] but memory issues meant the smaller file was used instead …


Generating Keyword Lists Related to Topics Represented by an Array of Topic Records, for Use in Targeting Online Advertisements and Other Uses

L Palaic, MH Gross, SA Schriber - US Patent App. 16/803,214, 2020

… For data gathering purposes, a custom heuristic can be used that operates on a Common Crawl Corpus. Documents gathered from a Common Crawl process might be automatically annotated with appropriate topics tags so …


Page 1. Generating Personalized Product Descriptions from User Reviews Guy Elad Technion - Computer Science Department - M.Sc. Thesis MSC-2019-25 - 2019 Page 2. Technion - Computer Science Department …


Generating Query Suggestions for Cross-language and Cross-terminology Health Information Retrieval

PM Santos, C Teixeira Lopes - Advances in Information Retrieval: 42nd European …, 2020

… The English collection is provided by the Consumer Health Search Task in the 2018 edition of the CLEF eHealth Lab2. This task uses a set of 50 English queries and a document corpus with 5,535,120 web pages acquired from a CommonCrawl dump …


X Gu, Y Mao, J Han, J Liu, H Yu, Y Wu, C Yu, D Finnie… - arXiv preprint arXiv …, 2020

… By fine-tuning the model on human-curated labels, we can combine the two sources of supervision and further improve performance. 6One can use CommonCrawl to fetch web articles. Page 5. Generating Representative Headlines for News Stories …


Generative Language Modeling for Automated Theorem Proving

S Polu, I Sutskever - arXiv preprint arXiv:2009.03393, 2020

… We pre-train our models on both GPT-3's post-processed version of CommonCrawl as well as a more reasoning-focused mix of Github, arXiv and Math StackExchange. 7 … 5.3 Pre-training Models are pre-trained on …


Generative Models are Unsupervised Predictors of Page Quality: A Colossal-Scale Study

D Bahri, Y Tay, C Zheng, D Metzler, C Brunk… - arXiv preprint arXiv …, 2020

… 3.1 Datasets This section describes the datasets used in our experiments. • Web500M. The core corpora used in our experiments consists of a random sample of 500 million English web documents obtained from the Common Crawl1. • GPT-2-Output …


Geographically-Balanced Gigaword Corpora for 50 Language Varieties

J Dunn, B Adams - Proceedings of The 12th Language Resources and …, 2020

… 3. Collecting Geo-Referenced Documents The data for this paper comes from the Common Crawl,2 as processed in the Corpus of Global Language Use (henceforth, CGLU). This project includes the Common Crawl data from …


Geoparsing the historical Gazetteers of Scotland: accurately computing location in mass digitised texts

R Filgueira, C Grover, M Terras, B Alex - Proceedings of the 8th Workshop on …, 2020

… Small size model (11MB). • en core web md: English multi-task CNN trained on OntoNotes, with GloVe vectors trained on Common Crawl … en core web lg: English multi-task CNN trained on OntoNotes, with GloVe vectors trained on Common Crawl …


GeoVectors: a Linked Open Corpus of OpenStreetMap Embeddings

N Tempelmeier, S Gottschalk, E Demidova - 2020

… As most of the OSM keys are in English, we chose the 300-dimensional English word vectors trained on the Common Crawl and Wikipedia [5]. Encoding: To encode an OSM entity o, we utilise the individual word em …


German's Next Language Model

B Chan, S Schweter, T Möller - arXiv preprint arXiv:2010.10906, 2020

… The XLM-RoBERTa model is trained on 2.5TB of data from a cleaned Common Crawl corpus (Wenzek et al., 2020) for 100 different languages … OSCAR (Ortiz Suárez et al., 2019) is a set of monolingual corpora extracted from Common Crawl …


Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks

… Similarity-based approaches. The first baseline represents a given text, a question, and each of the choices as the average of 300-dimensional CommonCrawl FastText word embeddings (Bojanowski et al. 2016) of its constituent words …


Getting Passive Aggressive About False Positives: Patching Deployed Malware Detectors

E Raff, B Filar, J Holt - arXiv preprint arXiv:2010.12080, 2020

… We base our testing and results on a representative sample of industry data. Our corpus consisted of 1,101,407 Microsoft Office documents that contained macros. Similar to [33] data was collected from Common Crawl (CC) [34] and VirusTotal [35] …


Getting Structured Data from the Internet

… WARC file format ..... 278 Common crawl index ..... 282 … 331 Processing parquet files for a common crawl index ..... 334 …


Give your Text Representation Models some Love: the Case for Basque

R Agerri, IS Vicente, JA Campos, A Barrena, X Saralegi… - arXiv preprint arXiv …, 2020

… Common Crawl word vectors (FastText-officialcommon-crawl) were trained on Common Crawl and Wikipedia using CBOW with position-weights … train our systems to perform the following comparisons: (i) FastText official models …


GLEAKE: Global and Local Embedding Automatic Keyphrase Extraction

JR Asl, JM Banda - arXiv preprint arXiv:2005.09740, 2020

… doc2vec_news_dbow AP News glove.6B Wikipedia + Gigaword GloVe 50-300 [28] glove.twitter.27B Twitter 25-200 glove.840B Common Crawl 300 TABLE 1 DIFFERENT PRE-TRAINED EMBEDDINGS USED BY GLEAKE Page 5. 5 …


Page 1. GoURMET H2020–825299 D5.3 Initial Integration Report Global Under-Resourced MEedia Translation (GoURMET) H2020 Research and Innovation Action Number: 825299 D5.3 – Initial Integration Report Nature Report Work Package WP5 …


Going Back in Time to Find What Existed on the Web and How much has been Preserved: How much of Palestinian Web has been Archived?

T Sammar, H Khalilia - مؤتمرات الآداب والعلوم الانسانية والطبيعية, 2020

… References 1. Common crawl url index. url (http://index.commoncrawl.org/). 2. International internet preservation consortium (iipc).http://www.netpreserve.org. 3. Internet archive (https://archive.org/). 4. Internet archive wayback machine. url (https://archive.org/web/) …


Goku's Participation in WAT 2020

D Wang, O Htun - Proceedings of the 7th Workshop on Asian Translation, 2020

… Secondly, we fine-tuned on the JPO patent corpus using the mBART auto-encoder model (Liu et al., 2020), which has been pre-trained on largescale monolingual CommonCrawl (CC) corpus in 25 languages using the BART objective (Lewis et al., 2020) …


GoodReads Book Recommendation Service

… Common Crawl: https://commoncrawl.org/ Models: ● Howard, Jeremy and Ruder, Sebastian. "Universal Language Model Fine-tuning for Text Classification." Paper presented at the meeting of the ACL, 2018. ● Conneau …


GottBERT: a pure German Language Model

R Scheible, F Thomczyk, P Tippmann, V Jaravine… - arXiv preprint arXiv …, 2020

… dbmz BERT used as source data a German Wikipedia dump, EU Bookshop corpus, Open Subtitles, CommonCrawl, ParaCrawl and News Crawl which … than mBERT, the multilingual XLM-RoBERTa (Conneau et al., 2019) was …


GPT-3 AI language tool calls for cautious optimism

Oxford Analytica - Emerald Expert Briefings

… The training process leveraged this by exposing GPT-3 to historical sweeps of the internet, known as 'crawls'. One substantial component was the Common Crawl dataset, a multilingual capture of almost 1 trillion words …


GPT-3 Creative Fiction

… For GPT-2, I saw finetuning as doing 2 things: Fixing ignorance: missing domain knowledge. GPT-2 didn't know many things about most things—it was just a handful (1.5 billion) of parameters trained briefly on …


Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses

S Flachs, O Lacroix, H Yannakoudakis, M Rei… - arXiv preprint arXiv …, 2020

… matical errors. The source texts are randomly se- lected from the first 18 dumps of the CommonCrawl4 dataset and represent a wide range of data seen online such as blogs, magazines, corporate or educational websites. These …


Graph Attention Network with Memory Fusion for Aspect-level Sentiment Analysis

L Yuan, J Wang, LC Yu, X Zhang - Proceedings of the 1st Conference of the Asia …, 2020

Page 1. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 27–36 December 4 - 7, 2020 …


S Hu, Z Xiong, M Qu, X Yuan, MA Côté, Z Liu, J Tang - arXiv preprint arXiv …, 2020

Page 1. Graph Policy Network for Transferable Active Learning on Graphs Shengding Hu Tsinghua University hsd16@mails.tsinghua.edu.cn Zheng Xiong Tsinghua University harryfootball@163.com Meng Qu MILA meng.qu@umontreal.ca …


Page 1. Aalto University School of Science Master's Programme in ICT Innovation Lukas Brückner Graphical User Interface Auto-Completion with Element Constraints Master's Thesis Espoo, September 25, 2020 Supervisor: Prof …


Page 1. GraphWalker: An I/O-Efficient and Resource-Friendly Graph Analytic System for Fast and Scalable Random Walks Rui Wang1, Yongkun Li1, Hong Xie2, Yinlong Xu1, John CS Lui3 1University of Science and Technology …


GREEK-BERT: The Greeks visiting Sesame Street

J Koutsikakis, I Chalkidis, P Malakasiotis… - arXiv preprint arXiv …, 2020

… and (c) the Greek part of OSCAR [25], a clean version of Common Crawl.5 Accents and other diacritics were removed, and all words were … 5https://commoncrawl.org 6https://github.com/google-research/bert 7The …


N Pappas, P Mulcaire, NA Smith - arXiv preprint arXiv:2009.11523, 2020

Page 1. Grounded Compositional Outputs for Adaptive Language Modeling Nikolaos Pappas♣ Phoebe Mulcaire♣ Noah A. Smith♣♦ ♣Paul G. Allen School of Computer Science & Engineering, University of Washington ♦Allen …


Guided Generation of Cause and Effect

… 3629 Page 2. Processed Common Crawl Corpus Causal Patterns Based Matching and Filtering … Thus we harvest a large causal dataset from the preprocessed large-scale English Common Crawl corpus (5.14 TB) [Buck et al., 2014] …


Harbsafe-162. A Domain-Specific Data Set for the Intrinsic Evaluation of Semantic Representations for Terminological Data

S Arndt, D Schnäpp - arXiv preprint arXiv:2005.14576, 2020

Page 1. Harbsafe-162 – A Domain-Specific Data Set for the Intrinsic Evaluation of Semantic Representations for Terminological Data Susanne Arndt, MA∗ Technische Universität Braunschweig Dieter Schnäpp, MA∗∗ Technische Universität Braunschweig …


Hard-Coded Gaussian Attention for Neural Machine Translation

W You, S Sun, M Iyyer - arXiv preprint arXiv:2005.00742, 2020

… 10As the full WMT14 En→Fr is too large for us to feasibly train on, we instead follow Akoury et al. (2019) and train on just the Europarl / Common Crawl subset, while evaluating using the full dev/test sets. 11https://github.com/dojoteef/synst Page 5 …


Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languages

X Garcia, A Siddhant, O Firat, AP Parikh - arXiv preprint arXiv:2009.11201, 2020

… In contrast, for an actual low-resource language, Gujarati, WMT only provides 500 thousand lines of monolingual data (in news domain) and an additional 3.7 million lines of monolingual data from Common Crawl (noisy, generaldomain) …


B Mathew, P Saha, SM Yimam, C Biemann, P Goyal… - arXiv preprint arXiv …, 2020

Page 1. HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection* Binny Mathew1†, Punyajoy Saha1†, Seid Muhie Yimam2 Chris Biemann2, Pawan Goyal1, Animesh Mukherjee1 1 Indian Institute of Technology …


HCA: Hierarchical Compare Aggregate model for question retrieval in community question answering

MS Zahedi, M Rahgozar, RA Zoroofi - Information Processing & Management, 2020

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Hidden in Plain Sight: Building a Global Sustainable Development Data Catalogue

J Hodson, A Spezzatti - ICT Analysis and Applications, 2020

… We iteratively re-train our model using 300-dimensional word-embedding features trained on the CommonCrawl web-scale data set 7 with the GLobal VEctors for Word Representation (GloVe) procedure (see [7]). Table 2 shows …


Hierarchical models vs. transfer learning for document-level sentiment classification

J Barnes, V Ravishankar, L Øvrelid, E Velldal - arXiv preprint arXiv:2002.08131, 2020

… Universal Language Model Fine-Tuning (ULMFIT): We use the AWD-LSTM architecture (Merity et al., 2018) and pretrain on Wikipedia data (or Common Crawl in the case of Norwegian) taken from the CONLL 2017 shared task (Zeman et al., 2017) …


Hierarchical Multimodal Attention for End-to-End Audio-Visual Scene-Aware Dialogue Response Generation

H Le, D Sahoo, NF Chen, SCH Hoi - Computer Speech & Language, 2020


High Accuracy Phishing Detection Based on Convolutional Neural Networks

… Their approach first encodes the URL strings using one-hot encoding and then inputs each encoded character vector into the LSTM neurons for training and testing. Their method achieved an accuracy of 0.935 on the Common …


HitAnomaly: Hierarchical Transformers for Anomaly Detection in System Log

S Huang, Y Liu, C Fung, R He, Y Zhao, H Yang, Z Luan - IEEE Transactions on …, 2020

… matrix. Finally, we obtain the word vector of 'terminating' as [16, 2, 11]. LogRobust [10] leverages off-the-shelf word vectors, which were pre-trained on Common Crawl Corpus dataset using the Fast-Text [20] algorithm. We initialize …


How “BERTology” Changed the State-of-the-Art also for Italian NLP

F Tamburini - Proceedings of the Seventh Italian Conference on …, 2020

… Page 2. billions of tokens. Also for GilBERTo it is available only the uncased model. • UmBERTo4: the more recent model de- veloped explicitly for Italian, as far as we know, is UmBERTo ('Musixmatch/umbertocommoncrawl-cased-v1' – umC) …


How Furiously Can Colourless Green Ideas Sleep? Sentence Acceptability in Context

JH Lau, CS Armendariz, S Lappin, M Purver, C Shu - arXiv preprint arXiv:2004.00881, 2020

… BERTUCS Transformer Bidir. 340M Uncased 13GB WordPiece Wikipedia, BookCorpus XLNET Transformer Hybrid 340M Cased 126GB SentenceWikipedia, BookCorpus, Giga5 Piece ClueWeb, Common Crawl Table 1: Language models and their configurations …


How Human is Machine Translationese? Comparing Human and Machine Translations of Text and Speech

Y Bizzoni, TS Juzek, C España-Bonet, KD Chowdhury… - Proceedings of the 17th …, 2020

… German translation and interpreting are both from English. lines de tokens en tokens Ct Cs CommonCrawl 2,212,292 49,870,179 54,140,396 MultiUN 108,387 4,494,608 4,924,596 NewsCommentary 324,388 8,316,081 46,222,416 …


How Language Shapes Prejudice Against Women: An Examination Across 45 World Languages

… context in which it occurs. Using text data from Wikipedia and the Common Crawl project … discussing gender issues. Wikipedia and a corpus of web crawl data from over five billion web pages, known as the Common Crawl, serve as our data source …


How Many Pages? Paper Length Prediction from the Metadata

E Çano, O Bojar - arXiv preprint arXiv:2010.15924, 2020

… We used static word embeddings of 300 dimensions from three sources: the 6 billion tokens collection of Common Crawl4 trained with Glove [23], the 840 billion tokens collection of Common Crawl trained with Glove, and …


How Much Self-attention Do We Need? Trading Attention for Feed-forward Layers

K Irie, A Gerstenberger, R Schlüter, H Ney - ICASSP, Barcelona, Spain, 2020

… to 6 (in contrast to what we typically observe eg, on LibriSpeech [28]): this is in fact due to some overlap between the common crawl training subset … Then we fine-tune the model on the TED-LIUM 2 transcriptions (2 M words) …


How Should Markup Tags Be Translated?

… especially large or noisy data sets. For EN–DE, we begin with the training data released by the WMT 2020 news task, ignoring the Common Crawl and Paracrawl corpora and heavily filtering WikiMatrix. Our EN–FR training data …


Human-in-the-Loop AI for Analysis of Free Response Facial Expression Label Sets

C Butler, H Oster, J Togelius - Proceedings of the 20th ACM International Conference …, 2020

… 1. GloVe, 300-dimensional vectors trained on Common Crawl [33]: distributional model that learns word vectors by examining word co- occurrences within a text corpus with logbilinear regression, global matrix factorization and local context window methods …


R Schuster, T Schuster, Y Meri, V Shmatikov - arXiv preprint arXiv:2001.04935, 2020

Page 1. Humpty Dumpty: Controlling Word Meanings via Corpus Poisoning* Roei Schuster Tel Aviv University † roeischuster@mail.tau.ac.il Tal Schuster CSAIL, MIT tals@csail.mit.edu Yoav Meri † 111yoav@gmail.com Vitaly …


Hungarian layer: A novel interpretable neural layer for paraphrase identification


HW-TSC's Participation in the WMT 2020 News Translation Shared Task

D Wei, H Shang, Z Wu, Z Yu, L Li, J Guo, M Wang…

… monolingual text from Common Crawl and news crawl 2018 for Km and En, respectively. 2.1.3 Ps/En Similar to Km/En, we also use the Para Crawl v5.1 (1M), Khmer and Pashto parallel data (0.03M) as bitext and select …


Hybrid Feature Model for Emotion Recognition in Arabic Text

N Alswaidan, MEB Menai - IEEE Access, 2020

… 5https://github.com/minimaxir/char-embeddings 6https://unicode.org/emoji/ charts/full-emoji-list.html • FastText [22]: 300-dimensional word vectors trained on Common Crawl7 using CBOW with position weights … 7https://commoncrawl …


HyCoNN: Hybrid Cooperative Neural Networks for Personalized News Discussion Recommendation

… For DeepCoNN and also HyCoNN, we use 300-dimensional fastText word embeddings, which were pre-trained on the English-language Common Crawl dataset [25]. The resulting embeddings function as input to a convolutional layer that consists of n neurons …


Identifying Cognates in English-Dutch and French-Dutch by means of Orthographic Information and Cross-lingual Word Embeddings

E Lefever, S Labat, P Singh - the 12th Conference on Language Resources and …, 2020

… The former approach was improved in the following way. Firstly, standard fastText word embeddings, which were pretrained on Common Crawl and Wikipedia and generated with the standard skip-gram model as proposed by Bo- janowski et al …


Identifying Phished Website Using Multilayer Perceptron

A Dev, V Jain - Advances in Distributed Computing and Machine …

… Phishing Webpage Source: PhishTank, OpenPhish. Legitimate Webpage Source: Alexa, Common Crawl. The main process in the phishing webpage is to work on its features and how effectively it is handling the dataset. Each …


Identifying Sensitive URLs at Web-Scale

M Srdjan, I Costas, S Georgios, N Laoutaris - 2020

S Matic, C Iordanou, G Smaragdakis, N Laoutaris - studies

… We then use our classifier to search for sensitive URLs in a corpus of 1 Billion URLs collected by the Common Crawl project. We identify more than 155 millions sensitive URLs in more than 4 million domains … Automated …


Y Tian, K Zhou, M Lalmas, D Pelleg - Proceedings of the 43rd International ACM …, 2020

Page 1. Identifying Tasks from Mobile App Usage Patterns Yuan Tian University of Nottingham Nottingham, UK yuan.tian@nottingham.ac.uk Ke Zhou University of Nottingham Nottingham, UK ke.zhou@nottingham.ac …


Igbo-English Machine Translation: An Evaluation Benchmark

I Ezeani, P Rayson, I Onyenwe, C Uchechukwu… - arXiv preprint arXiv …, 2020

… Page 2. Published as a conference paper at ICLR 2020 texts (eg Wikipedia, CommonCrawl, local government materials, local TV/Radio stations etc). Phase 2: Translation and correction In this phase, the 10,000 sentence pairs …


J Heo, J Won, Y Lee, S Bharuka, J Jang, TJ Ham…

Page 1. IIU: Specialized Architecture for Inverted Index Search Jun Heo∗ Jaeyeon Won∗ Yejin Lee∗ Shivam Bharuka†§ Jaeyoung Jang‡ Tae Jun Ham∗ Jae W. Lee∗ ∗Seoul National University, †Facebook, Inc., ‡Sungkyunkwan University …


Imitation Attacks and Defenses for Black-box Machine Translation Systems

E Wallace, M Stern, D Song - arXiv preprint arXiv:2004.15015, 2020

… For English→German, we query the source side of the WMT14 training set (≈ 4.5M sentences).3 For Nepali→English, we query the Nepali Language Wikipedia (≈ 100,000 sentences) and approximately two million sentences from Nepali common crawl …


Impact of News on the Commodity Market: Dataset and Results

A Sinha, T Khandait - arXiv preprint arXiv:2009.04202, 2020

… The GloVe pre-trained word-embeddings are known to capture the meaning of a word through a high dimensional vector [22]. For this research, we used the 300-dimensional vectors which were trained on 840 billion tokens through the common crawl …


Impact of sentence length on the readability of web for screen reader users

BB Kadayat, E Eika - International Conference on Human-Computer …, 2020

… They used MapReduce for real-time calculation of the readability of more than a billion webpages. The datasets called Common Crawl included 61 million domain-names, 92 million PDF documents, and seven million Word documents …


Improved method of word embedding for efficient analysis of human sentiments

S Sagnika, BSP Mishra, SK Meher - Multimedia Tools and Applications, 2020

… Designed by Pennington [23] in 2014, it creates a word vector space by training on word-word co-occurrence counts. The models are trained on Wikipedia dumps, Gigaword 5 and Common Crawl texts, and apply …


Improving Indonesian Text Classification Using Multilingual Language Model

IF Putra, A Purwarianti - arXiv preprint arXiv:2009.05713, 2020

… The XLM-R Large also has substantially more parameters and was trained on a larger balanced dataset from the CommonCrawl corpus that contains 100 languages. The XLM-R variant that we use in the experiment is …


Page 1. IMPROVING KNOWLEDGE ACCESSIBILITY ON THE WEB – from Knowledge Base Augmentation to Search as Learning Inaugural dissertation for the attainment of the title of doctor in the Faculty of Mathematics and …


Improving Low Compute Language Modeling with In-Domain Embedding Initialisation

C Welch, R Mihalcea, JK Kummerfeld - arXiv preprint arXiv:2009.14109, 2020

… We see that the value of additional data depends on the domain. Gigaword is also news text and is able to improve performance. The larger GloVe datasets use Wikipedia and CommonCrawl data, which is a poorer match and so does not improve performance …


Improving Network Security through Collaborative Sharing

Page 1. IMPROVING NETWORK SECURITY THROUGH COLLABORATIVE SHARING by Calvin Satiawan Ardi A Dissertation Presented to the FACULTY OF THE USC GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA …


Improving Personal Health Mention Detection on Twitter Using Permutation Based Word Representation Learning

PI Khan, I Razzak, A Dengel, S Ahmed - International Conference on Neural …, 2020

… In: Invited Talk at the SIGIR 2012 Workshop on Open-Source Information Retrieval (2012)Google Scholar. 24. Common CrawlCommon crawl corpus (2019). http://commoncrawl.org. Copyright information. © Springer Nature …


Improving Ranking in Document based Search Systems

RRK Menon, J Kaartik, ETK Nambiar, AK TK, A Kumar - 2020 4th International …, 2020

… The pre-trained word embedding models used were: 1. Fasttext i. – Wiki News 1M 16B Tokens ii. – Common Crawl 2M 600B Tokens 2. GloVe 2.2M 840B Tokens B. Evaluation Metrics The standard metrics include Precision, Recall, and F1 score …


Page 1. Increasing Accessibility of Electronic Theses and Dissertations (ETDs) Through Chapter-level Classification Palakh Mignonne Jude Thesis submitted to the Faculty of the Virginia Polytechnic Institute and State University …


INDEXING OF BIG TEXT DATA AND SEARCHING IN THE INDEXED DATA

… enhanced documents. Input data In [14, 8], the input data was CommonCrawl and Wikipedia. The English wikipedia … 8 Page 15. CommonCrawl3is a project that maintains an open repository of web crawl data. For the practical part of …


Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages

K Jain, A Deshpande, K Shridhar, F Laumann, A Dash - arXiv preprint arXiv …, 2020

… The Open Super-large Crawled ALMAnaCH coRpus (OSCAR) dataset [48] is a filtered version of the CommonCrawl dataset and has monolingual corpora for 166 languages. Prior to training, we normalize the OSCAR dataset for …


IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages

D Kakwani, A Kunchukuttan, S Golla, A Bhattacharyya…

… The OSCAR project (Ortiz Suarez et al., 2019), a recent processing of CommonCrawl, also contains much less data for most Indian languages than our crawls. The CCNet () and C4 () projects also provide tools to …


IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding

B Wilie, K Vincentio, GI Winata, S Cahyawijaya, X Li… - arXiv preprint arXiv …, 2020

… Page 5. Dataset # Words # Sentences Size Style Source OSCAR (Ortiz Suárez et al., 2019) 2,279,761,186 148,698,472 14.9 GB mixed OSCAR CoNLLu Common Crawl (Ginter et al., 2017) 905,920,488 77,715,412 6.1 GB mixed LINDAT/CLARIAH-CZ …


Inducing Language-Agnostic Multilingual Representations

W Zhao, S Eger, J Bjerva, I Augenstein - arXiv preprint arXiv:2008.09112, 2020

… XLM-R Contextualized word embeddings (Conneau et al., 2019) are pre-trained on the CommonCrawl corpora of 100 languages, which contain more monolingual data than Wikipedia corpora, with 1) a vocabulary size …


B Wang, G Wang, J Huang, J You, J Leskovec… - arXiv preprint arXiv …, 2020

Page 1. Inductive Learning on Commonsense Knowledge Graph Completion Bin Wang1, Guangtao Wang2, Jing Huang2, Jiaxuan You3, Jure Leskovec3, C.-C. Jay Kuo1 1 University of Southern California 2 JD AI Research …


Inexpensive Domain Adaptation of Pretrained Language Models: A Case Study on Biomedical Named Entity Recognition

N Poerner, U Waltinger, H Schütze - arXiv preprint arXiv:2004.03354, 2020

… 1 Introduction Pretrained Language Models such as BERT (De- vlin et al., 2019) have spearheaded advances on many NLP tasks. Usually, PTLMs are pretrained on unlabeled general-domain and/or mixed-domain text, such …


Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA

… Pretrained Language Models (PTLMs) such as BERT (Devlin et al., 2019) have spearheaded ad- vances on many NLP tasks. Usually, PTLMs are pretrained on unlabeled general-domain and/or mixed-domain text, such …


Infosys Machine Translation System for WMT20 Similar Language Translation Task

K Rathinasamy, A Singh, B Sivasambagupta… - Proceedings of WMT, 2020

… data. 2.2.2 Synthetic data CommonCrawl n-grams raw monolingual files are processed1 to remove sentences with invalid characters, strip leading and trailing whitespaces, and remove duplicate sentences. 3 System Overview …


iNLTK: Natural Language Toolkit for Indic Languages

G Arora - arXiv preprint arXiv:2009.12534, 2020

… iNLTK results were compared against results reported in (Kunchukuttan et al., 2020) for pre-trained embeddings released by the FastText project trained on Wikipedia (FT-W) (Bo- janowski et al., 2016), Wiki+CommonCrawl …


INSET: Sentence Infilling with INter-SEntential Transformer

Y Huang, Y Zhang, O Elachqar, Y Cheng - Proceedings of the 58th Annual Meeting of …, 2020

… For the effectiveness of human evaluation, we use the simplest strategy to mask sentences. The Recipe dataset is obtained from (https: //commoncrawl.org), where the metadata is formatted according to Schema.org (https:// schema.org/Recipe) …


Integrating Geospatial Data and Social Media in Bidirectional Long-Short Term Memory Models to Capture Human Nature Interactions

A Larkin, P Hystad - The Computer Journal, 2020

… language processing [32]. Tweet texts were transformed into word vector arrays using the Stanford GloVe (Global Vectors for Word Representation) Common Crawl dictionary (https://nlp.stanford.edu/projects/glove/). The GloVe …


Intelligent phishing detection scheme using deep learning algorithms

MA Adebowale, KT Lwin, MA Hossain - Journal of Enterprise Information …, 2020

… Half of the data set consisted of phishing sites from PhishTank, which is a site that is used as phishing URL depository, and half of the data set was comprised of legitimate sites from Common Crawl, a corpus of web crawl data …


Intermediate Training of BERT for Product Matching

… uct Corpus for Large-Scale Product Matching [26]. These datasets are derived from schema.org annotations from thousands of webshops extracted from the Common Crawl. Relying on schema.org annotations of product identifiers …


Interpretability Analysis for Named Entity Recognition to Understand System Predictions and How They Can Improve

O Agarwal, Y Yang, BC Wallace, A Nenkova - arXiv preprint arXiv:2004.04564, 2020

… representations are concatenated. We use 300 dimensional cased GloVe (Pennington et al., 2014) vectors trained on Common Crawl.2 We use the IO labeling scheme and evaluate the systems via micro-F1, at the token level. We use …


Interpretable & Time-Budget-Constrained Contextualization for Re-Ranking

S Hofstätter, M Zlabinger, A Hanbury - arXiv preprint arXiv:2002.01854, 2020

… The first section contains the traditional baselines; the second contains the neural re-ranking baselines; in the third section we report the results of our TK model with three 6 42B CommonCrawl lower-cased: https://nlp.stanford.edu/projects/glove/ Page 5 …


Introduction to Cloud Computing and Amazon Web Services (AWS)

JM Patel - Getting Structured Data from the Internet, 2020

… 5 examples. IAM and S3 sections are necessary for Chapters 6 and 7 since we will be using data compiled by a nonprofit called common crawl which is only publicly available on S3 through AWS open registry. You will have …


Introduction to Common Crawl Datasets

The Common Crawl Foundation (https://commoncrawl.org/) is a 501(c)(3) nonprofit involved in providing open access web crawl data going back to over eight years. They perform monthly web crawls which cover over 25 billion pages for each month. This …


Introduction to Web Scraping

… We will introduce natural language processing algorithms in Chapter 4, and we will put them into action in Chapters 6 and 7 on a Common Crawl dataset. The next step is loading the cleaned data from the preceding step into an appropriate database …


Is Everything Fine, Grandma? Acoustic and Linguistic Modeling for Robust Elderly Speech Emotion Recognition

G Sogancıoglu, O Verkholyak, H Kaya, D Fedotov… - INTERSPEECH, Shanghai …, 2020

G Soğancıoğlu, O Verkholyak, H Kaya, D Fedotov… - arXiv preprint arXiv …, 2020

… We use pre-trained 100-dimensional English and German word embeddings [23], which are trained on Common Crawl2, and finetune the pretrained model on our dataset … 1https://cloud.google.com/translate 2http://commoncrawl.org/ Page 3. cording to its POS …


Is language modeling enough? Evaluating effective embedding combinations

R Schneider, T Oberhauser, P Grundmann, FA Gers… - 2020

… 2.1. Universal Text Embeddings Recently, researchers explore universal text embeddings trained on extensive Web corpora, such as the Common Crawl6 (Mikolov et al., 2018; Radford et al., 2019), the billion … 5https …


Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation

B Eikema, W Aziz - arXiv preprint arXiv:2005.10283, 2020

… For English-Nepali we also use a translated version of the Penn Treebank4 and for English-Sinhala we additionally use Open Subtitles (Lison et al., 2018). We use a filtered crawl of Wikipedia and Common Crawl released in Guzmán et al …


Is Wikipedia succeeding in reducing gender bias? Assessing changes in gender bias in Wikipedia using word embeddings

KG Schmahl, TJ Viering, S Makrodimitris, AN Jahfari… - Proceedings of the Fourth …, 2020

… These categories have shown significant bias towards male or female words in embeddings from Google News corpora [Mikolov et al., 2013a], Google Books [Jones et al., 2020], as well as a 'Common Crawl' corpus [Caliskan et al., 2017] …


It's the Best Only When It Fits You Most: Finding Related Models for Serving Based on Dynamic Locality Sensitive Hashing

L Zhou, Z Wang, A Das, J Zou - arXiv preprint arXiv:2010.09474, 2020

… BAIR CORD-19 LSUN Bedroom iNaturalist (iNat) 2017 ImageNet OpenImagesV4 Wikipedia 1 Billion Word Benchmark CommonCrawl Multillingual Wikipedia Natural Questions 3 15 3 3 8 10 9 7 2 5 58 3 5 2 5 2 CelebA HQ iMet Collection 2019 …


Italian Transformers Under the Linguistic Lens

A Miaschip, G Sartim, D Brunato, F Dell'Orletta… - Proceedings of the Seventh …, 2020

… For instance, we can notice that, for both the probing models, features related to the distribution of syntactic relations (SyntacticDep) are better predicted by GePpeTto, while GilBERTo and UmBERTo-Commoncrawl are the best …


JASS: Japanese-specific Sequence to Sequence Pre-training for Neural Machine Translation

Z Mao, F Cromieres, R Dabre, H Song, S Kurohashi - arXiv preprint arXiv:2005.03361, 2020

… Mono Ja Common Crawl 22M En News Crawl 22M Ru News Crawl 22M … 5.1.2. Monolingual data We use monolingual data containing 22M Japanese, 22M English and 22M Russian sentences randomly sub-sampled from Common Crawl dataset and News crawl4 dataset …


Joint Multiclass Debiasing of Word Embeddings

R Popović, F Lemmerich, M Strohmaier - arXiv preprint arXiv:2003.11520, 2020

… As in previous studies [7], evaluation was done on three pretrained Word Embedding models with vector dimension of 300: FastText2(English we- bcrawl and Wikipedia, 2 million words), GloVe3(Common Crawl, Wikipedia …


Joint translation and unit conversion for end-to-end localization

G Dinu, P Mathur, M Federico, S Lauly, Y Al-Onaizan - arXiv preprint arXiv …, 2020

… Europarl (Koehn, 2005) and news commentary data from WMT En→De shared task 2019 totalling 2.2 million sentences.2 Standard translation test sets do not have, however, enough examples of unit conversions and in fact corpora …


KBPearl: a knowledge base population system supported by joint entity and relation linking

X Lin, H Li, H Xin, Z Li, L Chen - Proceedings of the VLDB Endowment, 2020

Page 1. KBPearl: A Knowledge Base Population System Supported by Joint Entity and Relation Linking Xueling Lin, Haoyang Li, Hao Xin, Zijian Li, Lei Chen Department of Computer Science and Engineering The Hong Kong …


Keeping Models Consistent between Pretraining and Translation for Low-Resource Neural Machine Translation

W Zhang, X Li, Y Yang, R Dong, G Luo - Future Internet, 2020

Recently, the pretraining of models has been successfully applied to unsupervised and semi-supervised neural machine translation. A cross-lingual language model uses a pretrained masked language model to initialize the …


Kernel compositional embedding and its application in linguistic structured data classification

H Ganji, MM Ebadzadeh, S Khadivi - Knowledge-Based Systems, 2020

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Key Phrase Classification in Complex Assignments

M Ravikiran - arXiv preprint arXiv:2003.07019, 2020

… Corpus and English Wikipedia used in BERT was found to be useful for training. The additional data included Common Crawl News dataset (76 GB), Web text corpus (38 GB) and Stories from Common Crawl (31 GB). This coupled …


Keynote speaker


Keyphrase Extraction as Sequence Labeling Using Contextualized Embeddings

D Sahrawat, D Mahata, H Zhang, M Kulkarni, A Sharma… - Advances in Information …, 2020

… We also use 300 dimensional fixed embeddings from Glove [20], Word2Vec [19], and FastText [13] (common-crawl, wiki-news). We also compare the proposed architecture against four popular baselines …


KGvec2go--Knowledge Graph Embeddings as a Service

J Portisch, M Hladik, H Paulheim - arXiv preprint arXiv:2003.05809, 2020

… knowledge graphs. 4.3. WebIsALOD The WebIsA database (Seitner et al., 2016) is a data set which consists of hypernymy relations ex- tracted from the Common Crawl8, a downloadable copy of the Web. The extraction was …


KIT's IWSLT 2020 SLT Translation System

NQ Pham, F Schneider, TN Nguyen, TL Ha, TS Nguyen… - Proceedings of the 17th …, 2020

… Table 2: Text Training Data Dataset Sentences TED Talks (TED) 220K Europarl (EPPS) 2.2MK CommonCrawl 2.1M Rapid 1.21M ParaCrawl 25.1M OpenSubtitles 12.6M WikiTitle 423K Back-translated News 26M Page 2. 56 3 Simultaneous Speech Translation …


KLEJ: Comprehensive Benchmark for Polish Language Understanding

P Rybak, R Mroczkowski, J Tracz, I Gawlik - arXiv preprint arXiv:2005.00630, 2020

… word vectors. To evaluate their impact on KLEJ tasks, we initialize word embeddings with fastText (Bojanowski et al., 2016) trained on Common Crawl and Wikipedia for Polish language (Grave et al., 2018). 4.1.3 ELMo ELMo …


KLUMSy@ KIPoS: Experiments on Part-of-Speech Tagging of Spoken Italian

… The PAISÀ corpus of Italian texts from the web (Lyding et al., 2014),5 the text of the Italian Wikimedia dumps,6 ie Wiki(pedia|books|news|versity|voyage), as ex- tracted by Wikipedia Extractor,7 and the Italian subset of OSCAR …


Page 1. MASTERTHESIS Knowledge Augmented Aspect Category Detection for Aspect-based Sentiment Analysis Kai Martinen 01.12.2019 University of Hamburg MIN-Faculty Department of Computer Science Language Technologies …


H Wang - arXiv preprint arXiv:2008.12878, 2020

Page 1. Knowledge Efficient Deep Learning for Natural Language Processing by Hai Wang A thesis submitted in partial fulfillment for the degree of Doctor of Philosophy in Computer Science at the Toyota Technological Institute …


Knowledge Graphs Evolution and Preservation--A Technical Report from ISWS 2019

N Abbas, K Alghamdi, M Alinam, F Alloatti, G Amaral… - arXiv preprint arXiv …, 2020

Page 1. Knowledge Graphs Evolution and Preservation A Technical Report from ISWS 2019 December 23, 2020 Bertinoro, Italy arXiv:2012.11936v1 [cs.AI] 22 Dec 2020 Page 2. Authors Main Editors Valentina Anita Carriero …


KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding

J Ham, YJ Choe, K Park, I Choi, H Soh - arXiv preprint arXiv:2004.03289, 2020

… 20 days). We also use XLM-R (Conneau and Lample, 2019), a publicly available cross-lingual language model that was pre-trained on 2.5TB of Common Crawl corpora in 100 languages including Korean (54GB). Note that …


LAMBERT: Layout-Aware language Modeling using BERT for information extraction

Ł Garncarek, R Powalski, T Stanisławek, B Topolski… - arXiv preprint arXiv …, 2020

… Dataset pages EDGAR 119 088 RVL-CDIP 90 054 Common Crawl 389 469 cTDaR 782 private 151 074 Total 750 467 Table 1: Sizes of training datasets … Common Crawl PDFs This is a dataset produced by downloading PDF …


Language model domain adaptation for automatic speech recognition

… By exploring and exploiting various datasets like Common Crawl, Europarl, news and TEDLIUM and by experimenting different techniques in training a model, we achieve the goal of adapting a general purpose LM to a domain like talks …


D Loureiro, K Rezaee, MT Pilehvar… - arXiv preprint arXiv …, 2020

Page 1. Language Models and Word Sense Disambiguation: An Overview and Analysis Daniel Loureiro∗ LIAAD - INESC TEC Department of Computer Science - FCUP University of Porto, Portugal Kiamehr Rezaee∗ Department …


Language Models are Few-Shot Learners

TB Brown, B Mann, N Ryder, M Subbiah, J Kaplan… - arXiv preprint arXiv …, 2020

… of our held-out validation set as an accurate measure of overfitting, and (3) we also added known high-quality reference corpora to the training mix to augment CommonCrawl and increase its diversity. Details of the first two …


Language Models are Open Knowledge Graphs

C Wang, X Liu, D Song - arXiv preprint arXiv:2010.11967, 2020

… In fact, these pre-trained LMs automatically acquire factual knowledge from large-scale corpora (eg, BookCorpus (Zhu et al., 2015), Common Crawl (Brown et al., 2020)) via pre-training. The learned knowledge in pre-trained LMs is the key to the current success …


Language Models not just for Pre-training: Fast Online Neural Noisy Channel Modeling

S Bhosale, K Yee, S Edunov, M Auli - arXiv preprint arXiv:2011.07164, 2020

… Big architecture. The model is trained on de-duplicated Romanian CommonCrawl data consisting of 623M sentences or 21.7B words after normalization and tokenization (Conneau et al., 2019; Wenzek et al., 2020). The German …


Language-agnostic BERT Sentence Embedding

F Feng, Y Yang, D Cer, N Arivazhagan, W Wang - arXiv preprint arXiv:2007.01852, 2020

… The sentences are filtered using a sentence 1https://commoncrawl.org/ 2https://www.wikipedia.org/ 3Long lines are usually JavaScript or attempts at SEO … Finally, we pretrain on common crawl which is much larger, albeit …


Large Scale Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training

O Agarwal, H Ge, S Shakeri, R Al-Rfou - arXiv preprint arXiv:2010.12688, 2020

… 3We use only one annotator at the moment but are expanding this evaluation to multiple annotators. Page 7. such as Wikipedia or common crawl. KGs are a rich source of factual information that can serve as additional succint informatopm …


Large-Scale Analysis of HTTP Response Headers

… The data come from the Common Crawl's monthly web crawls that collect responses from what we can consider to be the entire internet … The data come from the Common Crawl's monthly web crawls that collect responses …


M Li, H Bai, L Tan, K Xiong, J Lin - arXiv preprint arXiv:2010.11351, 2020

Page 1. Latte-Mix: Measuring Sentence Semantic Similarity with Latent Categorical Mixtures Minghan Li*1, 2 , He Bai1, 3 , Luchen Tan1 , Kun Xiong1 , Ming Li1, 3 , Jimmy Lin1, 3 1RSVP.ai, 2University of Toronto, 3David R. Cheriton …


LEAPME: Learning-based Property Matching with Embeddings

D Ayala Hernández, IC Hernández Salmerón… - ArXiv. org, arXiv …, 2020

D Ayala, I Hernández, D Ruiz, E Rahm - arXiv preprint arXiv:2010.01951, 2020

… To compute embeddings, we use the pre-trained GloVe approach [43]1, specifically for the uncased Common Crawl corpus that includes 300-dimensional vectors for 1.9 million words, promising a good coverage …


Learning Accurate Integer Transformer Machine-Translation Models

E Wu - arXiv preprint arXiv:2001.00926, 2020

… sor2Tensor v1.12 English-to-German translation task (translate_ende_wmt32k_packed). This dataset has 4.6 million sentence pairs drawn from three WMT18 [Bojar et al., 2018a] parallel corpora: News Commentary V13, Europarl V7, and Common Crawl …


Learning and Evaluating Emotion Lexicons for 91 Languages

S Buechel, S Rücker, U Hahn - arXiv preprint arXiv:2005.05672, 2020

… We use the fastText embedding models from Grave et al. (2018) trained for 157 languages on the respective WIKIPEDIA and the respective part of COMMONCRAWL. These resources not only greatly facilitate our work …


A Adhikari, X Yuan, MA Côté, M Zelinka, MA Rondeau… - arXiv preprint arXiv …, 2020

Page 1. Learning Dynamic Knowledge Graphs to Generalize on Text-Based Games Ashutosh Adhikari * 1 Xingdi Yuan * 2 Marc-Alexandre Côté * 2 Mikuláš Zelinka 3 Marc-Antoine Rondeau 2 Romain Laroche 2 Pascal Poupart …


Learning Engineering Properties with Bag-of-Tricks. For the Automated Evaluation of a Piping Design

WC Tan, KH Chua, CB Yan, IM Chen - 2020 IEEE 16th International Conference on …, 2020

… and T. Mikolov, “Learning word vectors for 157 languages,” in Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018), 2018. [28] [Online]. Available: https://commoncrawl.org/ 1280


LEARNING FROM MULTIMODAL WEB DATA

… (2018); Roemmele et al. (2011); De Marneffe et al. (2019); Clark et al. (2019). 7https://commoncrawl.org/ 5 Page 22. model; they achieve high performance on several video understanding tasks (eg, retrieval), and …


Learning Geometric Word Meta-Embeddings

P Jawanpuria, NTV Dev, A Kunchukuttan, B Mishra - arXiv preprint arXiv:2004.09219, 2020

… GloVe (Pennington et al., 2014): has 1 917 494 word embeddings trained on 42B tokens of web data from the common crawl. • fastText (Bojanowski et al., 2017): has 2 000 000 word embeddings trained on common crawl …


Page 1. Learning hierarchical relationships for object-goal navigation Yiding Qiu ∗ UC San Diego yiqiu@eng.ucsd.edu Anwesan Pal ∗ UC San Diego a2pal@eng.ucsd.edu Henrik I. Christensen UC San Diego hichristensen@eng.ucsd.edu …


Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task

T Sellam, A Pu, HW Chung, S Gehrmann, Q Tan… - arXiv preprint arXiv …, 2020

… Details of MBERT-WMT pre-training We trained MBERT-WMT model with an MLM loss (Devlin et al., 2019), using a combination of public datasets: Wikipedia, the WMT 2019 News Crawl (Barrault et al.), the C4 variant of Com …


D Fried, JB Alayrac, P Blunsom, C Dyer, S Clark… - arXiv preprint arXiv …, 2020

Page 1. Learning to Segment Actions from Observation and Narration Daniel Fried‡ Jean-Baptiste Alayrac† Phil Blunsom† Chris Dyer† Stephen Clark† Aida Nematzadeh† †DeepMind, London, UK ‡Computer Science Division, UC Berkeley …


N Stiennon, L Ouyang, J Wu, DM Ziegler, R Lowe… - arXiv preprint arXiv …, 2020

Page 1. Learning to summarize from human feedback Nisan Stiennon∗ Long Ouyang∗ Jeff Wu∗ Daniel M. Ziegler∗ Ryan Lowe∗ Chelsea Voss∗ Alec Radford Dario Amodei Paul Christiano∗ OpenAI Abstract As language …


Learning unbiased zero-shot semantic segmentation networks via transductive transfer

H Liu, Y Wang, J Zhao, G Yang, F Lv - arXiv preprint arXiv:2007.00515, 2020

… IV. EXPERIMENTS Following [7], we use the concatenation of two different word vectors, ie word2vec trained on Google News [12] and fastText trained on Common Crawl [12], to construct the semantic space shared by source and target classes …


Learning User Representations for Open Vocabulary Image Hashtag Prediction

T Durand - Proceedings of the IEEE/CVF Conference on Computer …, 2020

… We train our model using ADAM [26] during 20 epochs with a start learning rate 5e-5. We use ResNet-50 [22] as the ConvNet and GloVe embeddings [34] as pre-trained word em- beddings. GloVe was trained on Common …


Learning Word and Sub-word Vectors for Amharic (Less Resourced Language)

… The large text collection from Wikipedia and common crawl are commonly used data source to train and learn word vectors (Al-Rfou et al., 2013; Bojanowski et al … (2014) released GloVe models trained on Wikipedia …


Leveraging ParsBERT and Pretrained mT5 for Persian Abstractive Text Summarization

M Farahani, M Gharachorloo, M Manthouri - arXiv preprint arXiv:2012.11204, 2020

… T5, on the other hand, is a unified Seq2Seq framework that employs Text-to- Text format to address NLP text-based problems. A multilingual variation of the T5 model is called mT5 [16] that covers 101 different …


Leveraging Structured Metadata for Improving Question Answering on the Web

X Du, A Hassan, A Fourney, R Sim, P Bennett… - … of the 1st Conference of the …, 2020

… website content. The Web Data Commons project (Mühleisen and Bizer, 2012) estimates that 0.9 billion HTML pages out of the 2.5 billion pages (37.1%) in the Common Crawl web corpus1 contain structured metadata. Figure …


LIG-Health at Adhoc and Spoken IR Consumer Health Search: expanding queries using UMLS and FastText.

P Mulhem, GG Saez, A Mannion, D Schwab, J Frej - Conference and Labs of the …, 2020

… The FastText embedding vector of a word is the sum of the vectors of its component ngrams. We used the pre-trained word vectors for English language, trained on Common Crawl and Wikipedia using FastText. The features of the model used are as follows; …


LIMSI@ WMT 2020

SA Rauf, JC Rosales, I Paris, PM Quang, S Paris…

… Domain Corpus sents. words words (en) (de) web Paracrawl 50,875 978 919 economy Tilde EESC 2,858 61 58 news Commoncrawl 2,399 51 47 Tilde rapid 940 20 19 News commentary 361 8 8 tourism Tilde tourism 7 …


Linguistic Structure Guided Context Modeling for Referring Image Segmentation

T Hui, S Liu, S Huang, G Li, S Yu, F Zhang, J Han

… rate. CNN is fixed during training. We use batch size 1 and stop training after 700K iterations. GloVe word embeddings [30] pretrained on Common Crawl with 840B tokens are used to replace randomly initialized ones. For fair …


Linguistically-aware Attention for Reducing the Semantic-Gap in Vision-Language Tasks

G KV, A Nambiar, KS Srinivas, A Mittal - arXiv preprint arXiv:2008.08012, 2020

… The pre-trained word-to-vector networks such as Glove [29] and Bert [30] are inexpensive and rich in making linguistic correlations (since they are already trained on a large textual corpus such as Common Crawl and Wikipedia2014) …


LNMap: Departures from Isomorphic Assumption in Bilingual Lexicon Induction Through Non-Linear Mapping in Latent Space

T Mohiuddin, MS Bari, S Joty - arXiv preprint arXiv:2004.13889, 2020

… English, Italian, and German em- beddings were trained on WacKy crawling corpora using CBOW (Mikolov et al., 2013b), while Spanish and Finnish embeddings were trained on WMT News Crawl and Common Crawl, respectively. 4.2 Baseline Methods …


M Moradshahi, G Campagna, SJ Semnani, S Xu… - arXiv preprint arXiv …, 2020

Page 1. Localizing Open-Ontology QA Semantic Parsers in a Day Using Machine Translation Mehrad Moradshahi Giovanni Campagna Sina J. Semnani Silei Xu Monica S. Lam Computer Science Department Stanford University …


Localizing Q&A Semantic Parsers for Any Language in a Day

M Moradshahi, G Campagna, S Semnani, S Xu, M Lam - Proceedings of the 2020 …, 2020

Page 1. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5970–5983, November 16–20, 2020. c 2020 Association for Computational Linguistics 5970 Localizing Open-Ontology …


Locally Constructing Product Taxonomies from Scratch Using Representation Learning

… The WDC schema.org project, which relies on the webpages in the Common Crawl, is able to automatically extract schema.org data from webpages due to its unique syntax and make it available as a dataset in Resource Description Framework (RDF) …


LOREM: Language-consistent Open Relation Extraction from Unstructured Text

… Since these sentences are automatically tagged, we do expect a higher noise level than in the manually tagged test sets. For the language-individual model, we use FastText word em- beddings [11] which are trained on Common Crawl and Wikipedia dataset …


Lost in Embedding Space: Explaining Cross-Lingual Task Performance with Eigenvalue Divergence

H Dubossarsky, I Vulić, R Reichart, A Korhonen - arXiv preprint arXiv:2001.11136, 2020

… We prefer Wikipedia as the main embedding training corpus over the larger Common Crawl corpus, because the text in Wikipedia is much cleaner or even hand-curated, and adheres to the rules of standard language (Grave et al., 2018) …


Low-Resource Knowledge-Grounded Dialogue Generation

X Zhao, W Wu, C Tao, C Xu, D Zhao, R Yan - arXiv preprint arXiv:2002.10348, 2020

Page 1. Published as a conference paper at ICLR 2020 LOW-RESOURCE KNOWLEDGE-GROUNDED DIALOGUE GENERATION Xueliang Zhao1,2, Wei Wu3, Chongyang Tao1, Can Xu3, Dongyan Zhao1,2, Rui Yan1,2,4∗ 1Wangxuan …


Low-Resource Text Classification via Cross-lingual Language Model Fine-tuning

… XLM - R shows the possibility of training one model for many languages while not sacrificing per-language performance. It is trained on 2.5TB of CommonCrawl data, in 100 languages and uses a large vocabulary size of CCL 2020 …


Page 1. LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020 8th Workshop on Challenges in the Management of Large Corpora (CMLC-8) PROCEEDINGS Editors: Piotr Ba´nski, Adrien Barbaresi, Simon Clematide …


Machine Bias and Fundamental Rights

D Amilevičius - Smart Technologies and Fundamental Rights, 2020

Jump to Content Jump to Main Navigation. English; 中文; français; Deutsch. Access via: Google Googlebot - Web Crawler SEO. Login to my Brill account Create Brill Account. Publications. Subjects. African Studies American Studies …


Machine Translation for English–Inuktitut with Segmentation, Data Acquisition and Pre-Training

C Roest, L Edman, G Minnema, K Kelly, J Spenader… - Proceedings of the Fifth …, 2020

… 2), we train XLM models with the News Crawl data for English and Common Crawl data for Inuktitut, as specified in Table 2. We also use Hansards and Newsdevtrain oversampled 5 times for parallel data. We try both tagging …


Machine Translation in Natural Language Processing by Implementing Artificial Neural Network Modelling Techniques: An Analysis

FA Khan, A Abubakar - International Journal on Perceptive and Cognitive …, 2020

… The experiment for the proposed model has followed with BERT, to BookCorpus [35] and Wikipedia for English as an initializing point for pretraining. Similarly, other text includes, Giga5 (16Gb) ClueWeb 2012-B and Common Crawl respectively …


Machine Translation Reference-less Evaluation using YiSi-2 with Bilingual Mappings of Massive Multilingual Language Model

C Lo, S Larkin - Proceedings of the Fifth Conference on Machine …, 2020

… The differencesbetweenXLM-RandBERTare1)XLM-Ris trained on the CommonCrawl corpus which is significantly larger than the Wikipedia training data used by BERT; 2) instead of a uniform data sampling rate used in BERT …


Machine Translation System Selection from Bandit Feedback

J Naradowsky, X Zhang, K Duh - arXiv preprint arXiv:2002.09646, 2020

… Specifically, we in- clude OpenSubtitles2018 [Lison and Tiedemann, 2016] and WMT 2017 [Bojar et al., 2017], which contains data from eg parliamentary proceedings (Europarl, UN), political/economic news, and web-crawled parallel corpus (Common Crawl) …


MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer

J Pfeiffer, I Vulić, I Gurevych, S Ruder - arXiv preprint arXiv:2005.00052, 2020

… It is a Transformer-based model pretrained for one hundred languages on large cleaned Common Crawl corpora (Wenzek et al., 2019). For efficiency purposes, we use the XLM-R Base configuration as the basis for all of our experiments …


N Reimers, I Gurevych - arXiv preprint arXiv:2004.09813, 2020

Page 1. Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation Nils Reimers and Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP-TUDA) Department of Computer Science …


MantisTable SE: an Efficient Approach for the Semantic Table Interpretation

M Cremaschi, R Avogadro, A Barazzetti, D Chieregato - Semantic Web Challenge on …, 2020

… Web Tables: The WebTables system [1] extracts 14.1 billion HTML tables and finds 154 million are high-quality tables (1.1%); – Web Tables: Lehmberg et al. [5] extract 233 million content tables from Common Crawl 2015 (2.25 …


Mapping crime descriptions to law articles using deep learning

M Vink, N Netten, MS Bargh, S van den Braak… - Proceedings of the 13th …, 2020

… This is a popular word embedding created by Facebook and available in many languages. The FastText embeddings are trained on Wikipedia texts and the data from the common crawl project. The word embeddings have a vector size of 300 …


Mapping Languages: The Corpus of Global Language Use

… language) and 156 countries (again with over 1 million words from each country), all distilled from Common Crawl web data … region: (i) the number of sites indexed by the Common Crawl; (ii) the population's degree of access …


Mapping the market for remanufacturing: An application of “Big Data” analytics

JQF Netoa, M Dutordoira - International Journal of Production Economics, 2020

… The vectors are created with Global Vectors for Word Representation (GloVe), one of the most well-known word embedding methods (Pennington et al., 2014), and are based on a data set obtained from Common Crawl, a nonprofit …


MASK: A flexible framework to facilitate de-identification of clinical texts

N Milosevic, G Kalappa, H Dadafarin, M Azimaee… - arXiv preprint arXiv …, 2020

… The first of these ap- proaches, used GLoVe (Global Vector) word embeddings [14]. We used GLoVe embeddings trained on common crawl data containing 840 billion tokens, 2.2 million unique tokens in vocabulary, and 300dimensional vectors …


Masked ELMo: An evolution of ELMo towards fully contextual RNN language models

G Senay, E Salin - arXiv preprint arXiv:2010.04302, 2020

… It should be noted that ELMo 5.5B is trained on a larger corpus than ELMo and Masked ELMo (Wikipedia: 1.9B and the common crawl from WMT 2008-2012: 3.6B). Moreover, a BERT baseline (BERT*) trained on the same …


Massive vs. Curated Embeddings for Low-Resourced Languages: the Case of Yorùbá and Twi

J Alabi, K Amponsah-Kaakyire, D Adelani… - Proceedings of The 12th …, 2020

… The resource par excellence is Wikipedia2, an online encyclopedia currently available in 307 languages3. Other initiatives such as Common Crawl4 or the Jehovahs Witnesses site5 are also repositories for multilingual …


Massively Multilingual Document Alignment with Cross-lingual Sentence-Mover's Distance

A El-Kishky, F Guzmán - arXiv preprint arXiv:2002.00761, 2020

… selected for evaluation. Baseline Methods. For comparison, we implemented two existing and intuitive document scoring baselines previously evaluated on this URL-Aligned CommonCrawl dataset [11]. The first method dubbed …


Page 1. Master Thesis: Developing a Cross-Lingual Named Entity Recognition Model Jowita Podolak1, Philine Zeinert2 1jopo@itu.dk 2phze@itu.dk June 1, 2020 Course Code: KISPECI1SE Page 2. Abstract To build a Cross …


Page 1. IN DEGREE PROJECT COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS , STOCKHOLM SWEDEN 2020 Matching Job Applicants to Free Text Job Ads Using Semantic Networks and Natural Language Inference ANTON THUN …


MDD@ AMI: Vanilla Classifiers for Misogyny Identification

S El Abassi, S Nisioi - Proceedings of Sixth Evaluation Campaign of Natural …, 2020

… CBOW embeddings pre-trained on Wikipedia and OSCAR (Common Crawl)3. The second run is trained on English glove em- beddings that surprisingly contain the representation of more than half of our Italian …


Page 1. Running Head: MEASURING ORIGINALITY 1 MEASURING DIVERGENT THINKING ORIGINALITY WITH HUMAN RATERS AND TEXT-MINING MODELS: A PSYCHOMETRIC COMPARISON OF METHODS …


Measuring prominence of scientific work in online news as a proxy for impact

J Ravenscroft, A Clare, M Liakata - arXiv preprint arXiv:2007.14454, 2020

… In our task we employ pre-trained GloVe4 feature embeddings trained on the Common Crawl dataset5, a multi-petabyte archive of content scraped from the world wide web containing 42 billion tokens and a vocabulary 1.9 million words …


Media-Analytics. org: A Resource to Research Language Usage by News Media Outlets

… news media outlets. News articles textual content are available in outlet-specific domains and Internet cache repositories such as the Internet Archive Wayback Machine, Google Cache and Common Crawl. Articles' headlines …


Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?

… Page 3. CommonCrawl subcorpus … We now describe how Carol prepares the data for Alice and Bob. First, Carol se- lects 4 subcorpora for the training data of Al- ice, namely CommonCrawl, Europarl v7, News Commentary v13, and Rapid 2016 …