DinoStackAI/qasper-rag huggingface dataset:
(infixes 700K, 318K, and 40K) each representing the number of approximate parameters
We evaluate the false-reject (FR) and false-accept (FA) tradeoff across several end-to-end models of distinct sizes and computational complexities.
Our baseline system (Baseline_1850K) is taken from BIBREF13 .
applying the rule INLINEFORM4 to a set of natural language questions INLINEFORM5 both models are improved following the back-translation protocol that target sequences should follow the real data distribution
WikiSQL - 2 rules (SELECT, WHERE)
SimpleQuestions - 1 rule
SequentialQA - 3 rules (SELECT, WHERE, COPY)
WikiSQL SimpleQuestions SequentialQA
10k training and 1k test 1,101 sentences (26k tokens)
Bi-LSTM: For low resource <17k clean data: Using distant supervision resulted in huge boost of F1 score (1k eg. ~9 to ~36 wit distant supervision)
BERT: <5k clean data boost of F1 (1k eg. ~32 to ~47 with distant supervision)
Bi-LSTM BERT
Nigeria Benin, Ghana, Cameroon, Togo, Côte d'Ivoire, Chad, Burkina Faso, and Sudan Republic of Togo, Ghana, Côte d'Ivoire, Sierra Leone, Cuba and Brazil
Relevance and Subject categorizations are annotated at a percent agreement of $0.71$ and $0.70$, their agreement scores are only fair, at $\alpha =0.27$ and $\alpha =0.29$ Stance and Sentiment, which carry more categories than the former two, is $0.54$ for both. Their agreement scores are also fair, at $\alpha =0.35$ and $\alpha =0.34$ This holds for the Relevant category ($0.81$), the Vaccine category ($0.79$) and the Positive category ($0.64$) The Negative category yields a mutual F-score of $0.42$, which is higher than the more frequently annotated categories Neutral ($0.23$) and Not clear ($0.31$).
27,534 messages
word unigrams, bigrams, and trigrams
Evaluation datasets used:
CMRC 2018 - 18939 questions, 10 answers
DRCD - 33953 questions, 5 answers
NIST MT02/03/04/05/06/08 Chinese-English - Not specified

Source language train data:
SQuAD - Not specified
Microsoft Research dataset containing movie, taxi and restaurant domains.
It has three sequentially connected units to output continue, act and slots generating multi-acts in a doble recurrent manner.
CoNLL-2012 shared task BIBREF21 corpus
counts of predicate-argument tuples from English Wikipedia
attention parsing
Cora Hepth Zhihu
MMB DeepWalk LINE Node2vec TADW CENE CANE WANE DMTE
Test set 1 contained 57 drug labels and 8208 sentences and test set 2 contained 66 drug labels and 4224 sentences
Training-22 NLM-180
Twitter, Yelp reviews and movie reviews
Lurking Horror Afflicted Anchorhead 9:05 TextWorld games
the knowledge graph is used to prune this space by ranking actions based on their presence in the current knowledge graph and the relations between the objects in the graph as in BIBREF7
the baseline provided by BIBREF8 the baselines provided by the ABSA organizers
ABSA SemEval 2014-2016 datasets
Yelp Academic Dataset
Wikipedia dumps
Dutch French Russian Spanish Turkish English
Local, shallow features based mostly on orthographic, word shape and n-gram features plus their context
Penn Treebank Text8
using multiple pivot sentences
The second method it to learn a common space for the two modalities before concatenation (project) The first method is concatenation of the text and image representation (concat)
largest improvement ($22-26\%$ E.R) when text-based unsupervised models are combined with image representations
similarity is computed as the cosine of the produced $h_{L}$ and $h_{R}$ sentence/image representations
1338 pairs for training
functional dissimilarity score nearest neighbours experiment
The dataset was created by using translations provided by Tatoeba and OpenSubtitles BIBREF16.
For the Oshiete-goo dataset, the NAGM model's ROUGE-L score is higher than the highest performing conventional model, Trans, by 0.021, and its BLEU-4 score is higher than the highest performing model CLSTM by 0.037.  For the nfL6 dataset, the NAGM model's ROUGE-L score is higher than the highest performing conventional model, CLSTM, by 0.028, and its BLEU-4 score is higher than the highest performing model CLSTM by 0.040. Human evaluation of the NAGM's generated outputs for the Oshiete-goo dataset had 47% ratings of (1), the highest rating, while CLSTM only received 21% ratings of (1). For the nfL6 dataset, the comparison of (1)'s was NAGM's 50% to CLSTM's 30%.
our Open model achieves more than 3 points of f1-score than the state-of-the-art result
In closed setting 84.22 F1 and in open 87.35 F1.
dependency head and dependency relation label, denoted as Dep and Rel for short Tree-based Position Feature (TPF) as Dependency Path (DepPath) Shortest Dependency Path (SDP) as Relation Path (RelPath)
YouTube ASR system
2000 images
the UBC database BIBREF14
over 41,250 videos and 825,000 captions in both English and Chinese. over 206,000 English-Chinese parallel translation pairs
constructively by selecting the parameters of the multi-head self-attention layer so that the latter acts like a convolutional layer
attention probabilities learned tend to respect the conditions of Lemma UNKREF15, corroborating our hypothesis validate that our model learns a meaningful classifier we compare it to the standard ResNet18
Brent corpus PTB Beijing University Corpus Penn Chinese Treebank
movies restaurants English Korean
Qualitatively through efficiency, effectiveness and satisfaction aspects and quantitatively through metrics such as precision, recall, accuracy, BLEU score and even human judgement.
EMPATHETICDIALOGUES dataset a dataset containing 1.5 million Twitter conversation, gathered by using Twitter API from customer care account of 62 brands across several industries SEMAINE corpus BIBREF30
how to incorporate affective information into chatbots, what are resources that available and can be used to build EAC, and how to evaluate EAC performance
we report P@1, which is equivalent to accuracy we also provide results with P@5 and P@10 in the Appendix
Answer with content missing: (Chapter 3) The concept can be easily explained with an example, visualized in Figure 1. Consider the Portuguese (Pt) word trabalho which, according to the MUSE Pt–En dictionary, has the words job and work as possible En translations. In turn, these two En words can be translated to 4 and 5 Czech (Cs) words respectively. By utilizing the transitive property (which translation should exhibit) we can identify the set of 7 possible Cs translations for the Pt word trabalho.
relational entities general text-based attributes descriptive text of images nodes in graph structure of networks queries
BABEL speech corpus
Train languages are: Cantonese, Bengali, Pashto, Turkish, Vietnamese, Haitian, Tamil, Kurdish, Tokpisin and Georgian, while Assamese, Tagalog, Swahili, Lao are used as target languages.
VGG-BLSTM character-level RNNLM
the image can play the role of a pivot “language" to bridge the two languages without paralleled corpus
Base Base+Noise Cleaning Dynamic-CM Global-CM Global-ID-CM Brown-CM K-Means-CM
They  evaluate newly proposed models in several low-resource settings across different languages with real, distantly supervised data with non-synthetic noise
logistic regression (LR), recurrent neural network (RNN) BIBREF35, convolutional neural network (CNN) BIBREF36 and Google BERT BIBREF37
the generated sentences often did not include all desired attributes.
Overall, our AMRAN outperforms all baselines, achieving 0.657 HR@10 and 0.410 NDCG@10.
Twitter dataset obtained from the authors of BIBREF12
Perplexity score 142.84 on dev and 138.91 on test
Mandarin English
Parallel monolingual corpus in English and Mandarin
VATEX, WMT 2014 English-to-German, and VQA-v2 datasets
our method still can improve the state-of-the-art accuracy BIBREF7 from 60.32% to 60.34%
LASSO optimization problem
the basic question generation module (Module 1) and co-attention visual question answering module (Module 2)
biaffine attention BIBREF14
level of agreement (Krippendorff's INLINEFORM0 )
participants in group one very rarely gave different answers to questions one and two (18 of 500 instances or 3.6%)
Twitter definition of hateful conduct
Personal thought of the annotator.
10 hashtags that can be used in an insulting or offensive way
Word embeddings trained on GoogleNews and Word embeddings trained on Reddit dataset
rule-based and dictionary-based methods
Entity identification with offset mapping and concept indexing
LSTM-LM
They randomly sample sentences from Wikipedia that contains an object RC and add them to training data
Lexicon of the cities tend to use most forms of a particular concept
It uses particular forms of a concept rather than all of them uniformly
simple linear model with rank constraint Hierarchical softmax N-gram features
Overall accuracy per model is: 5-gram (60.5), LSTM (68.9), TXL (68.7), GPT-2 (80.1)
The data generation scripts use a basic template to create each paradigm, pulling from a vocabulary of over 3000 words annotated for morphological, syntactic, and semantic features needed to create grammatical and semantically felicitous sentences.
GloVe concatenation of average embeddings calculated separately for the interview and for the medical examination
k-means hierarchical clustering with Ward's method for merging clusters BIBREF23
estimated test accuracy is highly correlated with actual test accuracy for various datasets appropriateness of the proposed function for modeling the complex error landscape
bag of words, tf-idf, bag-of-means
baseline for each sampled caption is defined as the average reward of the rest samples
Aristo Corpus
Regents 4th
Regents 8th
Regents `12th
ARC-Easy
ARC-challenge
local businesses (i.e. restaurants)
RS-Average RS-Linear RS-Item RS-MF Sum-Opinosis Sum-LSTM-Att
manually reviewed
manually inspect
concatenating to the embedding vector
clear mapping between plans and text helps to reduce these issues greatly, the system in BIBREF0 still has 2% errors work in neural text generation and summarization attempt to address these issues
plan-to-DFS mapping to perform the correct sequence of traversals, and train a neural classifier to act as a controller
we suggest to estimate the interpretability of $k$ th component as $ \operatorname{interp}_k W = \sum _{i,j=1}^N W_{i,k} W_{j,k} \left(W_i \cdot W_j \right). $
original models were better in some tasks (CR, MPQA, MRPC) utilizing self-attentive sentence representation further improves performances in 5 out of 8 tasks
(Layer Normalized Skip-Thoughts, ST-LN) BIBREF31 Cap2All, Cap2Cap, Cap2Img
Karpathy and Fei-Fei's split for MS-COCO dataset BIBREF10
- Font & Keyboard
- Speech-to-Text
- Text-to-Speech
- Text Prediction
- Spell Checker
- Grammar Checker
- Text Search
- Machine Translation
- Voice to Text Search
- Voice to Speech Search
the extent to which the text obtained from the two platforms of Yahoo! Answers and Twitter reflect the true attributes of neighbourhoods
ROUGE F1 METEOR
The reciprocal of the variance of the attention distribution
median cosine similarity
a reliability study for the proposed scheme
Our annotation scheme introduces opportunities for the educational community to conduct further research Once automated classifiers are developed, such relations between talk and learning can be examined at scale automatic labeling via a standard coding scheme can support the generalization of findings across studies, and potentially lead to automated tools for teachers and students collecting and annotating corpora that can be used by the NLP community to advance the field in this particular area
Measuring three aspects: argumentation, specificity and knowledge domain.
2008 Punyakanok et al. 
2009 Zhao et al. + ME 
2008 Toutanova et al. 
2010 Bjorkelund et al.  
2015 FitzGerald et al. 
2015 Zhou and Xu 
2016 Roth and Lapata 
2017 He et al. 
2017 Marcheggiani et al.
2017 Marcheggiani and Titov 
2018 Tan et al. 
2018 He et al. 
2018 Strubell et al. 
2018 Cai et al. 
2018 He et al. 
2018 Li et al.
LSTM and BERT
$\textsc {Lead-X}$ $\textsc {PTGen}$ $\textsc {DRM}$ $\textsc {TConvS2S}$ $\textsc {BottomUp}$ ABS DRGD SEQ$^3$ BottleSum GPT-2
ROUGE-1, ROUGE-2 and ROUGE-L F-measure ROUGE on XSUM and CNN/DailyMail, and use limited-length recall-measure ROUGE on NYT and DUC
hree years of online news articles from June 2016 to June 2019
many news articles begin with reporter names, media agencies, dates or other contents irrelevant to the content to ensure that the summary is concise and the article contains enough salient information, we only keep articles with 10-150 words in the top three sentences and 150-1200 words in the rest, and that contain at least 6 sentences in total we try to remove articles whose top three sentences may not form a relevant summary
three years of online news articles from June 2016 to June 2019
increases F1-score by 10.2% and 3% compared with the existing best systems $LS \cup KLD \cup CONN$ and $KLD \cup LS \cup LS_{inter}$
TextCNN, TextRNN, SASE, DPCNN, and BERT $LS \cup KLD \cup CONN$ and $KLD \cup LS \cup LS_{inter}$ are the best systems with the highest recall and F1-score respectively
we integrate object-pairs with global representation and make a pair-wise inference to detect the relationship among the segments
in this dataset, the input query is the concatenation of a Wikipedia article title with the title of one of its section. The relevant passages are the paragraphs within that section
we use the context of the word to predict its label and by doing so our model learns label-aware context for each word in the sentence
micro-averaged F1 score
improves the F1 score by almost 2%, which corresponds to a 12.3% error rate reduction
Adobe internal NLU tool Pytext Rasa
Dataset has 1737 train, 497 dev and 559 test sentences.
conditional random field model Seq2Seq attention model
some configurations used in some architectures (e.g., additional RNN layers) are actually irrelevant
pre-trained word embeddings BIBREF11, BIBREF12 recurrent BIBREF13 transformer-based sentence encoders BIBREF14 distinct convolutional neural networks standard fusion strategies two main attention mechanisms BIBREF18, BIBREF19
pre-trained text representations transformer-based encoders together with GRU models attention mechanisms are paramount for learning top performing networks Top-Down is the preferred attention method
MT metrics Readability metrics and other sentence-level features Metrics based on the baseline QuEst features Metrics based on other features
a BiLSTM without attention (BiLSTM) as well as a single forward-LSTM layer with attention (LSTM+att) and without attention (LSTM) baselines are defined by BIBREF32 who already proposed an LSTM-based architecture that only uses non-temporal features, and the SVM-based estimation model as originally used for reward estimation by BIBREF24
Hierarchical Disentangled Self-Attention
LEAD QUERY_SIM MultiMR SVR DocEmb ISOLATION
Tesla and Ford are investigated on how Twitter sentiment could impact the stock price
linear logistic regression to a set of stock technical signals
can be biased by dataset used and may generate categories which are suboptimal compared to human designed categories
it is less expensive and quantifies interpretability using continuous values rather than binary evaluations
hand crafted by users
conceptualization task
By computing number of unique responses and number of responses divided by the number of unique responses to that question for each of the questions
comparing the summary with the text instead of the reference and labeling the candidate bad if it is incorrect or irrelevant
(1) Good (3 points): The response is grammatical, semantically relevant to the query, and more importantly informative and interesting (2) Acceptable (2 points): The response is grammatical, semantically relevant to the query, but too trivial or generic (3) Failed (1 point): The response has grammar mistakes or irrelevant to the query
BLEU Distinct-1 & distinct-2
Seq2seq CVAE Hierarchical Gated Fusion Unit (HGFU) Mechanism-Aware Neural Machine (MANM)
we connect each latent variable with a word in the vocabulary, thus each latent variable has an exact semantic meaning.
collection of headlines published by HuffPost BIBREF12 between 2012 and 2018
number of relevant output words as a function of the headline’s category label
Wikipedea Corpus and BooksCorpus
EN, JA, ES, AR, PT, KO, TH, FR, TR, RU, IT, DE, PL, NL, EL, SV, FA, VI, FI, CS, UK, HI, DA, HU, NO, RO, SR, LV, BG, UR, TA, MR, BN, IN, KN, ET, SL, GU, CY, ZH, CKB, IS, LT, ML, SI, IW, NE, KM, MY, TL, KA, BO
langid.py library encoder-decoder EquiLID system GRU neural network LanideNN system CLD2 CLD3
Bantu
French-Mboshi 5K corpus
precision, recall, and F-measure on boundaries (BP, BR, BF), and tokens (WP, WR, WF) exact-match (X) metric
model overall still gives 1.0% higher average UAS and LAS than the previous best parser, BIAF our model reports more than 1.0% higher average UAS than STACKPTR and 0.3% higher than BIAF
Proposed vs best baseline:
Decoding: 8541 vs 8532 tokens/sec
Training: 8h vs 8h
WMT18 EnDe bitext WMT16 EnRo bitext WMT15 EnFr bitext We perform our experiments on WMT18 EnDe bitext, WMT16 EnRo bitext, and WMT15 EnFr bitext respectively. We use WMT Newscrawl for monolingual data (2007-2017 for De, 2016 for Ro, 2007-2013 for En, and 2007-2014 for Fr). For bitext, we filter out empty sentences and sentences longer than 250 subwords. We remove pairs whose whitespace-tokenized length ratio is greater than 2. This results in about 5.0M pairs for EnDe, and 0.6M pairs for EnRo. We do not filter the EnFr bitext, resulting in 41M sentence pairs.
14 volunteers
GloVe embeddings trained by BIBREF10 on Wikipedia and Gigaword 5 (vocab: 400K, dim: 300) w2v-gn, Word2vec BIBREF5 trained on the Google News dataset (vocab: 3M, dim: 300) DeepWalk node2vec
To verify the reliability of the transformed semantic space, we propose an evaluation benchmark on the basis of word similarity datasets. Given an enriched space INLINEFORM0 and a similarity dataset INLINEFORM1 , we compute the similarity of each word pair INLINEFORM2 as the cosine similarity of their corresponding transformed vectors INLINEFORM3 and INLINEFORM4 from the two spaces, where INLINEFORM5 and INLINEFORM6 for LS and INLINEFORM7 and INLINEFORM8 for CCA.
judged by 10 raters on a [0,10] scale
WikiText-TL-39
3,206
Online sites tagged as fake news site by Verafiles and NUJP and news website in the Philippines, including Pilipino Star Ngayon, Abante, and Bandera
Siamese neural network consisting of an embedding layer, a LSTM layer and a feed-forward layer with ReLU activations
unsupervised morphological analyzer capable of using lattices improved the accuracy of phoneme recognition and word segmentation Consequently, this result suggests that this word segmentation method considers the multiple hypothesis of speech recognition as a whole and reduces uncertainty such as variability in recognition
PAR score
unsupervised word segmentation method latticelm
a BiLSTM over all words in the respective sequences with randomly initialised word embeddings, following BIBREF30
monolingual
Comparing BLEU score of model with and without attention
Calinski-Harabasz score t-SNE UMAP
not researched as much as English
3600 user-generated comments
the author and the supervisor
Matching features from matching sentences from various perspectives.
RSS feeds in French on a daily basis
multinomial logistic regression
WeedsPrec BIBREF8 invCL BIBREF11 SLQS model cosine similarity
noun-noun subset of bless leds BIBREF13 bless wbless bibless hyperlex BIBREF20
Detection Direction Graded Entailment
(i) zero NER-specific BiRNN layers, (ii) zero RE-specific BiRNN layers, or (iii) zero task-specific BiRNN layers of any kind
The basic idea of the visualization, drawing on Isaac Newton’s visualization of the color spectrum BIBREF8 , is to express a mixture in terms of its constituents as represented in barycentric coordinates.
DSTC2 M2M-sim-M M2M-sim-R
attention probes using visualizations of the activations created by different pieces of text
dependency relation between two words word sense
Bulgarian, Czech, French, German, Korean, Polish, Portuguese, Russian, Thai, Vietnamese South African English These features are typically obtained by training a deep neural network jointly on several languages for which labelled data is available. The final shared layer often has a lower dimensionality than the input layer, and is therefore referred to as a `bottleneck'.
Bulgarian, Czech, French, German, Korean, Polish, Portuguese, Russian, Thai, Vietnamese
model is trained to predict language IDs as well as the subwords we add language IDs in the CS point of transcriptio
merging, concatenating, or averaging the entity and its features to compute its embeddings graph embedding approaches matrix factorization to jointly embed KB and textual relations
BIBREF7 BIBREF26
S2VT, RGB (VGG), RGB (VGG)+Flow (AlexNet), LSTM-E (VGG), LSTM-E (C3D) and Yao et al.
Viet Treebank corpus for POS tagging and chunking tasks, and on VLSP shared task 2016 corpus for NER task
Precision, Recall and INLINEFORM0 score
Burckhardt et al. BIBREF22 Liu et al. BIBREF18 Dernoncourt et al. BIBREF9 Yang et al. BIBREF10
2014 i2b2 de-identification challenge data set BIBREF2 nursing notes corpus BIBREF3
using mean opinion score (MOS) naturalness judgments produced by a crowd-sourced pool of raters
The Wikipedia Toxic Comments dataset
Yes, Transformer based seq2seq is evaluated with average BLEU 0.519, METEOR 0.388, ROUGE 0.631 CIDEr 2.531 and SER 2.55%.
manually cleaned human-produced utterances
derive rewards from human-human dialogues by assigning positive values to contextualised responses seen in the data, and negative values to randomly chosen responses due to lacking coherence
The task aims at extracting information about persons, organizations or geo-political entities from a large collection of news, web and discussion forum documents.
Generate a query out of the claim and querying a search engine, rank the words by means of TF-IDF, use IBM's AlchemyAPI to identify named entities, generate queries of 5–10 tokens, which execute against a search engine, and collect the snippets and the URLs in the results, skipping any result that points to a domain that is considered unreliable.
task-specific embedding of the claim together with all the above evidence about it, which comes from the last hidden layer of the NN
embedding of the claim Web evidence
NELL
the annotated dataset reported by degen2015investigating, a dataset of the utterances from the Switchboard corpus of telephone dialogues BIBREF21 that contain the word some
BERT, ERNIE, and BERT-wwm
ttention vector sequence is segmented into several subsequences and each subsequence represents the attention of one word we devise an appropriate aggregation module to fuse the inner-word character attention
weibo-100k, Ontonotes, LCQMC and XNLI
Emotion Classification (EC) Named Entity Recognition (NER) Sentence Pair Matching (SPM) Natural Language Inference (NLI)
entity match rate BLEU score Success F1 score
3029
KVRET
calendar weather navigation
Caravel, COAV and NNCD
80 excerpts from scientific works collection of 1,645 chat conversations collection of 200 aggregated postings
restrict the content of each text to the abstract and conclusion of the original work considered other parts of the original works such as introduction or discussion sections extracted text portions are appropriate for the AV task, each original work was preprocessed manually removed tables, formulas, citations, quotes and sentences that include non-language content such as mathematical constructs or specific names of researchers, systems or algorithms
MOCC, OCCAV, COAV, AVeer, GLAD, DistAV, Unmasking, Caravel, GenIM, ImpGI, SPATIUM and NNCD
Annotation was done with the help of annotators from Amazon Mechanical Turk on snippets of conversations
unigram, bigram and trigram
They look at the performance accuracy of explanation and the prediction performance
This article presented a brief overview of embedding models of entity and relationships for KB completion.
by converting human advice to first-order logic format and use as an input to calculate gradient
affective features provided by different emotion models such as Emolex, EmoSenticNet, Dictionary of Affect in Language, Affective Norms for English Words and Linguistics Inquiry and Word Count