DinoStackAI/qasper-rag huggingface dataset:
The first one is that it should be highly correlated with human judgement of similarity. The second one is that it should be able to distinguish sentences which are in logical contradiction, logically unrelated or in logical agreement. The third one is that a robust evaluator should also be able to identify unintelligible sentences. The last criteria is that a good evaluation metric should not give high scores to semantically distant sentences and low scores to semantically related sentences.
High scores to semantically opposite translations/summaries, Low scores to semantically related translations/summaries and High scores to unintelligible translations/summaries.
SUMBT BIBREF17 is the current state-of-the-art model on WOZ 2.0 TRADE BIBREF3 is the current published state-of-the-art model
WSD is predominantly evaluated on English, we are also interested in evaluating our approach on Chinese
Senseval-2 (SE2), Senseval-3 (SE3), SemEval 2013 task 12 (SE13), and SemEval 2015 task 13 (SE15) OntoNotes Release 5.0
Full Testing Set accuracy: 84.02
Cleaned Testing Set accuracy: 93.48
Full Testing Set Accuracy: 84.02
Cleaned Testing Set Accuracy: 93.48
Operation accuracy: 71.89
Execution accuracy: 55.95
Table TABREF20 Table TABREF22 Table TABREF23
DA recognition is aimed to assign a label to each utterance in a conversation. It can be formulated as a supervised classification problem.
target some heuristically extracted contents treat INLINEFORM1 as a latent variable and co-train selector and generator by maximizing the marginal data likelihood reinforcement learning to approximate the marginal likelihood Variational Reinforce-Select (VRS) which applies variational inference BIBREF10 for variance reduction
The baseline models used are DrQA modified to support answering no answer questions, DrQA+CoQA which is pre-tuned on CoQA dataset, vanilla BERT, BERT+review tuned on domain reviews, BERT+CoQA tuned on the supervised CoQA data
Non-English code is a large-scale phenomena. Transliteration is common in identifiers for all languages. Languages clusters into three distinct groups based on how speakers use identifiers/comments/transliteration. Non-latin script users write comments in their L1 script but write identifiers in English. Right-to-left (RTL) language scripts, such as Arabic, have no observed prevalence on GitHub identifiers
dialogue simulator
back translation between English and Chinese
Active Intent Accuracy Requested Slot F1 Average Goal Accuracy Joint Goal Accuracy
Arabic, German, Portuguese, Russian, Swedish
Spanish, English, Italian, Arabic, German, Portuguese, Russian and Swedish
text-transformations to the messages vector space model Support Vector Machine
The three baseline models are the i-vector model, a standard RNN LID system and a multi-task RNN LID system.
Proposing an improved RNN model, the phonetic temporal neural LID approach, based on phonetic features that results in better performance
Training and testing are done in alternating steps: In each epoch, for training, we first present to an LSTM network 1000 samples in a given language, which are generated according to a certain discrete probability distribution supported on a closed finite interval. We then freeze all the weights in our model, exhaustively enumerate all the sequences in the language by their lengths, and determine the first $k$ shortest sequences whose outputs the model produces inaccurately. experimented with 1, 2, 3, and 36 hidden units for $a^n b^n$ ; 2, 3, 4, and 36 hidden units for $a^n b^n c^n$ ; and 3, 4, 5, and 36 hidden units for $a^n b^n c^n d^n$ . Following the traditional approach adopted by BIBREF7 , BIBREF12 , BIBREF9 and many other studies, we train our neural network as follows. At each time step, we present one input character to our model and then ask it to predict the set of next possible characters, based on the current character and the prior hidden states. Given a vocabulary $\mathcal {V}^{(i)}$ of size $d$ , we use a one-hot representation to encode the input values; therefore, all the input vectors are $d$ -dimensional binary vectors. The output values are $(d+1)$ -dimensional though, since they may further contain the termination symbol $\dashv $ , in addition to the symbols in $\mathcal {V}^{(i)}$ . The output values are not always one-hot encoded, because there can be multiple possibilities for the next character in the sequence, therefore we instead use a $k$ -hot representation to encode the output values. Our objective is to minimize the mean-squared error (MSE) of the sequence predictions.
These are well-known formal languages some of which was used in the literature to evaluate the learning capabilities of RNNs.
$2.2\%$ absolute accuracy improvement on the laptops test set $3.6\%$ accuracy improvement on the restaurants test set
results that for the in-domain training case, our models BERT-ADA Lapt and BERT-ADA Rest achieve performance close to state-of-the-art on the laptops dataset new state-of-the-art on the restaurants dataset with accuracies of $79.19\%$ and $87.14\%$, respectively.
graph-like structures where arcs connect nodes representing multiple hypothesized words, thus allowing multiple incoming arcs unlike 1-best sequences
Mandarin dataset Cantonese dataset
NMI between cluster assignments and ground truth tones for all sylables is:
Mandarin: 0.641
Cantonese: 0.464
Perceptron model using the local features.
a multilingual word representation which aims to learn a linear mapping from a source to a target embedding space
Convolutional Neural Network (CNN)
language-independent (e.g., punctuation marks, positive and negative emoticons, quotations, personal pronouns, tweet's length, named entities) language-dependent relying on dedicated lexicons (e.g., negation, opinion lexicons, opposition words)
AraVec for Arabic, FastText for French, and Word2vec Google News for English.
RNN LSTM
low-rank approximation of the co-occurrence matrix
One model per topic.
They evaluate quantitatively.
Logistic Regression (LR) Random Forest (RF) Support Vector Machines (SVM)
words extracted from YouTube video comments and descriptions for all YouTube videos shared in the user's timeline
text in their tweets and profile descriptions, their emoji use, their profile images, and music interests embodied by links to YouTube music videos, can help a classifier distinguish between gang and non-gang member profiles
EER 16.04, Cmindet 0.6012, Cdet 0.6107
Cebuano and Mandarin Tagalog and Cantonese
direct similarity over ConceptNet Numberbatch embeddings, the relationships inferred over ConceptNet by SME, features that compose ConceptNet with other resources (WordNet and Wikipedia), and a purely corpus-based feature that looks up two-word phrases in the Google Books dataset
$\textsc {BERT}_{\textsc {BASE}}$ ensemble from BIBREF3
3.1 F1 gain on the original dev set 11 F1 gain on the multi-hop dev set 10 F1 gain on the out-of-domain dev set.
RoBERTa baseline
small_parallel_enja Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF5
model is composed of an encoder (§SECREF5) and a decoder with the attention mechanism (§SECREF7) that are both implemented using recurrent neural networks (RNNs)
BLEU BIBREF8 RIBES BIBREF9 token-level delay
IMG-only QUES-only SAN SANDY VOES-Oracle VOES
tweets about `ObamaCare' in USA collected during march 2010
landing pages of URLs
Parts of Speech (POS) tags Prior polarity of the words Capitalization Negation Text Feature
only 1 in 9 qualitative papers in Human-Computer Interaction reported inter-rater reliability metrics low-effort responses from crowdworkers
sampled all papers published in the Computer Science subcategories of Artificial Intelligence (cs.AI), Machine Learning (cs.LG), Social and Information Networks (cs.SI), Computational Linguistics (cs.CL), Computers and Society (cs.CY), Information Retrieval (cs.IR), and Computer Vision (CS.CV), the Statistics subcategory of Machine Learning (stat.ML), and Social Physics (physics.soc-ph) filtered for papers in which the title or abstract included at least one of the words “machine learning”, “classif*”, or “supervi*” (case insensitive) filtered to papers in which the title or abstract included at least “twitter” or “tweet” (case insensitive)
“coding scheme” is defined coders are trained with the coding scheme Training sometimes results in changes to the coding scheme calculation of “inter-annotator agreement” or “inter-rater reliability.” there is a process of “reconciliation” for disagreements
structured content analysis (also called “closed coding”) is used to turn qualitative or unstructured data of all kinds into structured and/or quantitative data Projects usually involve teams of “coders” (also called “annotators”, “labelers”, or “reviewers”), with human labor required to “code”, “annotate”, or “label” a corpus of items.
the full tweet object including the id of the tweet, username, hashtags, and geolocation of the tweet
from January 1, 2020 until April 15, 2020
more than 3,934,610 million tweets
By using a Bayesian approach  and by using word-pairs, where they extract all the pairs of co-occurring words within each tweet.  They search for the words that achieve the highest number of spikes matching the days of events.
fastText and SVM BIBREF16
They use a left-to-right attention mask so that the input tokens can only attend to other input tokens, and the target tokens can only attend to the input tokens and already generated target tokens.
The image feature vectors are mapped into BERT embedding dimensions and treated like a text sequence afterwards.
Answer with content missing: (whole introduction) However, recent
studies observe the limits of ROUGE and find in
some cases, it fails to reach consensus with human.
judgment (Paulus et al., 2017; Schluter, 2017).
manually labeled and tell exactly if one sentence should be extracted (assuming our annotations are in agreement), to further verify that FAR correlates with human preference,
They perform t-SNE clustering to analyze discourse embeddings
character bigram CNN classifier
They derive entity grid with grammatical relations and RST discourse relations and concatenate them with pooling vector for the char-bigrams before feeding to the resulting vector to the softmax layer.
Entity grid with grammatical relations and RST discourse relations.
'= ( , { ll k(h:, g:) if hV, gV

1 otherwise } )

where $_{h:}$ and $_{g:}$ denote the embedding representations of $h$ and $g$ , respectively.
Raw data from Gigaword, Automatically segmented text from Gigaword, Heterogenous training data from People's Daily, POS data from People's Daily
five-character window context
human preference annotation is available $Q(x_1, x_2) \in \lbrace >,<,\approx \rbrace $ is the true label for the pair
Text Overlap Metrics, including BLEU Perplexity Parameterized Metrics
Pearson correlation to human judgement - proposed vs next best metric
Sample level comparison:
- Story generation: 0.387 vs 0.148
- Dialogue: 0.472 vs 0.341
Model level comparison:
- Story generation:  0.631 vs 0.302
- Dialogue: 0.783 vs 0.553
is to minimize the negative likelihood of the aligned unanswerable question $\tilde{q}$ given the answerable question $q$ and its corresponding paragraph $p$ that contains the answer
learn to ask unanswerable questions by editing answerable ones with word exchanges, negations, etc
best results were obtained using new word embeddings best group of word embeddings is EC The highest type F1-score was obtained for EC1 model, built using binary FastText Skip-gram method utilising subword information ability of the model to provide vector representation for the unknown words seems to be the most important
identify the boundaries of timexes and assign them to one of the following classes: date, time, duration, set Then we evaluated these results using more detailed measures for timexes
predicting the word given its context
CBOW and Skip-gram methods in the FastText tool BIBREF9
most relevant content of the website, including all subsites
relative WER improvement of 10%.
Transcribed data is available for duration of 38h 54m 38s for 8 speakers.
In the speaker-closed condition, two episodes were set aside from each speaker as development and test sets. In the speaker-open condition, all the data except for the test speaker's were used for training
HotspotQA: Yang, Ding, Muppet
Fever: Hanselowski, Yoneda, Nie
We treated the neural semantic retrieval at both the paragraph and sentence level as binary classification problems with models' parameters updated by minimizing binary cross entropy loss.
BERT-Base BIBREF2 to provide the state-of-the-art contextualized modeling
This seems to indicate that the downstream QA module relies more on the upstream paragraph-level retrieval whereas the verification module relies more on the upstream sentence-level retrieval.
significant MAP performance improvement compared to the previous best model, CompClip-LM (0.696 to 0.734 absolute)
we fully connect nodes that represent sentences from the same passage we fully connect nodes that represent the first sentence of each passage we add an edge between the question and every node for each passage
BiDAF BERT
Books DVDs Electronics Kitchen appliances
string kernels SST KE-Meta SFA CORAL TR-TrAdaBoost Transductive string kernels transductive kernel classifier
String kernel is a technique that uses character n-grams to measure the similarity of strings
Fisher Corpus English Part 1
They compute Pearson’s correlation between NED measure for patient-to-therapist and patient-perceived emotional bond rating and NED measure for therapist-to-patient and patient-perceived emotional bond rating
Bengali English, German, Spanish, Dutch Amharic Arabic Hindi Somali
character unit the RNN-transducer with additional attention module For subword units, classic RNN-transducer, RNN-transducer with attention and joint CTC-attention show comparable performance
1) investigate errors produced by the end-to-end methods and explore several approaches to correct common errors done in French 2) compare the end-to-end methods in a SLU context and evaluate the semantic value of the partially correct produced words
1) Connectionist Temporal Classification (CTC) 2) Attention-based methods 3) RNN-tranducer
the Transformer with 10 encoder layers and 2 decoder layers is $2.32$ times as fast as the 6-layer Transformer
The language is Chinese, which is not easy for non-Chinese-speaking researchers to work on.
300,000 sentences with 1.5 million single-quiz questions
sequence classification sequence labeling
a family of grammar formalisms built on a foundation of logic and type theory. Type-logical grammars originated when BIBREF4 introduced his Syntactic Calculus (called the Lambek calculus, L, by later authors).
parameter sharing
We see that A-gen performance improves significantly with the joint model: both F1 and EM increase by about 10 percentage points.
300-dimensional vectors
10 million sentences gathered from Wikipedia
the CMU ARCTIC database BIBREF33 the M-AILABS speech dataset BIBREF34
a RNN-based seq2seq VC model called ATTS2S based on the Tacotron model
Support Vector Machines (SVM), Gaussian Naive Bayes, Multinomial Naive Bayes, Decision Trees, Random Forests and a Maximum Entropy classifier
User location (uloc) User language (ulang) Timezone (tz) Tweet language (tlang) Offset (offset) User name (name) User description (description) Tweet content (content)
by adding extra supervision to generate the slots that will be present in the response
NDM, LIDM, KVRN, and TSCP/RL
180221 of 231162 author names could be matched successfully
Confusion in recognizing the words that are active at a given node by a speech recognition solution developed for Indian Railway Inquiry System.
English Russian
ROUGE BIBREF29 and METEOR BIBREF30
a two layer recurrent neural language model with GRU cells of hidden size 512 a two layer neural sequence to sequence model equipped with bi-linear attention function with GRU cells of hidden size 512 a linear dynamical system semi-supervised SLDS models with varying amount of labelled sentiment tags
Bi-directional LSTM self-attention
10-fold cross validation
precision ($P.$), recall ($R.$) and $F_1$ scores for target recognition and targeted sentiment
Proposed RCRN outperforms ablative baselines BiLSTM by +2.9% and 3L-BiLSTM by +1.1% on average across 16 datasets.
approximately equal parameterization
High-order CNN, Tree-LSTM, DRNN, DCNN, CNN-MC, NBoW and SVM
They used Stanford Sentiment Treebank benchmark for sentiment classification task and   AG English news corpus for the text classification task.
same sentences after applying character level perturbations
Akkadian.
dominant temporal associations can be learned from training data
DeepDive BIBREF1
FarsNet
The De7 database
They used 6 indicators for proficiency (same for written and spoken) each marked by bad, medium or good by one expert.
6 indicators:
- lexical richness
- pronunciation and fluency
- syntactical correctness
- fulfillment of delivery
- coherence and cohesion
- communicative, descriptive, narrative skills
Accuracy not available: WER results are reported 42.6 German, 35.9 English
Speech recognition system is evaluated using WER metric.
Total number of transcribed utterances including Train and Test for both Eng and Ger language is 5562 (2188 cleaned)
Total number of utterances available is: 70607 (37344 ENG + 33263 GER)
w.r.t Rouge-1 their model outperforms by 0.98% and w.r.t Rouge-L their model outperforms by 0.45%
neural attention model with a convolutional encoder with an RNN decoder and RNN encoder-decoder
identify all abbreviations using regular expressions
word2vec BIBREF0
1,160 physician logs of Medical ICU admission requests 42,506 Wikipedia articles 6 research papers and 2 critical care medicine textbooks
speaker systems in the real world
the $\mathbf {C}$hinese $\mathbf {A}$rtificial $\mathbf {I}$ntelligence $\mathbf {S}$peakers (CAIS)
BiLSTMs + CRF architecture BIBREF36 sententce-state LSTM BIBREF21
CNN model which additionally learns intermediate hidden layer representations and convolutional filters. Moreover the CNN model can take advantage of the semantic similarity encoded in the distributed word2vec representations
concatenation of GloVe pre-trained embeddings and DepecheMood BIBREF19 lexicon representation cannot directly concatenate re-build the latter in token-based form
DepecheMood
beneficial impact of word-representations and basic pre-processing
Questions are gathered from anonymized, aggregated queries to the Google search engine. Queries that are likely to be yes/no questions are heuristically identified: we found selecting queries where the first word is in a manually constructed set of indicator words and are of sufficient length, to be effective. Questions are only kept if a Wikipedia page is returned as one of the first five results, in which case the question and Wikipedia page are given to a human annotator for further processing. Annotators label question/article pairs in a three-step process. First, they decide if the question is good, meaning it is comprehensible, unambiguous, and requesting factual information. This judgment is made before the annotator sees the Wikipedia page. Next, for good questions, annotators find a passage within the document that contains enough information to answer the question. Annotators can mark questions as “not answerable" if the Wikipedia article does not contain the requested information. Finally, annotators mark whether the question's answer is “yes" or “no"
16k questions
Remove numbers and interjections
10 Epochs: pearson-Spearman correlation  drops  60 points when error increase by 20%
50 Epochs: pearson-Spearman correlation  drops  55 points when error increase by 20%
SST-2 dataset
non-canonical text such as spelling mistakes, typographic errors, colloquialisms, abbreviations, slang, internet jargon, emojis, embedded metadata (such as hashtags, URLs, mentions), non standard syntactic constructions and spelling variations, grammatically incorrect text, mixture of two or more languages
Hence WordPiece tokenizer tokenizes noisy words into subwords. However, it ends up breaking them into subwords whose meaning can be very different from the meaning of the original word. Often, this changes the meaning of the sentence completely, therefore leading to substantial dip in the performance.
we compressed English BERT using magnitude weight pruning BIBREF8 and observed the results on transfer learning to the General Language Understanding Evaluation (GLUE) benchmark BIBREF9, a diverse set of natural language understanding tasks including sentiment analysis, NLI, and textual similarity evaluation.
The increase is linearly from lowest on average 2.0 , medium around 3.5, and the largest is 6.0
human representative to review the IVA chat history and resume the failed task
computationally inexpensive means to understand what happened at the stopping point
321 videos
WikiTableQuestions
historical S&P 500 component stocks
 306242 news articles
A mechanism ${f}$ is a random function that takes a dataset $\mathcal {N}$ as input, and outputs a random variable ${f}(\mathcal {N})$.
it systematically holds out inputs in the training set containing basic primitive verb, "jump", and tests on sequences containing that verb.
non-native speech from the BULATS test
BULATS i-vector/PLDA
BULATS x-vector/PLDA
VoxCeleb x-vector/PLDA
PLDA adaptation (X1)
 Extractor fine-tuning (X2)
78,976
the food dataset has 3,806 images for training
VQA and GeoQA
allows models that are consistently performing well to train for more steps
Our search space consists of two stackable cells, one for the model encoder and one for the decoder Each cell contains NASNet-style blocks, which receive two hidden state inputs and produce new hidden states as outputs Our search space contains five branch-level search fields (input, normalization, layer, output dimension and activation), one block-level search field (combiner function) and one cell-level search field (number of cells).
It begins as ordinary tournament selection evolutionary architecture search with early stopping, with each child model training for a relatively small $s_0$ number of steps before being evaluated for fitness. However, after a predetermined number of child models, $m$ , have been evaluated, a hurdle, $h_0$ , is created by calculating the the mean fitness of the current population. For the next $m$ child models produced, models that achieve a fitness greater than $h_0$ after $s_0$ train steps are granted an additional $s_1$ steps of training and then are evaluated again to determine their final fitness. Once another $m$ models have been considered this way, another hurdle, $h_1$ , is constructed by calculating the mean fitness of all members of the current population that were trained for the maximum number of steps. For the next $m$ child models, training and evaluation continues in the same fashion, except models with fitness greater than $m$0 after $m$1 steps of training are granted an additional $m$2 number of train steps, before being evaluated for their final fitness. This process is repeated until a satisfactory number of maximum training steps is reached.
They use Monalog for data-augmentation to fine-tune BERT on this task
They derive it from Wordnet
gating function Dynamic Memory
strong competitive methods on the SQuAD leaderboard and TriviaQA leaderboard
when the gate has high value, more information flows from the word-level representation; otherwise, char-level will take the dominating place for unfamiliar noun entities, the gates tend to bias towards char-level representation in order to care richer morphological structure
parallel data available for the WMT 2016
systems were optimized on the tst2014 using Minimum error rate training BIBREF20
trained using Nematus default configuration
by replacing zero-width-non-joiner (ZWNJ) BIBREF2 and manually verifying the orthography
document-level accuracy precision recall F-score
Answer with content missing: (Names of many identifiers missing) TextCat, ChromeCLD, LangDetect, langid.py, whatlang, whatthelang, YALI, LDIG, Polyglot 3000, Lextek Language Identifier and Open Xerox Language Identifier.
candidates who have been liked or shortlisted are considered part of the hirable class
Paraphrase Identification