DinoStackAI/qasper-rag huggingface dataset:
a vocabulary of positive and negative predicates that helps determine the polarity score of an event
Using all data to train: AL -- BiGRU achieved 0.843 accuracy, AL -- BERT achieved 0.863 accuracy, AL+CA+CO -- BiGRU achieved 0.866 accuracy, AL+CA+CO -- BERT achieved 0.835, accuracy, ACP -- BiGRU achieved 0.919 accuracy, ACP -- BERT achived 0.933, accuracy, ACP+AL+CA+CO -- BiGRU achieved 0.917 accuracy, ACP+AL+CA+CO -- BERT achieved 0.913 accuracy. 
Using a subset to train: BERT achieved 0.876 accuracy using ACP (6K), BERT achieved 0.886 accuracy using ACP (6K) + AL, BiGRU achieved 0.830 accuracy using ACP (6K), BiGRU achieved 0.879 accuracy using ACP (6K) + AL + CA + CO.
cause relation: both events in the relation should have the same polarity; concession relation: events should have opposite polarity
7000000 pairs of events were extracted from the Japanese Web corpus, 529850 pairs of events were extracted from the ACP corpus
negative positive
by exploiting discourse relations to propagate polarity from seed predicates to final sentiment polarity
100 million sentences
confusion matrices of labels between annotators
feelings of suspense experienced in narratives not only respond to the trajectory of the plot's content, but are also directly predictive of aesthetic liking (or disliking) Emotions that exhibit this dual capacity have been defined as “aesthetic emotions”
Dynamic communities have substantially higher rates of monthly user retention than more stable communities. More distinctive communities exhibit moderately higher monthly retention rates than more generic communities. There is also a strong positive relationship between a community's dynamicity and the average number of months that a user will stay in that community - a short-term trend observed for monthly retention translates into longer-term engagement and suggests that long-term user retention might be strongly driven by the extent to which a community continually provides novel content.
communities that are characterized by specialized, constantly-updating content have higher user retention rates, but also exhibit larger linguistic gaps that separate newcomers from established members within distinctive communities, established users have an increased propensity to engage with the community's specialized content, compared to newcomers
They collect subreddits from January 2013 to December 2014,2 for which there are at
least 500 words in the vocabulary used to estimate the measures,
in at least 4 months of the subreddit’s history. They compute our measures over the comments written by users in a community in time windows of months, for each sufficiently active month, and manually remove communities where the bulk of the contributions are in a foreign language.
the average volatility of all utterances
the average specificity of all utterances
Chinese general corpus
QANet BIBREF39 BERT-Base BIBREF26
Clinical text structuring (CTS) is a critical task for fetching medical research data from electronic health records (EHRs), where structural patient medical data, such as whether the patient has specific symptoms, diseases, or what the tumor size is, how far from the tumor is cut at during the surgery, or what the specific laboratory test result is, are obtained. Unlike the traditional CTS task, our QA-CTS task aims to discover the most related text from original paragraph text.
three types of questions, namely tumor size, proximal resection margin and distal resection margin
the dataset consists of pathology reports including sentences and questions and answers about tumor size and resection margins so it does include additional sentences
2,714
integrate clinical named entity information into pre-trained language model
17,833 sentences, 826,987 characters and 2,714 question-answer pairs
state-of-the-art question answering models (i.e. QANet BIBREF39) and BERT-Base BIBREF26
Quality measures using perplexity and recall, and performance measured using latency and energy usage.
Kneser–Ney smoothing
Yelp Challenge dataset BIBREF2
AdaBoost-based classifier
1,006 fake reviews and 994 real reviews
The baseline model is a deep sequence-to-sequence encoder/decoder model with attention. The encoder is a bidirectional Long-Short Term Memory(LSTM) cell BIBREF14 and the decoder a single LSTM cell with attention mechanism. The attention mechanism is computed as in BIBREF9 and we use a greedy search for decoding. We train end-to-end including the words embeddings. The embedding size used is of 128 and the hidden state size of the LSTM cells is of 254.
We compare this method of bias mitigation with the no bias mitigation ("Orig"), geometric bias mitigation ("Geo"), the two pieces of our method alone ("Prob" and "KNN") and the composite method ("KNN+Prob"). We note that the composite method performs reasonably well according the the RIPA metric, and much better than traditional geometric bias mitigation according to the neighborhood metric, without significant performance loss according to the accepted benchmarks. To our knowledge this is the first bias mitigation method to perform reasonably both on both metrics.
RIPA, Neighborhood Metric, WEAT
only high-quality data helps
fastText CWE-LP
large Portuguese corpus
Continuous Bag-of-Words (CBOW)
it is necessary to evaluate the performance of the above mentioned part of the pipeline before proceeding further. The evaluation of the performance is summarised in Table TABREF11. It shows that organising the two models into the pipeline boosted the performance of the reference recognition model, leading to a higher F1 measure in the initial recognition of the text spans and their classification.
903019 references
Given we have four intensity, No PTSD, Low Risk PTSD, Moderate Risk PTSD and High Risk PTSD with a score of 0, 1, 2 and 3 respectively, the estimated intensity  is established as mean  squared error.
For each user, we calculate the proportion of tweets scored positively by each LIWC category.
DOSPERT, BSSS and VIAS
29,500 documents in the CORD-19 corpus (2020-03-13)
manually created lexicon in Czech BIBREF11 , German BIBREF12 , French BIBREF13 , Macedonian BIBREF14 , and Spanish BIBREF15
The contexts are manually labelled with WordNet senses of the target words
Answer with content missing: (Table 1)
Previous state-of-the art on same dataset: ResNet50 89% (6 languages), SVM-HMM 70% (4 languages)
CNN-mean CNN-avgmax
multiple language pairs including German-English, French-English, and Japanese-English.
we do not explicitly compare to previous research since most existing works either exploit smaller data (and so it will not be a fair comparison), use methods pre-dating BERT (and so will likely be outperformed by our models)
Arap-Tweet BIBREF19 an in-house Twitter dataset for gender the MADAR shared task 2 BIBREF20 the LAMA-DINA dataset from BIBREF22 LAMA-DIST Arabic tweets released by IDAT@FIRE2019 shared-task BIBREF24 BIBREF25, BIBREF26, BIBREF27, BIBREF1, BIBREF28, BIBREF29, BIBREF30, BIBREF31, BIBREF32, BIBREF33, BIBREF34
We construct a GAN model which combines different sets of word embeddings INLINEFORM4 , INLINEFORM5 , into a single set of word embeddings INLINEFORM6 .
CNN, TIME, 20 Newsgroups, and Reuters-21578
typos in spellings or ungrammatical words
In the sentiment classification task by 6% to 8% and in the intent classification task by 0.94% on average
34,432 user conversations
overall rating mean number of turns
Amazon Conversational Bot Toolkit natural language understanding (NLU) (nlu) module dialog manager knowledge bases natural language generation (NLG) (nlg) module text to speech (TTS) (tts)
modeled the relationship between word count and the two metrics of user engagement (overall rating, mean number of turns) in separate linear regressions
FastText BERT two-layer BiLSTM architecture with GloVe word embeddings
They used BERT-based models to detect subjective language in the WNC corpus
No, other baseline metrics they use besides ROUGE-L are n-gram overlap, negative cross-entropy, perplexity, and BLEU.
LSTM LMs
if the attention loose track of the objects in the picture and "gets lost", the model still takes it into account and somehow overrides the information brought by the text-based annotations
Soft attention Hard Stochastic attention Local Attention
dataset that contains article-comment parallel contents INLINEFORM0 , and an unpaired dataset that contains the documents (articles or comments) INLINEFORM1
Under the retrieval evaluation setting, their proposed model + IR2 had better MRR than NVDM by 0.3769, better MR by 4.6, and better Recall@10 by  20 . 
Under the generative evaluation setting the proposed model + IR2 had better BLEU by 0.044 , better CIDEr by 0.033, better ROUGE by 0.032, and better METEOR by 0.029
TF-IDF NVDM
from 50K to 4.8M
Chinese dataset BIBREF0
up to four percentage points in accuracy
2019 GermEval shared task on hierarchical text classification
all three representations are concatenated and passed into a MLP
Random Forest Ensemble classifiers
The Digital Library in the TextGrid Repository
automated attribute-value extraction score the attributes using the Bayes model evaluate their importance with several different frequency metrics aggregate the weights from different sources into one consistent typicality score using a Ranking SVM model OntoRank algorithm
FastText
SST-2 (Stanford Sentiment Treebank, version 2) Snips
wealth democracy population levels of ODA conflict
They focus on exclusivity and semantic coherence measures: Highly frequent words in a given topic that do not appear very often in other topics are viewed as making that topic exclusive. They select select the 16-topic model, which has the largest positive residual in the regression fit, and provides higher exclusivity at the same level of semantic coherence.
we measure our system's performance for datasets across various domains evaluations are done by managed judges who understands the knowledge base and then judge user queries relevance to the QA pairs
QnAMaker Portal QnaMaker Management APIs Azure Search Index QnaMaker WebApp Bot
boost the training BLEU very greatly the over-fitting problem of the Plackett-Luce models PL($k$) is alleviated with moderately large $k$
Plackett-Luce Model for SMT Reranking
perform experiments to utilize ASR $n$-best hypotheses during evaluation
on $\sim $ 8.7M annotated anonymised user utterances
Yes, Open IE
For the English version, we performed both a thorough manual analysis and automatic evaluation across three commonly used TS datasets from two different domains The evaluation of the German version is in progress.
the English version is evaluated. The German version evaluation is in progress
For example, one metric could consist in checking whether for any given word, all words that are known to belong to the same class are closer than any words belonging to different classes, independently of the actual cosine
coverage metric being distinct (cosine INLINEFORM0 0.7 or 0.8) belonging to the same class (cosine INLINEFORM1 0.7 or 0.8) being equivalent (cosine INLINEFORM2 0.85 or 0.95)
consistent increase in the validation loss after about 15 epochs
context is a procedural text, the question and the multiple choice answers are composed of images
Hasty Student Impatient Reader BiDAF BiDAF w/ static memory
Average accuracy of proposed model vs best prevous result:
Single-task Training: 57.57 vs 55.06
Multi-task Training: 50.17 vs 50.59
the scoring model takes the word embedding sequence as input, scoring over each individual candidate word from two perspectives: (1) the possibility that the candidate word itself can be regarded as a legal word; (2) the rationality of the link that the candidate word directly follows previous segmentation history
Active learning methods has a learning engine (mainly used for training of classification problems) and the selection engine (which chooses samples that need to be relabeled by annotators from unlabeled data). Then, relabeled samples are added to training set for classifier to re-train, thus continuously improving the accuracy of the classifier. In this paper, CRF-based segmenter and a scoring model are employed as learning engine and selection engine, respectively.
gated neural network
For event types and participant types, there was a moderate to substantial level of agreement using the Fleiss' Kappa. For coreference chain annotation, there was average agreement of 90.5%.
four different annotators
Kinship and Nations knowledge graphs, YAGO3-10 and WN18KGs knowledge graphs
if there is an error in the graph, the triple is likely to be inconsistent with its neighborhood, and thus the model should put least trust on this triple. In other words, the error triple should have the least influence on the model's prediction of the training data.
he proposed model outperforms all the baselines, being the svi version the one that performs best. the svi version converges much faster to higher values of the log marginal likelihood when compared to the batch version, which reflects the efficiency of the svi algorithm.
Bosch 2006 (mv) LDA + LogReg (mv) LDA + Raykar LDA + Rodrigues Blei 2003 (mv) sLDA (mv)
Reuters-21578 BIBREF30 LabelMe BIBREF31 20-Newsgroups benchmark corpus BIBREF29
Database Construction: we crawled travel information in Beijing from the Web, including Hotel, Attraction, and Restaurant domains (hereafter we name the three domains as HAR domains). Then, we used the metro information of entities in HAR domains to build the metro database. Goal Generation: a multi-domain goal generator was designed based on the database. The relation across domains is captured in two ways. One is to constrain two targets that locate near each other. The other is to use a taxi or metro to commute between two targets in HAR domains mentioned in the context. Dialogue Collection: before the formal data collection starts, we required the workers to make a small number of dialogues and gave them feedback about the dialogue quality. Then, well-trained workers were paired to converse according to the given goals. The workers were also asked to annotate both user states and system states. Dialogue Annotation: we used some rules to automatically annotate dialogue acts according to user states, system states, and dialogue histories.
BERTNLU from ConvLab-2 a rule-based model (RuleDST) TRADE (Transferable Dialogue State Generator) a vanilla policy trained in a supervised fashion from ConvLab-2 (SL policy)
The workers were also asked to annotate both user states and system states we used some rules to automatically annotate dialogue acts according to user states, system states, and dialogue histories
Only Bert base and Bert large are compared to proposed approach.
improving the score for WNLaMPro-medium by 50% compared to BERT$_\text{base}$ and 31% compared to Attentive Mimicking
MNLI BIBREF21 AG's News BIBREF22 DBPedia BIBREF23
WNLaMPro dataset
AIDA-CoNLL ACE2004 MSNBC AQUAINT WNED-CWEB WNED-WIKI OURSELF-WIKI
output of global LSTM network at time $V_{m_i}^t$5 , which encodes the mention context and target entity information from $V_{m_i}^t$6 to $V_{m_i}^t$7
Support Vector Regression (SVR) and Support Vector Classification (SVC) deep learning regression models of BIBREF2 to convert them to classification models
NNC SU4 F1 NNC top 5 Support Vector Classification (SVC)
irremediable annotation discrepancies differences in choice of attributes to annotate The resources themselves would need updating to encode the relevant morphosyntactic information. Some languages had a very low number of overlapping forms, and no tag matches or near-matches between them the two annotations encode distinct information incorrectly applied UniMorph annotation cross-lingual inconsistency in both resources
Ar, Bg, Ca, Cs, Da, De, En, Es, Eu, Fa, Fi, Fr, Ga, He, Hi, Hu, It, La, Lt, Lv, Nb, Nl, Nn, PL, Pt, Ro, Ru, Sl, Sv, Tr, Uk, Ur
confusion matrices $\text{F}_1$ score
For the emotion recognition from text they use described neural network as baseline.
For audio and face there is no baseline.
We apply an off-the-shelf tool for emotion recognition (the manufacturer cannot be disclosed due to licensing restrictions)
Training data with 159000, 80000, 40000, 20000, 10000 and 5000 sentences, and 7584 sentences for development
10.37 BLEU
gu-EtAl:2018:EMNLP1
highly data-inefficient underperform phrase-based statistical machine translation
Axelrod's causal mapping method
Ternary Trans-CNN
HEOT A labelled dataset for a corresponding english tweets
Resulting dataset was 7934 messages for train and 700 messages for test.
A labelled dataset for a corresponding english tweets were also obtained from a study conducted by Davidson et al HEOT obtained from one of the past studies done by Mathur et al
Ternary Trans-CNN Hybrid multi-channel CNN and LSTM
Table TABREF15 shows that word-alignment based on mBERT representations surpasses the outputs of the standard FastAlign tool even if it was provided large parallel corpus. This suggests that word-level semantics are well captured by mBERT contextual embeddings. For this task, learning an explicit projection had a negligible effect on the performance.
We thus try to remove the language-specific information from the representations by centering the representations of sentences in each language so that their average lies at the origin of the vector space.
contextual embeddings do not represent similar semantic phenomena similarly and therefore they are not directly usable for zero-shot cross-lingual tasks
Generative Pre-trained Transformer (GPT)
Across models and tasks: The degree (as grayscale) of faithfulness at the level of specific models and tasks. Across input space: The degree of faithfulness at the level of subspaces of the input space, such as neighborhoods of similar inputs, or singular inputs themselves.
Two models will make the same predictions if and only if they use the same reasoning process. On similar inputs, the model makes similar decisions if and only if its reasoning is similar. Certain parts of the input are more important to the model reasoning than others. Moreover, the contributions of different parts of the input are independent from each other.
Be explicit in what you evaluate. Faithfulness evaluation should not involve human-judgement on the quality of interpretation. Faithfulness evaluation should not involve human-provided gold labels. Do not trust “inherent interpretability” claims. Faithfulness evaluation of IUI systems should not rely on user performance.
we provide an extensive analysis of the state-of-the-art model
two LSTM layers
machine comprehension
69.10%/78.38%
Our baseline model is composed of the following typical components: word embedding, input encoder, alignment, aggregation, and prediction.
Our model achieves a 68.73% EM score and 77.39% F1 score
datasets from the NIST DUC-05, DUC-06 and DUC-07 shared tasks
BiGRUs with attention, ROUGE, Language model, and next sentence prediction
WN18RR BIBREF26 FB15k-237 BIBREF18 YAGO3-10 BIBREF27
TransE DistMult ComplEx ConvE RotatE
0.021 higher MRR, a 2.4% higher H@1, and a 2.4% higher H@3 against RotatE, respectively doesn't outperform the previous state-of-the-art as much as that of WN18RR and YAGO3-10 HAKE gains a 0.050 higher MRR, 6.0% higher H@1 and 4.6% higher H@3 than RotatE, respectively
radial coordinate and the angular coordinates correspond to the modulus part and the phase part, respectively
incorporating coding syntax tree model
A parallel corpus where the source is an English expression of code and the target is Python code.
seq2seq translation
phrase-based word embedding Abstract Syntax Tree(AST)
validation data
Even though natural language and image synthesis were part of several contributions on the supervised side of deep learning, unsupervised learning saw recently a tremendous rise in input from the research community specially on two subproblems: text-based natural language and image synthesis
give more independence to the several learning methods (e.g. less human intervention) involved in the studies increasing the size of the output images
HDGAN produced relatively better visual results on the CUB and Oxford datasets while AttnGAN produced far more impressive results than the rest on the more complex COCO dataset In terms of inception score (IS), which is the metric that was applied to majority models except DC-GAN, the results in Table TABREF48 show that StackGAN++ only showed slight improvement over its predecessor text to image synthesis is continuously improving the results for better visual perception and interception
Semantic Enhancement GANs: DC-GANs, MC-GAN
Resolution Enhancement GANs: StackGANs, AttnGAN, HDGAN
Diversity Enhancement GANs: AC-GAN, TAC-GAN etc.
Motion Enhancement GAGs: T2S, T2V, StoryGAN
gating mechanism acts upon each dimension of the word and character-level vectors
BiLSTM with max pooling
BIBREF13 , BIBREF18
MEN MTurk287 MTurk771 RG RW SimLex999 SimVerb3500 WS353 WS353R WS353S
WordDecoding (WDec) model
SPTree Tagging CopyR HRL GraphR N-gram Attention
WordDecoding (WDec) model achieves F1 scores that are $3.9\%$ and $4.1\%$ higher than HRL on the NYT29 and NYT24 datasets respectively PtrNetDecoding (PNDec) model achieves F1 scores that are $3.0\%$ and $1.3\%$ higher than HRL on the NYT29 and NYT24 datasets respectively
(1) Rank by the number of times a citation is mentioned in the document. (2) Rank by the number of times the citation is cited in the literature (citation impact). (3) Rank using Google Scholar Related Articles. (4) Rank by the TF*IDF weighted cosine similarity. (5) Rank using a learning-to-rank model trained on text similarity rankings.
SVMRank
90 annotated documents with 5 citations each ranked 1 to 5, where 1 is least relevant and 5 is most relevant for a total of 450 annotated citations
asked the authors to rank by closeness five citations we selected from their paper
accuracy of 82.6%
crowd-sourced the collection of the dataset on Amazon Mechanical Turk (MTurk)
localization accuracy
dataset on Mechanical Turk involving human perception, action and communication
F1 score of 0.71 for this task without any specific training, simply by choosing a threshold below which all sentence pairs are considered duplicates distances between duplicate and non-duplicate questions using different embedding systems
text clustering on the embeddings of texts
Quora duplicate question dataset BIBREF22
path ranking-based KGC (PRKGC)
WikiHop
True, Likely (i.e. Answerable), or Unsure (i.e. Unanswerable) why they are unsure from two choices (“Not stated in the article” or “Other”) The “summary” text boxes
two labels
political pundits of the Washington Post
peoples' sentiments expressed over social media
experts in Washington Post
Accuracy of best proposed method KANE (LSTM+Concatenation) are 0.8011, 0.8592, 0.8605 compared to best state-of-the art method R-GCN + LR 0.7721, 0.8193, 0.8229 on three datasets respectively.
we use t-SNE tool BIBREF27 to visualize the learned embedding
TransE, TransR and TransH PTransE, and ALL-PATHS R-GCN BIBREF24 and KR-EAR BIBREF26
Freebase BIBREF0, DBpedia BIBREF1 and a self-construction game knowledge graph
To capture both high-order structural information of KGs, we used an attention-based embedding propagation method.
entity types or concepts BIBREF13 relations paths BIBREF17 textual descriptions BIBREF11, BIBREF12 logical rules BIBREF23 deep neural network models BIBREF24
domain-targeted $~$ 80K sentences and 280 GB of plain text extracted from web pages used by BIBREF6 aristo2016:combining
for each multiple-choice question $(q,A) \in Q_\mathit {tr}$ and each choice $a \in A$ , we use all non-stopword tokens in $q$ and $a$ as an ElasticSearch query against S take the top 200 hits, run Open IE v4, and aggregate the resulting tuples over all $a \in A$ and over all questions in $Q_\mathit {tr}$
Word Sense Induction & Disambiguation
The dataset comes with a ranked set of relevant documents. Hence the baselines do not use a retrieval system.
Kyubyong Park Edouard Grave et al BIBREF11
No extraction, No annotation, Wrong range, Wrong tag, Wrong range and tag
Best BLSTM-CNN-CRF had F1 score 86.87 vs 86.69 of best BLSTM-CRF