DinoStackAI/qasper-rag huggingface dataset:
220 human-human dialogs. The average conversation length is 12.45 turns and the average utterance length is 11.13 words.
we design a hierarchical intent annotation scheme for non-collaborative tasks. We first separate on-task and off-task intents. As on-task intents are key actions that can vary among different tasks, we need to specifically define on-task intents for each task. On the other hand, since off-task content is too general to design task-specific intents, we choose common dialog acts as the categories. In the intent annotation scheme shown in Table TABREF2, we list the designed intent annotation scheme for the newly collected AntiScam dataset and the PersuasionForGood dataset. We first define on-task intents for the datasets, which are key actions in the task. Since our AntiScam focuses on understanding and reacting towards elicitations, we define elicitation, providing_information and refusal as on-task intents. In the PersuasionForGood dataset, we define nine on-task intents in Table TABREF2 based on the original PersuasionForGood dialog act annotation scheme For specific tasks, we also design a semantic slot annotation scheme for annotating sentences based on their semantic content. We identify 13 main semantic slots in the anti-scam task, for example, credit card numbers. We present a detailed semantic slot annotation in Table TABREF3. Following BIBREF1, we segment each conversation turn into single sentences and then annotate each sentence rather than turns.
TransferTransfo The vanilla TransferTransfo framework Hybrid Following BIBREF4 yu2017learning, we also build a hybrid dialog system by combining vanilla TransferTransfo and MISSA
Fluency Fluency is used to explore different models' language generation quality.

Coherence Different from single sentence's fluency, coherence focuses more on the logical consistency between sentences in each turn.

Engagement In the anti-scam scenario, one of our missions is to keep engaging with the attackers to waste their time. So we directly ask volunteers (attackers) to what extend they would like to continue chatting with the system.

Dialog length (Length) Engagement is a subjective metric. Anti-scam system's goal is to engage user in the conversation longer in order to limit their harm to other potential victims. So we count the dialog length as another metric to evaluate system performance.

Task Success Score (TaskSuc) The other goal of the anti-scam system is to elicit attacker's personal information. We count the average type of information (name, address and phone number) that the system obtained from attackers as the task success score.
The micro and macro f1-scores of this model are 0.482 and 0.399 on the AIDA-CoNLL dataset, 0.087 and 0.515 on the Microposts 2016 dataset, 0.870 and 0.858 on the ISTEX-1000 dataset, 0.335 and 0.310 on the RSS-500 dataset
two state-of-the-art early rumour detection baselines Liu et. al (2015) and Yang et. al (2012), which we re-implemented. Yang et. al (2012), dubbed Yang, because they proposed a feature set for early detection tailored to Sina Weibo and were used as a state-of-the-art baseline before by Liu et. al (2015). The algorithm by Liu et. al (2015), dubbed Liu, is said to operate in real-time and outperformed Yang, when only considering features available on Twitter.
accuracy to evaluate effectiveness Detection Error Trade-off (DET) curves, which show the trade-off between miss and false alarm probability throughput per second
No. They additionally use similarity to previously detected rumors to make the decision of whether a document is likely to be a rumor
Yes, consisting of trusted resources, rumours and non-rumours
Mandarin Chinese (see table 3)
the presence of information unconfirmed by the official media is construed as an indication of being a rumour.
LDA BIBREF2 Doc-NADE BIBREF24 HTMM BIBREF9 GMNTM BIBREF12 LDA BIBREF2 Doc-NADE BIBREF24 HTMM BIBREF9 GMNTM BIBREF12
generative document evaluation task document classification task topic2sentence task
CoNLL2003 OntoNotes 5.0 BIBREF35 released OntoNotes 4.0. In this paper, we use the Chinese part Chinese NER dataset MSRA Weibo NER Resume NER
calculate the attention scores  which can  distinguish different directions and distances
we achieve state-of-the-art performance among models without considering the pre-trained language models or designed features
KALM-QA achieves an accuracy of 95% for parsing the queries The second dataset we use is MetaQA dataset BIBREF14 , which contains contains almost 29,000 test questions and over 260,000 training questions. KALM-QA achieves 100% accuracy
SEMAFOR, SLING, and Stanford KBP system BIBREF14
first dataset is manually constructed general questions based on the 50 logical frames second dataset we use is MetaQA dataset BIBREF14 , which contains contains almost 29,000 test questions and over 260,000 training questions
the adaptation of a state-of-the-art sequence to sequence (seq2seq) neural system to event detection and classification for Italian in a single step approach an investigation on the quality of existing Italian word embeddings for this task a comparison against a state-of-the-art discrete classifier pre-trained models and scripts running the system
cascade of two SVM classifiers (one for detection and one for classification) based on rich linguistic features
Given a cluster, our algorithm proceeds with the following three steps:

Sense disambiguation: The goal is to assign each cluster word to one of its WordNet synsets; let $S$ represent the collection of chosen synsets. We know that these words have been clustered in domain-specific embedding space, which means that in the context of the domain, these words are very close semantically. Thus, we choose $S^*$ that minimizes the total distance between its synsets.

Candidate label generation: In this step, we generate $L$, the set of possible cluster labels. Our approach is simple: we take the union of all hypernyms of the synsets in $S^*$.

Candidate label ranking: Here, we rank the synsets in $L$. We want labels that are as close to all of the synsets in $S^*$ as possible; thus, we score the candidate labels by the sum of their distances to each synset in $S^*$ and we rank them from least to most distance.

In steps 1 and 3, we use WordNet pathwise distance, but we encourage the exploration of other distance representations as well.
First, we trained domain-specific word embeddings using the Word2Vec BIBREF33 CBOW model ($w \in R^{100}$). Then, we used k-means clustering to cluster the embeddings of the gender-associated words. Since k-means may converge at local optima, we ran the algorithm 50 times and kept the model with the lowest sum of squared errors.
Celebrity Dataset has 15,917 texts, 342,645 sentences, and the Female Male Proportions are  0.67/ 0.33. 
Professor Dataset has 283,973 texts, 976, 667 sentences, and the Femal Male Proportions are 0.28./ 0,72
The authors contrasted human evaluations against a random baseline, and used the centroid of the cluster as a strong baseline.
either by refusing politely, or, with flirtatious responses, or, by retaliating
9960 HITs from 472 crowd workers
Turkish, Finnish, Czech, German, Spanish, Catalan and English
char3 slides a character window of width $n=3$ over the token lemma of the token additional information for some languages, such as parts-of-speech tags for Turkish. Word segmenters such as Morfessor and Byte Pair Encoding (BPE) are other commonly used subword units. characters character sequences
The Semantic Scholar corpus Springer Nature's SciGraph The Textbook Question Answering corpus January 2018 English Wikipedia dataset Flickr30K COCO
Embedding network 2WayNet VSE++ DSVE-loc)
The direct combination baseline The supervised pre-training baseline
scientific publications middle school science curricula
Weka baseline BIBREF5
For the anger dataset, our experiments showed that GloVe embeddings of dimension 50 outperformed others, obtaining an average gain of 0.066 correlation over embeddings of size 25 and of 0.021 for embeddings of size 100. In the joy dataset, our experiments showed us that GloVe vectors of dimension 50 again outperformed others, in this case obtaining an average correlation gain of 0.052 ( INLINEFORM0 ) over embeddings of size 100, and of 0.062 ( INLINEFORM1 ) for size 25. On the fear dataset, again we observed that embeddings of size 50 provided the best results, offering average gains of 0.12 ( INLINEFORM0 ) and 0.11 ( INLINEFORM1 ) for sizes 25 and 100, respectively. on the sadness datasets again we experimentally observed that using embeddings of 50 offered the best results, with a statistically significant average gain of 0.092 correlation points INLINEFORM0 over size 25
training, validation and test datasets provided for the shared task BIBREF5 , which include tweets for four emotions: joy, sadness, fear, and anger
Hate speech is a text that contains one or more of the following aspects: directness, offensiveness, targeting a group or individual based on specific attributes, overall negativity.
English, French, and Arabic
(a) whether the text is direct or indirect; (b) if it is offensive, disrespectful, hateful, fearful out of ignorance, abusive, or normal; (c) the attribute based on which it discriminates against an individual or a group of people; (d) the name of this group; and (e) how the annotators feel about its content within a range of negative to neutral sentiments
5,647 English tweets, 4,014 French tweets, and 3,353 Arabic tweets
SWT can be applied to support the semantic disambiguation in MT: to  recognize ambiguous words before translation and  as a post-editing technique applied to  the output language. SWT may be used for translating KBs.
syntactic disambiguation problem which as yet lacks good solutions directly related to the ambiguity problem and therefore has to be resolved in that wider context In rare cases SMT can solve this problem, but considering that new idiomatic expressions appear every day and most of them are isolated sentences, this challenge still remains open
Excessive focus on English and European languages limitations of SMT approaches for translating across domains no-standard speech texts from users morphologically rich languages parallel data for training differs widely from real user speech
We extracted five statistical features for EEG, namely root mean square, zero crossing rate,moving window average,kurtosis and power spectral entropy BIBREF0 So in total we extracted 31(channels) X 5 or 155 features for EEG signals.The EEG features were extracted at a sampling frequency of 100Hz for each EEG channel We extracted Mel-frequency cepstrum (MFCC) as features for speech signal. We first extracted MFCC 13 features and then computed first and second order differentials (delta and delta-delta) thus having total MFCC 39 features.
The two types of simultaneous speech EEG recording databases:  A- five female and five male subjects took part in the experiment, and B- five male and three female subjects took part in the experiment.
LadaBERT -1, -2 achieves state of art on all datasets namely, MNLI-m MNLI-mm, SST-2, QQP, and QNLI. 
LadaBERT-3 achieves SOTA on the first four dataset. 
LadaBERT-4 achieves SOTA on MNLI-m, MNLI-mm, and QNLI
the benchmark DUC 2002 document summarization corpus DailyMail news highlights corpus
Knowledge than an ordinary person would have such as transitive entailment relation, complex ordering, compositionality, multi-word entities
In these models, the inferred embedding space creates a globally consistent structured prediction of the ontology, rather than the local relation predictions of previous models.
No, they used someone else's pretrained model.
unigrams and bigrams word2vec manually constructed lexica sentiment embeddings
2 (Spanish and English)
node types that represent different diagram elements The same features are used for both AI2D and AI2D-RST for nodes with layout information discourse relations information about semantic relations
The annotation for AI2D was
created by crowd-sourced non-expert annotators on AMT while AI2D-RST covers a subset of diagrams from AI2D annotated by trained experts
Expert annotators incorporate domain knowledge from multimodality theory while non-expert cannot but they are less time-consuming and use less resources.
Those who have domain knowledge on multimodal communication and annotation.
Recurrent Neural Networks Convolutional Neural Networks
Clueweb09 derived dataset new dataset based on Wikipedia crawl data
semantic representations of word embeddings
About 64M segments from YouTube videos comprising a total of 1.2B tokens.
64M segments from YouTube videos INLINEFORM0 B tokens vocabulary of 66K wordpieces
features are 1500-dimensional vectors, extracted from the video frames at 1-second intervals, similar to those used for large scale image classification tasks
architecture BIBREF10 based on LSTMs (long short-term memory cells) and implementing the attention mechanism
Symbolic rewriting is the method to rewrite ground and nonground data from one to another form using rules.
anchors are specified manually and more loosely defined as words having high mutual information with one or more latent factors
20 Newsgroups i2b2 2008 Obesity Challenge BIBREF22 data set
For Named Entity, F-Score Driven I model had 49.40 F1 score,  and F-Score Driven II model had 50.60  F1 score. In case of Nominal Mention, the scores were 58.16 and 59.32
Peng and Dredze peng-dredze:2016:P16-2
Sina Weibo service
Peng and Dredze peng-dredze:2016:P16-2 Peng and Dredze peng-dredze:2016:P16-2 from Sina Weibo service
These include 3 classification tasks: NLI (XNLI dataset), document classification (MLDoc dataset) and intent classification, and 2 sequence tagging tasks: POS tagging and NER.
we see that the gains are more pronounced in low resource languages
For a given language pair, $l$, let $D_l$ be the size of the available parallel corpus. Then if we adopt a naive strategy and sample from the union of the datasets, the probability of the sample being from language pair $l$ will be $p_l=\frac{D_l}{\Sigma _lD_l}$. However, this strategy would starve low resource language pairs. To control for the ratio of samples from different language pairs, we sample a fixed number of sentences from the training data, with the probability of a sentence belonging to language pair $l$ being proportional to $p_l^{\frac{1}{T}}$, where $T$ is the sampling temperature. As a result, $T=1$ would correspond to a true data distribution, and, $T=100$ yields an (almost) equal number of samples for each language pair (close to a uniform distribution with over-sampling for low-resource language-pairs). We set $T=5$ for a balanced sampling strategy. To control the contribution of each language pair when constructing the vocabulary, we use the same temperature based sampling strategy with $T=5$. Our SPM vocabulary has a character coverage of $0.999995$.
They use automatic evaluation using perplexity and BLEU scores with reference to the human-annotated responses and human evaluation on interestingness, engagingness, and humanness.
significant gap between the cross-lingual model and other models Table TABREF20
Translate source sentence to English with Google Translate API and then translate the result to the target language with Poly-encoder.
Chinese, French, Indonesian, Italian, Korean, and Japanese
It is automatically created from the OpenSubtitles corpus.
standard bidirectional RNN model with attention concat22 s-hier A multi-encoder architecture with hierarchical attention s-t-hier s-hier-to-2 A standard context-agnostic Transformer. concat22 concat21 BIBREF8
standard bidirectional RNN model with attention concat22 s-hier s-t-hier s-hier-to-2 standard context-agnostic Transformer concat22 concat21 BIBREF8
English German
we use a simple EL algorithm that directly links the mention to the entity with the greatest commonness score. Commonness BIBREF17, BIBREF18 is calculated base on the anchor links in Wikipedia. It estimates the probability of an entity given only the mention string.
BiLSTM with a three-layer perceptron
FIGER (GOLD) BIBREF0 BBN BIBREF5
One domain expert.
Precision, recall, f1-score, and support.
The model gives an F1-score of $0.89$ for the concept recognition task.
BIO Labelling Scheme
abb: represents abbreviations such as TRL representing Technology Readiness Level.

grp: represents a group of people or an individual such as Electrical Engineers, Systems Engineers or a Project Manager.

syscon: represents any system concepts such as engineering unit, product, hardware, software, etc. They mostly represent physical concepts.

opcon: represents operational concepts such as decision analysis process, technology maturity assessment, system requirements review, etc.

seterm: represents generic terms that are frequently used in SE text and those that do not fall under syscon or opcon such as project, mission, key performance parameter, audit etc.

event: represents event-like information in SE text such as Pre-Phase A, Phase A, Phase B, etc.

org: represents an organization such as `NASA', `aerospace industry', etc.

art: represents names of artifacts or instruments such as `AS1300'

cardinal: represents numerical values such as `1', `100', 'one' etc.

loc: represents location-like entities such as component facilities or centralized facility.

mea: represents measures, features, or behaviors such as cost, risk, or feasibility.
BERT
roughly 3700 sentences at the word-token level
Precision, recall and F-measure.
position of sentence sentence length tense qualifying adjectives meta-discourse features
INLINEFORM1 specific word embeddings were trained ( INLINEFORM2 )
sentence embeddings were learned from large text corpus as features to classify sentences into seven categories in the task of AZ. Three models were explored to obtain the sentence vectors: averaging the vectors of the words in one sentence, paragraph vectors and specific word vectors
Rhetorical sentence classification, also known as argumentative zoning (AZ) BIBREF3 , is a process of assigning rhetorical status to the extracted sentences
By crawling YouLikeHits and Like4Like sites and then using Twitter's REST API
spam detection method proposed by Wu et al. BIBREF4 ,  spam detection method  proposed by Rajdev et. al. BIBREF11,   feature vector by concatenating the tweet content features with Random Forest,  feature vector by concatenating the tweet content features with Multi-layer Perception and  feature vector by concatenating the tweet content features with Support Vector Machine.
Credit-based Freemium services
English French Chinese
6 transformer layers, each layer containing 1024 hidden units, 8 attention heads, and GELU activations.
10 transformer layers, each layer containing 1024 hidden units, 8 attentions heads, and GELU activations.
CorefNqg BIBREF33 Mp-Gsn BIBREF31 Xlm BIBREF5 Xlm Fine-tuning Pipeline (Xlm) Pipeline (Xlm) with Google Translator
Accuracy at pairing stories with the prompts used to generate them; accuracy of prompt ranking
model perplexity on the test set prompt ranking accuracy
gated convolutional language (GCNN) model of BIBREF4 and our additional self-attention mechanism LSTMs and convolutional seq2seq architectures, and Conv seq2seq with decoder self-attention an ensemble of two Conv seq2seq with self-attention models KNN model
convolutional language model from BIBREF4
Reddit's WritingPrompts forum
word2vec fastText GloVe Baroni SL999
STSB, SICK, MRPC
HCTI BIBREF5 InferSent BIBREF23
Fleiss's Kappa
Relative positions of the author and target accounts in the directed following network by
computing modified versions of Jaccard’s similarity index, friends count, followers count, verified status, number of tweets posted within 6 months.
Aggressive language Repetition Harmful intent Visibility among peers Power imbalance
They define cyberbullying as aggressive language, repetition, harmful intent, visibility among peers, and power imbalance
For measuring the similarity of the generated texts we used Phrase Overlap match and K-gram-Jaccard similarity as our criteria To evaluate the quality of the generated jokes, quotes, or tweets we rely on human judgment
CrowdTruth and Subreddits
1x3 filter size is used in convolutional layers.
improvements of INLINEFORM0 in MRR (which is about 25.1% relative improvement) INLINEFORM1 % absolute improvement in Hits@10
BERT, BERT+ Bi-LSTM ,  BERT+ DenseNet, BERT+HighwayLSTM,   Ensembled model, BERT+ BIMPM, BERT+ BIMPM(first bi-LSTM removed),  BERT + Sim-Transformer .
CoNLL03 dataset BIBREF5 Yahoo Answer Classification Dataset “Quora-Question-Pair” dataset
two inter-annotator agreement aw agreement and Cohen's kappa across three annotators computed by averaging three pairwise comparisons
The corpus is jointly annotated by three native Mandarin Chinese speakers, all of whom have received advanced training in theoretical and computational linguistics. Supersense labeling was performed cooperatively by 3 annotators for 25% (235/933) of the adposition targets, and for the remainder, independently by the 3 annotators, followed by cooperative adjudication. Annotation was conducted in two phases, and therefore we present two inter-annotator agreement studies to demonstrate the reproducibility of SNACS and the reliability of the adapted scheme for Chinese.
933 manually identified adpositions
https://github.com/Sairamvinay/Fake-News-Dataset
Artificial Neural Network (ANN) Long Short Term Memory networks (LSTMs) Random Forest Logistic Regression Support Vector Machine (SVM)
Accuracy metric
Light Gradient Boosting Machine
the hypothesis that needs be studied is whether LDA was just a clever way to model this information leak in the given data or not
The evaluation of the text alignment quality was conducted according to the 5-point scale used in KocabiyikogluETAL:18:

Wrong alignment

Partial alignment with slightly compositional translational equivalence

Partial alignment with compositional translation and additional or missing information

Correct alignment with compositional translation and few additional or missing information

Correct alignment and fully compositional translation

The evaluation of the audio-text alignment quality was conducted according to the following 3-point scale:

Wrong alignment

Partial alignment, some words or sentences may be missing

Correct alignment, allowing non-spoken syllables at start or end.

The evaluation experiment was performed by two annotators who each rated 30 items from each bin, where 10 items were the same for both annotators in order to calculate inter-annotator reliability.
The evaluation of the audio-text alignment quality was conducted according to the following 3-point scale:

Wrong alignment

Partial alignment, some words or sentences may be missing

Correct alignment, allowing non-spoken syllables at start or end.

The evaluation experiment was performed by two annotators who each rated 30 items from each bin, where 10 items were the same for both annotators in order to calculate inter-annotator reliability.
Compared to previous state-of-the-art approaches (i.e., PCNN+HATT and PCNN+BAG-ATT), the proposed model can also outperform them by a large margin, i.e., 10.3% and 5.3%
Their best implementation for semantic relatedness task comparison outperforms standard MaxEnt by 0,052 Pearson  Correlation.
Their best implementation for Textual Entailment task comparison (84,2 accuracy) DOES NOT outperform standard SVM (84,6 accuracy).
SICK (Sentences Involving Compositional Knowledge) dataset
SRE04/05/06/08/10/MIXER6
LDC98S75/LDC99S79/LDC2002S06/LDC2001S13/LDC2004S07
Voxceleb 1/2
Fisher + Switchboard I
Callhome+Callfriend
primary system is the linear fusion of all the above six subsystems
Variation decreases when frequencies of synonyms is enhanced; variation increases when frequencies of synonyms, hyponyms, hypernyms are enhanced
news articles related to Islam and articles discussing Islam basics
WordNet European Union EuroVoc RuThes
language that is used to expresses hatred towards a targeted group or is intended to be derogatory, to humiliate, or to insult the members of the group
logistic regression, naïve Bayes, decision trees, random forests, and linear SVMs