niryuu/sni-each-converted huggingface dataset:
Thus, to limit the number of different inputs to the classifier, we wish to reduce the number of distinct word recognition outputs that an attacker can induce, not just the number of words on which the model is “fooled”. We denote this property of a model as its sensitivity. 
 Question: What does the "sensitivity" quantity denote?
Output: Extractive
DNN based models coupled with transfer learning beat the best-known results for all three datasets. Previous best F1 scores for Wikipedia BIBREF4 and Twitter BIBREF8 datasets were 0.68 and 0.93 respectively. We achieve F1 scores of 0.94 for both these datasets using BLSTM with attention and feature level transfer learning (Table TABREF25 ). For Formspring dataset, authors have not reported F1 score. Their method has accuracy score of 78.5% BIBREF2 . We achieve F1 score of 0.95 with accuracy score of 98% for the same dataset. 
 Question: What were their performance results?
Output: Abstractive
We address the robustness problem on top of GE-FL BIBREF0 , a GE method which leverages labeled features as prior knowledge. 
 Question: What background knowledge do they leverage?
Output: Extractive
A representation that is more in line with observed user behavior is a concept map BIBREF5 , a labeled graph showing concepts as nodes and relationships between them as edges (Figure FIGREF2 ). 
 Question: How do the authors define a concept map?
Output: Extractive
Multiple papers have shown that 'shallow' naive Bayes classifiers BIBREF12, BIBREF8, BIBREF13, BIBREF14, SVMs BIBREF15 and similar models work very well for doing LID. The DSL 2017 paper BIBREF1, for example, gives an overview of the solutions of all of the teams that competed on the shared task and the winning approach BIBREF16 used an SVM with character n-gram, parts of speech tag features and some other engineered features. The winning approach for DSL 2015 used an ensemble naive Bayes classifier. The fasttext classifier BIBREF17 is perhaps one of the best known efficient 'shallow' text classifiers that have been used for LID . Multiple papers have proposed hierarchical stacked classifiers (including lexicons) that would for example first classify a piece of text by language group and then by exact language BIBREF18, BIBREF19, BIBREF8, BIBREF0. Researchers have investigated deeper LID models li 
 Question: What is the approach of previous work?
Output: Extractive
To test if the learned representations can separate phonetic categories, we use a minimal pair ABX discrimination task BIBREF19 , BIBREF20 . For the within-speaker task, all the phones triplets belong to the same speaker (e.g. $x$3 ) Finally the scores for every pair of central phones are averaged and subtracted from 1 to yield the reported within-talker ABX error rate. For the across-speaker task, $x$4 and $x$5 belong to the same speaker, and $x$6 to a different one (e.g. $x$7 ). The scores for a given minimal pair are first averaged across all of the pairs of speakers for which this contrast can be made. As above, the resulting scores are averaged over all contexts over all pairs of central phones and converted to an error rate. 
 Question: What is the metric that is measures in this paper?
Output: Abstractive
Our main contributions are (1) the publication of Turkish corpus for coarse-grained and fine-grained NER, and TC research, (2) six different versions of corpus according to noise reduction methodology and entity types, (3) an analysis of the corpus and (4) benchmark comparisons for NER and TC tasks against human annotators.  
 Question: Did they experiment with the dataset on some tasks?
Output: Yes-no
We sparsify this graph to the union of a minimum spanning tree and a k-Nearest Neighbors (MST-kNN) graph BIBREF14, a geometric construction that removes less important similarities but preserves global connectivity for the graph and, hence, for the dataset. The MST-kNN graph is then analysed with Markov Stability BIBREF15, BIBREF16, BIBREF17, BIBREF18, a multi-resolution graph partitioning method that identifies relevant subgraphs (i.e., clusters of documents) at different levels of granularity. 
 Question: What cluster identification method is used in this paper?
Output: Abstractive
We demonstrate our approach on the task of answering open-domain fill-in-the-blank natural language questions. 
 Question: What task do they evaluate on?
Output: Abstractive
For sentiment classification, we systematically study the effect of character-level adversarial attacks on two architectures and four different input formats.  We also consider the task of paraphrase detection. 
 Question: What end tasks do they evaluate on?
Output: Abstractive
In order to decode spatial connections between the electrodes from the channel covariance matrix, we use a CNN BIBREF19 , in particular a four-layered 2D CNN stacking two convolutional and two fully connected hidden layers. The INLINEFORM0 feature map at a given CNN layer with input INLINEFORM1 , weight matrix INLINEFORM2 and bias INLINEFORM3 is obtained as: INLINEFORM4 . At this first level of hierarchy, the network is trained with the corresponding labels as target outputs, optimizing a cross-entropy cost function. In parallel, we apply a four-layered recurrent neural network on the channel covariance matrices to explore the hidden temporal features of the electrodes. Namely, we exploit an LSTM BIBREF20 consisting of two fully connected hidden layers, stacked with two LSTM layers and trained in a similar manner as CNN. 
 Question: How was the spatial aspect of the EEG signal computed?
Output: Extractive
A total of 2,50,000 tweets over a period of August 31st, 2015 to August 25th,2016 on Microsoft are extracted from twitter API BIBREF15 . The news on twitter about Microsoft and tweets regarding the product releases were also included. Stock opening and closing prices of Microsoft from August 31st, 2015 to August 25th, 2016 are obtained from Yahoo! Finance BIBREF16 . 
 Question: What dataset is used to train the model?
Output: Extractive
The baselines are: (a) trained on S-SQuAD, (b) trained on T-SQuAD and then fine-tuned on S-SQuAD, and (c) previous best model trained on S-SQuAD BIBREF5 by using Dr.QA BIBREF20 . The proposed approach (row (f)) outperforms previous best model (row (c)) by 2% EM score and over 1.5% F1 score. 
 Question: What was the previous best model?
Output: Extractive
Many NLP tasks utilize POS as features, but human annotated POS sequences are difficult and expensive to obtain. Thus, it is important to know if we can learn sentences-level syntactic embeddings for low-sources languages without treebanks.

We performed zero-shot transfer of the syntactic embeddings for French, Portuguese and Indonesian. French and Portuguese are simulated low-resource languages, while Indonesian is a true low-resource language. 
 Question: Do they evaluate on downstream tasks?
Output: Yes-no
These models define parametrized similarity scoring functions $: Q\times T\rightarrow \mathbb {R}$ , where $Q$ is the set of natural language questions and $T$ is the set of paraphrases of logical forms. 
 Question: Does a neural scoring function take both the question and the logical form as inputs?
Output: Yes-no
To identify new keywords in the selected microposts, we again leverage crowdsourcing, as humans are typically better than machines at providing specific explanations BIBREF18, BIBREF19. In the crowdsourcing task, workers are first asked to find those microposts where the model predictions are deemed correct. Then, from those microposts, workers are asked to find the keyword that best indicates the class of the microposts as predicted by the model. 
 Question: How is the keyword specific expectation elicited from the crowd?
Output: Extractive
To collect article-level labels, we utilized a platform in the company that has been used by the market research team to collect surveys from the subscribers of different news publishers. The survey works as follows: The user is first presented with a set of selected pages (usually 4 pages and around 20 articles) from the print paper the day before. The user can select an article each time that he or she has read, and answer some questions about it. We added 3 questions to the existing survey that asked the level of partisanship, the polarity of partisanship, and which pro- or anti- entities the article presents. We also asked the political standpoint of the user. The complete survey can be found in Appendices. 
 Question: Did they crowdsource the annotations?
Output: Yes-no
In particular, we perform an extensive set of experiments on standard benchmarks for bilingual dictionary induction and monolingual and cross-lingual word similarity, as well as on an extrinsic task: cross-lingual hypernym discovery. For both the monolingual and cross-lingual settings, we can notice that our models generally outperform the corresponding baselines.  As can be seen in Table 1 , our refinement method consistently improves over the baselines (i.e., VecMap and MUSE) on all language pairs and metrics. First and foremost, in terms of model-wise comparisons, we observe that our proposed alterations of both VecMap and MUSE improve their quality in a consistent manner, across most metrics and data configurations. 
 Question: What are the tasks that this method has shown improvements?
Output: Abstractive
For instance, the sentence “The girls sang a song and they danced” is translated into French as “Les filles ont chanté une chanson et ils ont dansé” by Google Translate (GT), Bing Translate, and Yandex. 
 Question: Do the authors conduct experiments on the tasks mentioned?
Output: Yes-no
Since labels are unknown on INLINEFORM1 , EGL computes the expectation of the gradient norm over all possible labelings. BIBREF3 interprets EGL as “expected model change”. In the following section, we formalize the intuition for EGL and show that it follows naturally from reducing the variance of an estimator. Eq. ( EQREF7 ) indicates that to reduce INLINEFORM0 on test data, we need to minimize the expected variance INLINEFORM1 over the test set. A practical issue is that we do not know INLINEFORM0 in advance. We could instead substitute an estimate INLINEFORM1 from a pre-trained model, where it is reasonable to assume the INLINEFORM2 to be close to the true INLINEFORM3 . The batch selection then works by taking the samples that have largest gradient norms, DISPLAYFORM0

For RNNs, the gradients for each potential label can be obtained by back-propagation. Another practical issue is that EGL marginalizes over all possible labelings, but in speech recognition, the number of labelings scales exponentially in the number of timesteps. Therefore, we only marginalize over the INLINEFORM0 most probable labelings. They are obtained by beam search decoding, as in BIBREF7 . The EGL method in BIBREF3 is almost the same as Eq. ( EQREF8 ), except the gradient's norm is not squared in BIBREF3 . 
 Question: How do they calculate variance from the model outputs?
Output: Extractive
It includes a total 708 hours of French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh) speeches, with French and German ones having the largest durations among existing public corpora. 
 Question: Which languages are part of the corpus?
Output: Extractive
The Style baseline achieves an Accuracy of 83%, LDA features 89% and a combination of the two gets 90%. However, training on full poems reduces this to 42—52%. 
 Question: Is the outcome of the LDA analysis evaluated in any way?
Output: Yes-no
To implement the first tier of our framework and detect moral relevance, we complement our morally relevant seed words with a corresponding set of seed words approximating moral irrelevance based on the notion of valence, i.e., the degree of pleasantness or unpleasantness of a stimulus. We refer to the emotional valence ratings collected by BIBREF28 for approximately 14,000 English words, and choose the words with most neutral valence rating that do not occur in the MFD as our set of morally irrelevant seed words, for an equal total number of morally relevant and morally irrelevant words. 
 Question: How do they quantify moral relevance?
Output: Abstractive
For the evaluation of summaries we use the standard ROGUE metric. For comparison with previous AMR based summarization methods, we report the Recall, Precision and INLINEFORM0 scores for ROGUE-1. 
 Question: Which evaluation methods are used?
Output: Abstractive
We compare our HR-VAE model with three strong baselines using VAE for text modelling:

VAE-LSTM-base: A variational autoencoder model which uses LSTM for both encoder and decoder. KL annealing is used to tackled the latent variable collapse issue BIBREF0;

VAE-CNN: A variational autoencoder model with a LSTM encoder and a dilated CNN decoder BIBREF7;

vMF-VAE: A variational autoencoder model using LSTM for both encoder and decoder where the prior distribution is the von Mises-Fisher (vMF) distribution rather than a Gaussian distribution BIBREF5. 
 Question: Do they compare against state of the art text generation?
Output: Yes-no
The semi-parametric models were trained on 32 GPUs with each replica split over 2 GPUs, one to train the translation model and the other for computing the CSTM. 
 Question: Does their combination of a non-parametric retrieval and neural network get trained end-to-end?
Output: Yes-no
We compared our models with the following state-of-the-art baselines:

Sequence to Sequence (Seq2Seq): A simple encoder-decoder model which concatenates four sentences to a long sentence with an attention mechanism BIBREF31 .

Hierarchical LSTM (HLSTM): The story context is represented by a hierarchical LSTM: a word-level LSTM for each sentence and a sentence-level LSTM connecting the four sentences BIBREF29 . A hierarchical attention mechanism is applied, which attends to the states of the two LSTMs respectively.

HLSTM+Copy: The copy mechanism BIBREF32 is applied to hierarchical states to copy the words in the story context for generation.

HLSTM+Graph Attention(GA): We applied multi-source attention HLSTM where commonsense knowledge is encoded by graph attention.

HLSTM+Contextual Attention(CA): Contextual attention is applied to represent commonsense knowledge. 
 Question: Did they compare to Transformer based large language models?
Output: Yes-no
TF-IDF: We applied TF-IDF features as inputs to four classifiers: support vector machine classifier (SVM), logistic regression classifier (LR), Naive Bayes classifier (NB) and random forest (RF); CNN: We followed the same architecture as BIBREF9; LSTM: We applied an LSTM model BIBREF20 to classify the sentences with pre-trained word embeddings;LSTM-soft: We then added a soft-attention BIBREF21 layer on top of the LSTM model where we computed soft alignment scores over each of the hidden states; LSTM-self: We applied a self-attention layer BIBREF22 to LSTM model.  
 Question: To what baseline models is proposed model compared?
Output: Extractive
Spanish (SPA), in contrast, is morphologically rich, and disposes of much larger verbal paradigms than English. Like English, it is a suffixing language, and it additionally makes use of internal stem changes (e.g., o $\rightarrow $ ue). We choose Zulu (ZUL), a Bantoid language. In contrast to the first two, it is strongly prefixing. Second, the system pretrained on HUN performing well suggests again that a source language with an agglutinative, as opposed to a fusional, morphology seems to be beneficial as well. Again, TUR and HUN obtain high accuracy, which is an additional indicator for our hypothesis that a source language with an agglutinative morphology facilitates learning of inflection in another language. 
 Question: Are agglutinative languages used in the prediction of both prefixing and suffixing languages?
Output: Yes-no
Since none of the datasets from previous works have been published, we decide to build a new one. We randomly collect INLINEFORM0 cases from China Judgments Online, among which INLINEFORM1 cases are for training, INLINEFORM2 each for validation and testing. 
 Question: what datasets are used in the experiment?
Output: Extractive
We quantify the efficiency-accuracy tradeoff compared to two rule-based baselines: Unif and Stopword.  
 Question: What are the baselines used?
Output: Extractive
Previously, datasets openly available in abusive language detection research on Twitter ranged from 10K to 35K in size BIBREF9 , BIBREF11 . This quantity is not sufficient to train the significant number of parameters in deep learning models. Due to this reason, these datasets have been mainly studied by traditional machine learning methods. Most recently, Founta et al. founta2018large introduced Hate and Abusive Speech on Twitter, a dataset containing 100K tweets with cross-validated labels. Although this corpus has great potential in training deep models with its significant size, there are no baseline reports to date.

This paper investigates the efficacy of different learning models in detecting abusive language. We compare accuracy using the most frequently studied machine learning classifiers as well as recent neural network models. Reliable baseline results are presented with the first comparative study on this dataset. Additionally, we demonstrate the effect of different features and variants, and describe the possibility for further improvements with the use of ensemble models. 
 Question: Does the dataset feature only English language data?
Output: Yes-no
Compared to using external models for confidence modeling, an advantage of the proposed method is that the base model does not change: the binary classification loss just provides additional supervision. For supervised open IE systems, the confidence score of an assertion is typically computed based on its extraction likelihood given by the model BIBREF3 , BIBREF5  We follow the evaluation metrics described by Stanovsky:2016:OIE2016: area under the precision-recall curve (AUC) and F1 score. 
 Question: How does this compare to traditional calibration methods like Platt Scaling?
Output: Abstractive
We also measure the usage of words related to people's core values as reported by Boyd et al. boyd2015. The sets of words, or themes, were excavated using the Meaning Extraction Method (MEM) BIBREF10 . 
 Question: How do they obtain psychological dimensions of people?
Output: Extractive
he first dataset (WIKI) was the same set of 200 sentences from Wikipedia used in BIBREF7 .  The second dataset (SCI) was a set of 220 sentences from the scientific literature. We sourced the sentences from the OA-STM corpus. In total 2247 triples were extracted. In total, 11262 judgements were obtained after running the annotation process. 
 Question: What is the size of the released dataset?
Output: Abstractive
While $\beta $-VAE offers regularizing the ELBO via an additional coefficient $\beta \in {\rm I\!R}^+$, a simple extension BIBREF16 of its objective function incorporates an additional hyperparameter $C$ to explicitly control the magnitude of the KL term,

where $C\!\! \in \!\! {\rm I\!R}^+$ and $| . |$ denotes the absolute value. 
 Question: How does explicit constraint on the KL divergence term that authors propose looks like?
Output: Abstractive
The library design of Torch-Struct follows the distributions API used by both TensorFlow and PyTorch BIBREF29. 
 Question: Does API provide ability to connect to models written in some other deep learning framework?
Output: Yes-no
We explore how to utilize additional commonsense knowledge (i.e. rationales) as the input to the task. Like we mentioned in Section SECREF6, we search relevant sentences from the OMCS corpus as the additional distant rationales, and ground truth rationale sentences for dev/test data. The inputs are no longer the concept-sets themselves, but in a form of “[rationales$|$concept-set]” (i.e. concatenating the rationale sentences and original concept-set strings). 
 Question: Are the models required to also generate rationales?
Output: Yes-no
There are three main components to the model: 1) input encoder 2) dynamic memory, and 3) output module. We will describe these three modules in details. The input encoder and output module implementations are similar to the Entity Network BIBREF17 and main novelty lies in the dynamic memory. We describe the operations executed by the network for a single example consisting of a document with $T$ sentences, where each sentence consists of a sequence of words represented with $K$ -dimensional word embeddings $\lbrace e_1, \ldots , e_N\rbrace $ , a question on the document represented as another sequence of words and an answer to the question. 
 Question: How is knowledge stored in the memory?
Output: Abstractive
Here we describe the components of our probabilistic model of question generation.  The details of optimization are as follows. First, a large set of 150,000 questions is sampled in order to approximate the gradient at each step via importance sampling. Second, to run the procedure for a given model and training set, we ran 100,000 iterations of gradient ascent at a learning rate of 0.1. 
 Question: Is it a neural model? How is it trained?
Output: Abstractive
Baselines. We compare our approach (FacTweet) to the following set of baselines:

[leftmargin=4mm]

LR + Bag-of-words: We aggregate the tweets of a feed and we use a bag-of-words representation with a logistic regression (LR) classifier.

Tweet2vec: We use the Bidirectional Gated recurrent neural network model proposed in BIBREF20. We keep the default parameters that were provided with the implementation. To represent the tweets, we use the decoded embedding produced by the model. With this baseline we aim at assessing if the tweets' hashtags may help detecting the non-factual accounts.

LR + All Features (tweet-level): We extract all our features from each tweet and feed them into a LR classifier. Here, we do not aggregate over tweets and thus view each tweet independently.

LR + All Features (chunk-level): We concatenate the features' vectors of the tweets in a chunk and feed them into a LR classifier.

FacTweet (tweet-level): Similar to the FacTweet approach, but at tweet-level; the sequential flow of the tweets is not utilized. We aim at investigating the importance of the sequential flow of tweets.

Top-$k$ replies, likes, or re-tweets: Some approaches in rumors detection use the number of replies, likes, and re-tweets to detect rumors BIBREF21. Thus, we extract top $k$ replied, liked or re-tweeted tweets from each account to assess the accounts factuality. We tested different $k$ values between 10 tweets to the max number of tweets from each account. Figure FIGREF24 shows the macro-F1 values for different $k$ values. It seems that $k=500$ for the top replied tweets achieves the highest result. Therefore, we consider this as a baseline. 
 Question: What baselines do they compare to?
Output: Extractive
Our architecture was designed to show that using a model that captures better information about the context and sequential nature of the text can outperform lexicon-based methods commonly used in the literature. 
 Question: What meaningful information does the GRU model capture, which traditional ML models do not?
Output: Extractive
In order to derive the discourse features, an entity grid is constructed by feeding the document through an NLP pipeline to identify salient entities. Two flavors of discourse features are created by populating the entity grid with either (i) grammatical relations (GR) or (ii) RST discourse relations (RST).  
 Question: What discourse features are used?
Output: Abstractive
Our attempt at language pre-training fell short of our expectations in all but one tested dataset. Our pre-training was unsuccessful in improving accuracy, even when applied to networks larger than those reported. 
 Question: Does pre-training on general text corpus improve performance?
Output: Yes-no
We feed these representations into a clustering algorithm that produces exactly eleven clusters of five cities each and evaluate them against both Calvino's original labels and crowdsourced human judgments. While the overall correlation with Calvino's labels is low, both computers and humans can reliably identify some thematic groups associated with concrete objects. 
 Question: How do they obtain human judgements?
Output: Abstractive
Our user study compares the correctness of three scenarios:

Parser correctness - our baseline is the percentage of examples where the top query returned by the semantic parser was correct.

User correctness - the percentage of examples where the user selected a correct query from the top-7 generated by the parser.

Hybrid correctness - correctness of queries returned by a combination of the previous two scenarios. The system returns the query marked by the user as correct; if the user marks all queries as incorrect it will return the parser's top candidate. 
 Question: Do they conduct a user study where they show an NL interface with and without their explanation?
Output: Yes-no
We compare our personalized models against two baselines. The first is a name-based Nearest-Neighbor model (NN). We initially adapted the Neural Checklist Model of BIBREF0 as a baseline; however, we ultimately use a simple Encoder-Decoder baseline with ingredient attention (Enc-Dec), which provides comparable performance and lower complexity. 
 Question: What are the baseline models?
Output: Extractive
we use a common sense definition of racist language, including all negative utterances, negative generalizations and insults concerning ethnicity, nationality, religion and culture. 
 Question: how did they ask if a tweet was racist?
Output: Abstractive
All cases exhibit high scores—in the vast majority of the cases substantially higher than reported in previous work. In particular, in BIBREF1 we assess the ability of LSTMs to learn subject-verb agreement patterns in English, and evaluate on naturally occurring wikipedia sentences.  BIBREF2 also consider subject-verb agreement, but in a “colorless green ideas” setting in which content words in naturally occurring sentences are replaced with random words with the same part-of-speech and inflection, thus ensuring a focus on syntax rather than on selectional-preferences based cues.  BIBREF3 consider a wider range of syntactic phenomena (subject-verb agreement, reflexive anaphora, negative polarity items) using manually constructed stimuli, allowing for greater coverage and control than in the naturally occurring setting. 
 Question: Were any of these tasks evaluated in any previous work?
Output: Yes-no
While the dataset was collected with simulated environments, no structure was imposed on the navigation instructions while crowd-sourcing data. Thus, many instructions in our dataset are ambiguous. Moreover, the order of the behaviors in the instructions is not always the same. For instance, a person said “turn right and advance” to describe part of a route, while another person said “go straight after turning right” in a similar situation. The high variability present in the natural language descriptions of our dataset makes the problem of decoding instructions into behaviors not trivial. See Appendix A of the supplementary material for additional details on our data collection effort. 
 Question: What language is the experiment done in?
Output: Abstractive
We extend the Amazon Conversational Bot Toolkit (CoBot) BIBREF6 which is a flexible event-driven framework. CoBot provides ASR results and natural language processing pipelines through the Alexa Skills Kit (ASK) BIBREF7. Gunrock corrects ASR according to the context (asr) and creates a natural language understanding (NLU) (nlu) module where multiple components analyze the user utterances. A dialog manager (DM) (dm) uses features from NLU to select topic dialog modules and defines an individual dialog flow. Each dialog module leverages several knowledge bases (knowledge). Then a natural language generation (NLG) (nlg) module generates a corresponding response. While we provide an overview of the system in the following sections, for detailed system implementation details, please see the technical report BIBREF1. 
 Question: Do they specify the model they use for Gunrock?
Output: Yes-no
We also are especially interested in seeing this model applied to different languages. 
 Question: Have the authors tried this approach on other languages?
Output: Yes-no
Our dataset is annotated based on Chinese pathology reports provided by the Department of Gastrointestinal Surgery, Ruijin Hospital. It contains 17,833 sentences, 826,987 characters and 2,714 question-answer pairs. All question-answer pairs are annotated and reviewed by four clinicians with three types of questions, namely tumor size, proximal resection margin and distal resection margin.  
 Question: Is all text in this dataset a question, or are there unrelated sentences in between questions?
Output: Abstractive
The primary feed for the analysis collected INLINEFORM0 million tweets containing the keywords `breast' AND `cancer'.  
 Question: How were breast cancer related posts compiled from the Twitter streaming API?
Output: Abstractive
We use MS COCO, Bing and Flickr datasets from BIBREF26 to train the model that generates questions. These datasets contain natural questions about images with the purpose of knowing more about the picture. As can be seen in the Figure FIGREF8, questions cannot be answered by only looking at the image. Each source contains 5,000 images with 5 questions per image, adding a total of 15,000 images with 75,000 questions. We use two datasets to train our chatbot model. The first one is the Persona-chat BIBREF15 which contains dialogues between two people with different profiles that are trying to know each other. It is complemented by the Cornell-movie dialogues dataset BIBREF27, which contains a collection of fictional conversations extracted from raw movie scripts. Persona-chat's sentences have a maximum of 15 words, making it easier to learn for machines and a total of 162,064 utterances over 10,907 dialogues. While Cornell-movie dataset contains 304,713 utterances over 220,579 conversational exchanges between 10,292 pairs of movie characters. 
 Question: How big dataset is used for training this system?
Output: Abstractive
A growing body of evidence shows that state-of-the-art models learn to exploit spurious statistical patterns in datasets BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, BIBREF17, instead of learning meaning in the flexible and generalizable way that humans do. Given this, human annotators—be they seasoned NLP researchers or non-experts—might easily be able to construct examples that expose model brittleness. 
 Question: What are the weaknesses found by non-expert annotators of current state-of-the-art NLI models?
Output: Extractive
Generic Sarcasm. We first examine the different patterns learned on the Gen dataset.  We observe that the not-sarcastic patterns appear to capture technical and scientific language, while the sarcastic patterns tend to capture subjective language that is not topic-specific. We observe an abundance of adjective and adverb patterns for the sarcastic class, although we do not use adjective and adverb patterns in our regex retrieval method. Rhetorical Questions. We notice that while the not-sarcastic patterns generated for RQs are similar to the topic-specific not-sarcastic patterns we find in the general dataset, there are some interesting features of the sarcastic patterns that are more unique to the RQs.

Many of our sarcastic questions focus specifically on attacks on the mental abilities of the addressee. This generalization is made clear when we extract and analyze the verb, subject, and object arguments using the Stanford dependency parser BIBREF32 for the questions in the RQ dataset. Hyperbole. One common pattern for hyperbole involves adverbs and adjectives, as noted above. We did not use this pattern to retrieve hyperbole, but because each hyperbolic sarcastic utterance contains multiple cues, we learn an expanded class of patterns for hyperbole.  We learn a number of verbal patterns that we had not previously associated with hyperbole, as shown in Table TABREF34 . Interestingly, many of these instantiate the observations of CanoMora2009 on hyperbole and its related semantic fields: creating contrast by exclusion, e.g. no limit and no way, or by expanding a predicated class, e.g. everyone knows. Many of them are also contrastive.  
 Question: What are the linguistic differences between each class?
Output: Abstractive
We test the character and word-level variants by predicting hashtags for a held-out test set of posts. 
 Question: What other tasks do they test their method on?
Output: Abstractive
Experimental Setup 
 Question: what english datasets were used?
Output: Abstractive
Visual Question Generation (VQG) is another emerging topic which aims to ask questions given an image. Motivated by this, BIBREF10 proposed open-ended VQG that aims to generate natural and engaging questions about an image. 
 Question: Do they survey visual question generation work?
Output: Yes-no
While BERT-QC achieves large gains over existing methods on the ARC dataset, here we demonstrate that BERT-QC also matches state-of-the-art performance on TREC BIBREF6 , while surpassing state-of-the-art performance on the GARD corpus of consumer health questions BIBREF3 and MLBioMedLAT corpus of biomedical questions BIBREF4 . 
 Question: Which datasets are used for evaluation?
Output: Extractive
We choose the following three models as the baselines:

K-means is a well known data clustering algorithm, we implement the algorithm using sklearn toolbox, and represent documents using bag-of-words weighted by TF-IDF.

LEM BIBREF13 is a Bayesian modeling approach for open-domain event extraction. It treats an event as a latent variable and models the generation of an event as a joint distribution of its individual event elements. We implement the algorithm with the default configuration.

DPEMM BIBREF14 is a non-parametric mixture model for event extraction. It addresses the limitation of LEM that the number of events should be known beforehand. We implement the model with the default configuration. 
 Question: What baseline approaches does this approach out-perform?
Output: Extractive
Therefore, we decided to compile our own corpora based on English documents, which we crawled from different publicly accessible sources. 
 Question: Do they report results only on English data?
Output: Yes-no
Inspired by the StackLite tag recommendation task on Kaggle, we build a new benchmark based on the public StackExchange data. We use questions with titles as source, and user-assigned tags as target keyphrases.

Since oftentimes the questions on StackExchange contain less information than in scientific publications, there are fewer keyphrases per data point in StackEx. Furthermore, StackExchange uses a tag recommendation system that suggests topic-relevant tags to users while submitting questions; therefore, we are more likely to see general terminology such as Linux and Java. This characteristic challenges models with respect to their ability to distill major topics of a question rather than selecting specific snippets from the text. 
 Question: How was the StackExchange dataset collected?
Output: Abstractive
We assume that speech transcripts can be extracted from audio signals with high accuracy, given the advancement of ASR technologies BIBREF7 . 
 Question: Do they use datasets with transcribed text or do they determine text from the audio?
Output: Abstractive
In contrast to most encoder-decoder architectures, which are purely based on recurrent neural network (RNNs) framework, we construct an encoder with several convolutional layers BIBREF14 followed by NIN layers BIBREF15 as the lower part in the encoder and integrate them with deep bidirectional long short-term memory (Bi-LSTM) BIBREF16 at the higher part. On the decoder side, we use a standard deep unidirectional LSTM with global attention BIBREF13 that is calculated by a multi-layer perceptron (MLP) as described in Eq. EQREF2 . 
 Question: Which architecture do they use for the encoder and decoder?
Output: Extractive
We experimented with four different classifiers, namely, support vector machine BIBREF18 , random forest, extra tree and naive bayes classifier BIBREF19 . Chi square feature selection algorithm is applied to reduces the size of our feature vector. For training our system classifier, we used Scikit-learn BIBREF19 . 
 Question: What type of system does the baseline classification use?
Output: Extractive
The previously mentioned datasets are all in English 
 Question: For which languages most of the existing MRC datasets are created?
Output: Abstractive
We evaluate the QRNN architecture on a popular document-level sentiment classification benchmark, the IMDb movie review dataset BIBREF17 . The dataset consists of a balanced sample of 25,000 positive and 25,000 negative reviews, divided into equal-size train and test sets, with an average document length of 231 words BIBREF18 . We compare only to other results that do not make use of additional unlabeled data (thus excluding e.g., BIBREF19 ). 
 Question: What sentiment classification dataset is used?
Output: Extractive
We chose SVMhmm BIBREF111 implementation of Structural Support Vector Machines for sequence labeling. SVM implementation of   
 Question: Which machine learning methods are used in experiments?
Output: Abstractive
We have training and testing sets in three different languages: English, Chinese and Korean. When fine-tuning, we simply adopted the official training script of BERT, with default hyperparameters, to fine-tune each model until training loss converged. 
 Question: what does the model learn in zero-shot setting?
Output: Extractive
We test our proposed approach for binary classification on either sarcasm or irony, on seven benchmark datasets retrieved from different media sources. Below we describe each dataset, please see Table TABREF1 below for a summary. Twitter: We use the Twitter dataset provided for the SemEval 2018 Task 3, Irony Detection in English Tweets BIBREF18  Finally, we use the dataset by BIBREF20 , who collected a user self-annotated corpus of tweets with the #sarcasm hashtag. Reddit: BIBREF21 collected SARC, a corpus comprising of 600.000 sarcastic comments on Reddit. We use main subset, SARC 2.0, and the political subset, SARC 2.0 pol. Online Dialogues: We utilize the Sarcasm Corpus V1 (SC-V1) and the Sarcasm Corpus V2 (SC-V2), which are subsets of the Internet Argument Corpus (IAC). Compared to other datasets in our selection, these differ mainly in text length and structure complexity BIBREF22 .  To augment each dataset with our external data, we first filter out tweets that are not in English using language guessing systems. 
 Question: Do they evaluate only on English?
Output: Yes-no
Last but not least, ethics and fairness are important considerations, that deserve to be studied. In that sense, detection of individual and global bias should be prioritized in order to give useful feedbacks to practitioners. Furthermore we are considering using adversarial learning as in BIBREF33 in order to ensure fairness during the training process. 
 Question: Is there any ethical consideration in the research?
Output: Yes-no
The test-set accuracies obtained by different learning methods, including the current state-of-the-art results, are presented in Table TABREF11 . 
 Question: what models did they compare to?
Output: Abstractive
Davidson et al. BIBREF2 showed that there were systematic and substantial racial biases in five benchmark Twitter datasets annotated for offensive language detection. By examining more samples and with respect to recently studies BIBREF2, BIBREF24, BIBREF19, it is clear that many errors are due to biases from data collection BIBREF19 and rules of annotation BIBREF24 and not the classifier itself.  
 Question: What are the existing biases?
Output: Extractive
We compare the SBERT sentence embeddings to other sentence embeddings methods on the following seven SentEval transfer tasks:

MR: Sentiment prediction for movie reviews snippets on a five start scale BIBREF25.

CR: Sentiment prediction of customer product reviews BIBREF26.

SUBJ: Subjectivity prediction of sentences from movie reviews and plot summaries BIBREF27.

MPQA: Phrase level opinion polarity classification from newswire BIBREF28.

SST: Stanford Sentiment Treebank with binary labels BIBREF29.

TREC: Fine grained question-type classification from TREC BIBREF30.

MRPC: Microsoft Research Paraphrase Corpus from parallel news sources BIBREF31. 
 Question: What transfer learning tasks are evaluated?
Output: Extractive
We demonstrated the utility of Katecheo by deploying the system for question answering in two topics, Medical Sciences and Christianity. 
 Question: how many domains did they experiment with?
Output: Abstractive
The resulting dataset is nearly balanced, with 52.3% of the data (1,857 instances) labeled stressful. 
 Question: Is the dataset balanced across categories?
Output: Yes-no
In addition to the traditional VAE structure, we introduces an extra context-aware latent variable in CWVAE to learn the event background knowledge. In the pretrain stage, CWVAE is trained on an auxiliary dataset (consists of three narrative story corpora and contains rich event background knowledge), to learn the event background information by using the context-aware latent variable. Subsequently, in the finetune stage, CWVAE is trained on the task-specific dataset to adapt the event background information to each specific aspect of If-Then inferential target (e.g., intents, reactions, etc.). 
 Question: How does the context-aware variational autoencoder learn event background information?
Output: Abstractive
Table TABREF14 shows the results of our main experiments on the 2016 and 2018 test sets for French and German. We use Meteor BIBREF31 as the main metric, as in the WMT tasks BIBREF25 . We compare our transformer baseline to transformer models enriched with image information, as well as to the deliberation models, with or without image information.

We first note that our multimodal models achieve the state of the art performance for transformer networks (constrained models) on the English-German dataset, as compared to BIBREF30 . Second, our deliberation models lead to significant improvements over this baseline across test sets (average INLINEFORM0 , INLINEFORM1 ). 
 Question: What dataset does this approach achieve state of the art results on?
Output: Extractive
We validate our approach on the Gigaword corpus, which comprises of a training set of 3.8M article headlines (considered to be the full text) and titles (summaries), along with 200K validation pairs, and we report test performance on the same 2K set used in BIBREF7.  
 Question: What dataset they use for evaluation?
Output: Abstractive
We use UltraSuite: a repository of ultrasound and acoustic data gathered from child speech therapy sessions BIBREF15 . We used all three datasets from the repository: UXTD (recorded with typically developing children), and UXSSD and UPX (recorded with children with speech sound disorders). In total, the dataset contains 13,815 spoken utterances from 86 speakers, corresponding to 35.9 hours of recordings. The utterances have been categorised by the type of task the child was given, and are labelled as: Words (A), Non-words (B), Sentence (C), Articulatory (D), Non-speech (E), or Conversations (F).  
 Question: Do they annotate their own dataset or use an existing one?
Output: Abstractive
For the purpose of our experiments, we use data released by the APDA shared task organizers. The dataset is divided into train and test by organizers. The training set is distributed with labels for the three tasks of age, dialect, and gender. 
 Question: What are the three datasets used in the paper?
Output: Abstractive
. We could observe that compared with the base model, we have an improvement of 7.36% on accuracy and 9.69% on F1 score.  
 Question: How big are improvements of small-scale unbalanced datasets when sentence representation is enhanced with topic information?
Output: Extractive
We collect a novel dataset of 230K+ recipe texts and 1M+ user interactions (reviews) over 18 years (2000-2018) from Food.com. 
 Question: How did they obtain the interactions?
Output: Extractive
We conducted an experiment to determine whether we could maintain or improve classifier performances by applying the following three-tiered feature elimination approach:

Reduction We reduced the dataset encoded for each class by eliminating features that occur less than twice in the full dataset.

Selection We iteratively applied Chi-Square feature selection on the reduced dataset, selecting the top percentile of highest ranked features in increments of 5 percent to train and test the support vector model using a linear kernel and 5-fold, stratified cross-validation.

Rank We cumulatively plotted the average F1-score performances of each incrementally added percentile of top ranked features. We report the percentile and count of features resulting in the first occurrence of the highest average F1-score for each class. 
 Question: What are the three steps to feature elimination?
Output: Extractive
To remain consistent with experiments performed with LSTM's we use the morfessor for the subword tokenization in the Finnish Language. 
 Question: Is the LSTM baseline a sub-word model?
Output: Yes-no
After Korektor, there are 4262 unique sentences (including 150 seed sentences) that form the COSTRA 1.0 dataset. 
 Question: How many sentence transformations on average are available per unique sentence in dataset?
Output: Abstractive
The first data batch consists of tweets relevant to blizzards, hurricanes, and wildfires, under the constraint that they are tweeted by “influential" tweeters, who we define as individuals certain to have a classifiable sentiment regarding the topic at hand. For example, we assume that any tweet composed by Al Gore regarding climate change is a positive sample, whereas any tweet from conspiracy account @ClimateHiJinx is a negative sample. The assumption we make in ensuing methods (confirmed as reasonable in Section SECREF2 ) is that influential tweeters can be used to label tweets in bulk in the absence of manually-labeled tweets.  
 Question: What methodology is used to compensate for limited labelled data?
Output: Abstractive
Co-Reference Resolution: To support multi-turn interactions, it is sometimes necessary to use co-reference resolution techniques for effective retrieval. In Macaw, we identify all the co-references from the last request of user to the conversation history. The same co-reference resolution outputs can be used for different query generation components. This can be a generic or action-specific component.

Query Generation: This component generates a query based on the past user-system interactions. The query generation component may take advantage of co-reference resolution for query expansion or re-writing.

Retrieval Model: This is the core ranking component that retrieves documents or passages from a large collection. Macaw can retrieve documents from an arbitrary document collection using the Indri python interface BIBREF9, BIBREF10. We also provide the support for web search using the Bing Web Search API. Macaw also allows multi-stage document re-ranking.

Result Generation: The retrieved documents can be too long to be presented using some interfaces. Result generation is basically a post-processing step ran on the retrieved result list. In case of question answering, it can employ answer selection or generation techniques, such as machine reading comprehension models. For example, Macaw features the DrQA model BIBREF11 for question answering. 
 Question: What functionality does Macaw provide?
Output: Extractive
We built a phrase-based Chinese-to-English SMT system by using Moses BIBREF18 .  In the end, the final parallel text consists of around 8.8M sentence pairs, 228M Chinese tokens, and 254M English tokens (a token can be a word or punctuation symbol).  The total number of words in these two corpora is 1.81M for Chinese and 2.03M for English. 
 Question: Did they only experiment with one language pair?
Output: Yes-no
Encoders with induced latent structures have been shown to benefit several tasks including document classification, natural language inference BIBREF12, BIBREF13, and machine translation BIBREF11.  
 Question: Is there any evidence that encoders with latent structures work well on other tasks?
Output: Yes-no