niryuu/sni-each-converted huggingface dataset:
BIBREF3 also use bilingual crowd workers, but the studies supporting the use of crowdsourcing for MT evaluation were performed with older MT systems, and their findings may not carry over to the evaluation of contemporary higher-quality neural machine translation (NMT) systems. In addition, the MT developers to which crowd workers were compared are usually not professional translators.  We hypothesise that an evaluation of sentences in isolation, as applied by BIBREF3, precludes raters from detecting translation errors that become apparent only when inter-sentential context is available, and that they will judge MT quality less favourably when evaluating full documents. BIBREF3 used all source texts of the WMT 2017 Chinese-English test set for their experiments, of which only half were originally written in Chinese;  
 Question: What was the weakness in Hassan et al's evaluation design?
Output: Abstractive
Following titovcrosslingual, we run our experiments on the English (EN) and German (DE) sections of the CoNLL 2009 corpus BIBREF13 , and EN-DE section of the Europarl corpus BIBREF14 . 
 Question: Which parallel corpora are used?
Output: Extractive
As a first baseline we use Bag-of-Words, a well-known and robust text representations used in various domains BIBREF21 , combined with a standard shallow classifier, namely, a Support Vector Machine with linear kernel. We used LIBSVM implementation of SVM. 
 Question: Which shallow approaches did they experiment with?
Output: Extractive
Probing Methodology and Modeling ::: Task Definition and Modeling ::: Baselines and Sanity Checks.
When creating synthetic datasets, it is important to ensure that systematic biases, or annotation artifacts BIBREF41, are not introduced into the resulting probes and that the target datasets are sufficiently challenging (or good, in the sense of BIBREF42). To test for this, we use several of the MCQA baseline models first introduced in BIBREF0, which take inspiration from the LSTM-based models used in BIBREF43 for NLI and various partial-input baselines based on these models. 
 Question: How do they control for annotation artificats?
Output: Extractive
In this paper, we propose the first large-scale dataset for QA over social media data.   Table TABREF3 gives an example from our TweetQA dataset. The dataset now includes 10,898 articles, 17,794 tweets, and 13,757 crowdsourced question-answer pairs 
 Question: What is the size of this dataset?
Output: Extractive
We use hyperopt library to select the hyper-parameters on the following values: LSTM layer size (16, 32, 64), dropout ($0.0-0.9$), activation function ($relu$, $selu$, $tanh$), optimizer ($sgd$, $adam$, $rmsprop$) with varying the value of the learning rate (1e-1,..,-5), and batch size (4, 8, 16) 
 Question: What activation function do they use in their model?
Output: Extractive
For the sake of interpretability, we will also project back coefficients from the embeddings as well as PCA models into the dummy variable space. 
 Question: How do their interpret the coefficients?
Output: Abstractive
Since we were also interested in whether argumentation differs across registers, we included four different registers — namely (1) user comments to newswire articles or to blog posts, (2) posts in discussion forums (forum posts), (3) blog posts, and (4) newswire articles.  
 Question: How is the data in the new corpus come sourced?
Output: Extractive
For reliable multi-hop reasoning using OpenIE tuples, we can add inter-tuple connections to the support graph search, controlled by a small number of rules over the OpenIE predicates. Learning such rules for the Science domain is an open problem and potential avenue of future work. 
 Question: Is their method capable of multi-hop reasoning?
Output: Yes-no
We also observe that the best combination seems to consist in training our model on the original out-of-context dataset and testing it on the in-context pairs. In this configuration we reach an F-score (0.72) only slightly lower than the one reported in BIBREF3 (0.74), and we record the highest Pearson correlation, 0.3 (which is still not strong, compared to BIBREF3 's best run, 0.75).  
 Question: What were the results of the first experiment?
Output: Abstractive
VQA research began in earnest in late 2014 when the DAQUAR dataset was released BIBREF0 
 Question: From when are many VQA datasets collected?
Output: Abstractive
As far as we know, all existing systems treat this task as a pipeline of two separate subtasks, i.e., event extraction and temporal relation classification, and they also assume that gold events are given when training the relation classifier BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5. Specifically, they built end-to-end systems that extract events first and then predict temporal relations between them (Fig. FIGREF1). In these pipeline models, event extraction errors will propagate to the relation classification step and cannot be corrected afterwards. Our first contribution is the proposal of a joint model that extracts both events and temporal relations simultaneously (see Fig. FIGREF1). 
 Question: Is this the first paper to propose a joint model for event and temporal relation extraction?
Output: Yes-no
A base model, INLINEFORM0 , is trained on 190 hours ( INLINEFORM1 100K instances) of transcribed speech data. Then, it selects a subset of a 1,700-hour ( INLINEFORM2 1.1M instances) unlabeled dataset. We query labels for the selected subset and incorporate them into training. Learning rates are tuned on a small validation set of 2048 instances. The trained model is then tested on a 156-hour ( INLINEFORM3 100K instances) test set and we report CTC loss, Character Error Rate (CER) and Word Error Rate (WER). 
 Question: Which dataset do they use?
Output: Extractive
As a further application of our work, we have carried out a supervised classification task aimed at predicting the degree of harm of an incident directly from the text and the hand-coded features (e.g., external category, medical specialty, location).  We also checked if using our unsupervised content-driven cluster labels as additional features can improve the performance of the supervised classification. 
 Question: How are content clusters used to improve the prediction of incident severity?
Output: Abstractive
The first dataset is an expanded version of the annotated Wikipedia conversations dataset from BIBREF9. This dataset uses carefully-controlled crowdsourced labels, strictly filtered to ensure the conversations are civil up to the moment of a personal attack. This is a useful property for the purposes of model analysis, and hence we focus on this as our primary dataset. However, we are conscious of the possibility that these strict labels may not fully capture the kind of behavior that moderators care about in practice. We therefore introduce a secondary dataset, constructed from the subreddit ChangeMyView (CMV) that does not use post-hoc annotations. Instead, the prediction task is to forecast whether the conversation will be subject to moderator action in the future. Wikipedia data. BIBREF9's `Conversations Gone Awry' dataset consists of 1,270 conversations that took place between Wikipedia editors on publicly accessible talk pages. The conversations are sourced from the WikiConv dataset BIBREF59 and labeled by crowdworkers as either containing a personal attack from within (i.e., hostile behavior by one user in the conversation directed towards another) or remaining civil throughout. Reddit CMV data. The CMV dataset is constructed from conversations collected via the Reddit API. In contrast to the Wikipedia-based dataset, we explicitly avoid the use of post-hoc annotation. Instead, we use as our label whether a conversation eventually had a comment removed by a moderator for violation of Rule 2: “Don't be rude or hostile to other users”. 
 Question: What labels for antisocial events are available in datasets?
Output: Abstractive
We evaluated our model on three benchmark datasets, namely the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), and XSum BIBREF22. 
 Question: What are the datasets used for evaluation?
Output: Extractive
Automation of cQA forums can be divided into three tasks: question-comment relevance (Task A), question-question relevance (Task B), and question-external comment relevance (Task C). In our cQA tasks, the pair of objects are (question, question) or (question, comment), and the relationship is relevant/irrelevant. For task C, in addition to an original question (oriQ) and an external comment (relC), the question which relC commented on is also given (relQ). To incorporate this extra information, we consider a multitask learning framework which jointly learns to predict the relationships of the three pairs (oriQ/relQ, oriQ/relC, relQ/relC). 
 Question: What supplemental tasks are used for multitask learning?
Output: Abstractive
We assess the optimal number of topics that need to be specified for the STM analysis. We follow the recommendations of the original STM paper and focus on exclusivity and semantic coherence measures. Highly frequent words in a given topic that do not appear very often in other topics are viewed as making that topic exclusive.  Following BIBREF8 we generate a set of candidate models ranging between 3 and 50 topics. We then plot the exclusivity and semantic coherence (numbers closer to 0 indicate higher coherence), with a linear regression overlaid (Figure FIGREF3 ). Models above the regression line have a “better” exclusivity-semantic coherence trade off. We select the 16-topic model, which has the largest positive residual in the regression fit, and provides higher exclusivity at the same level of semantic coherence. The topic quality is usually evaluated by highest probability words, which is presented in Figure FIGREF4 . 
 Question: How are the main international development topics that states raise identified?
Output: Abstractive
The Software Ontology (SWO) BIBREF5 is included because its set of CQs is of substantial size and it was part of Ren et al.'s set of analysed CQs. The CQ sets of Dem@Care BIBREF8 and OntoDT BIBREF9 were included because they were available. CQs for the Stuff BIBREF6 and African Wildlife (AWO) BIBREF7 ontologies were added to the set, because the ontologies were developed by one of the authors (therewith facilitating in-depth domain analysis, if needed), they cover other topics, and are of a different `type' (a tutorial ontology (AWO) and a core ontology (Stuff)), thus contributing to maximising diversity in source selection. 
 Question: How many domains of ontologies do they gather data from?
Output: Abstractive
We evaluate the proposed architecture on two publicly available datasets: the Adverse Drug Events (ADE) dataset BIBREF6 and the CoNLL04 dataset BIBREF7. We show that our architecture is able to outperform the current state-of-the-art (SOTA) results on both the NER and RE tasks in the case of ADE. In the case of CoNLL04, our proposed architecture achieves SOTA performance on the NER task and achieves near SOTA performance on the RE task. On both datasets, our results are SOTA when averaging performance across both tasks. 
 Question: Do they repot results only on English data?
Output: Yes-no
We evaluate our approach by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews BIBREF6 (see Appendix for details). We quantify the efficiency of a communication scheme $(q_{\alpha },p_{\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords. The accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence. 
 Question: How are models evaluated in this human-machine communication game?
Output: Extractive
GPT-2 achieves the highest score and the $n$-gram the lowest. Transformer-XL and the LSTM LM perform in the middle, and at roughly the same level as each other. We report the 12-category accuracy results for all models and human evaluation in Table TABREF14. 
 Question: What is the performance of the models on the tasks?
Output: Abstractive
Adding word alignments in parallel sentences results in small, non significant improvements, even if there is some labeled data available in the source language. This difficulty in showing the usefulness of parallel corpora for SRI may be due to the current assumptions about role alignments, which mean that only a small percentage of roles are aligned. Further analyses reveals that annotating small amounts of data can easily outperform the performance gains obtained by adding large unlabeled dataset as well as adding parallel corpora. 
 Question: Overall, does having parallel data improve semantic role induction across multiple languages?
Output: Yes-no
We evaluate the sequence-to-sequence QRNN architecture described in SECREF5 on a challenging neural machine translation task, IWSLT German-English spoken-domain translation, applying fully character-level segmentation.  
 Question: What languages pairs are used in machine translation?
Output: Extractive
The dataset MSParS is published by NLPCC 2019 evaluation task. The whole dataset consists of 81,826 samples annotated by native English speakers. 80% of them are used as training set. 10% of them are used as validation set while the rest is used as test set. 3000 hard samples are selected from the test set. Metric for this dataset is the exactly matching accuracy on both full test set and hard test subset. Each sample is composed of the question, the logical form, the parameters(entity/value/type) and question type as the Table TABREF3 demonstrates. 
 Question: Does the training dataset provide logical form supervision?
Output: Yes-no
Multi-task Pairwise Neural Ranking
We propose a multi-task pairwise neural ranking approach to better incorporate and distinguish the relative order between the candidate segmentations of a given hashtag. Our model adapts to address single- and multi-token hashtags differently via a multi-task learning strategy without requiring additional annotations. In this section, we describe the task setup and three variants of pairwise neural ranking models (Figure FIGREF11 ). Pairwise Neural Ranking Model Margin Ranking (MR) Loss Adaptive Multi-task Learning 
 Question: What set of approaches to hashtag segmentation are proposed?
Output: Extractive
We defined a sequence labeling task to extract custom entities from user input. We assumed seven (7) possible entities (see Table TABREF43) to be recognized by the model: topic, subtopic, examination mode and level, question number, intent, as well as the entity other for remaining words in the utterance.   We defined a classification problem to predict the system's next action according to the given user input. We assumed 13 custom actions (see Table TABREF42) that we considered being our labels. In the conversational dataset, each input was automatically labeled by the rule-based system with the corresponding next action and the dialogue-id. Thus, no additional post-labeling was required.  
 Question: How does the IPA label data after interacting with users?
Output: Abstractive
The data stems from a joint ADAPT-Microsoft project.  
 Question: what dataset was used?
Output: Abstractive
We compared our model with MLE, RL and GAN baselines. Since COCO and EMNLP2017 WMT don't have input while WeiboDial regards posts as input, we chose the following baselines respectively:

MLE: a RNN model trained with MLE objective BIBREF4 . Its extension, Seq2Seq, can work on the dialogue dataset BIBREF2 .

SeqGAN: The first text GAN model that updates the generator with policy gradient based on the rewards from the discriminator BIBREF7 .

LeakGAN: A variant of SeqGAN that provides rewards based on the leaked information of the discriminator for the generator BIBREF11 .

MaliGAN: A variant of SeqGAN that optimizes the generator with a normalized maximum likelihood objective BIBREF8 .

IRL: This inverse reinforcement learning method replaces the discriminator with a reward approximator to provide dense rewards BIBREF12 .

RAML: A RL approach to incorporate MLE objective into RL training framework, which regards BLEU as rewards BIBREF17 .

DialogGAN: An extension of SeqGAN tuned to dialogue generation task with MLE objective added to the adversarial objective BIBREF16 .

DPGAN: A variant of DialogGAN which uses a language model based discriminator and regards cross-entropy as rewards BIBREF13 . 
 Question: What GAN models were used as baselines to compare against?
Output: Extractive
We first pre-train our generator on the Gutenberg dataset BIBREF24 for 20 epochs and then fine-tune BIBREF19 them to our target datasets with a language modeling objective. 
 Question: What objective function is used in the GAN?
Output: Extractive
Wikilinks can be seen as a large-scale, naturally-occurring, crowd-sourced dataset where thousands of human annotators provide ground truths for mentions of interest. This means that the dataset contains various kinds of noise, especially due to incoherent contexts. We prepare our dataset from the local-context version of Wikilinks, and resolve ground-truth links using a Wikipedia dump from April 2016. We use the page and redirect tables for resolution, and keep the database pageid column as a unique identifier for Wikipedia entities. We discard mentions where the ground-truth could not be resolved (only 3% of mentions). 
 Question: How was a quality control performed so that the text is noisy but the annotations are accurate?
Output: Abstractive
Finally, we transform text to be classified into scalars representing their distance from the constructed hate vector and use these as input to a Random Forest classifier. 
 Question: What classifier did they use?
Output: Extractive
To show the effectiveness of our approach, we show results on the SICK dataset BIBREF1, a common benchmark for logic-based NLI, and find MonaLog to be competitive with more complicated logic-based approaches (many of which require full semantic parsing and more complex logical machinery). 
 Question: Do they beat current state-of-the-art on SICK?
Output: Yes-no
Below, we explore different ways to back off when the ScRNN predicts UNK (a frequent outcome for rare and unseen words):

Pass-through: word-recognizer passes on the (possibly misspelled) word as is.

Backoff to neutral word: Alternatively, noting that passing $\colorbox {gray!20}{\texttt {UNK}}$ -predicted words through unchanged exposes the downstream model to potentially corrupted text, we consider backing off to a neutral word like `a', which has a similar distribution across classes.

Backoff to background model: We also consider falling back upon a more generic word recognition model trained upon a larger, less-specialized corpus whenever the foreground word recognition model predicts UNK. Figure 1 depicts this scenario pictorially. 
 Question: How do the backoff strategies work?
Output: Abstractive
In a recent work on sentiment analysis in Turkish BIBREF10, they learn embeddings using Turkish social media. They use the word2vec algorithm, create several unsupervised hand-crafted features, generate document vectors and feed them as input into the support vector machines (SVM) approach. We outperform this baseline approach using more effective word embeddings and supervised hand-crafted features. 
 Question: What baseline method is used?
Output: Abstractive
Our dataset is annotated based on Chinese pathology reports provided by the Department of Gastrointestinal Surgery, Ruijin Hospital. It contains 17,833 sentences, 826,987 characters and 2,714 question-answer pairs. 
 Question: How many questions are in the dataset?
Output: Abstractive
Specifically, we group the models based on the objective function it optimizes. We believe this work can aid the understanding of the existing literature.  We believe that the performance of these models is highly dependent on the objective function it optimizes - predicting adjacent word (within-tweet relationships), adjacent tweet (inter-tweet relationships), the tweet itself (autoencoder), modeling from structured resources like paraphrase databases and weak supervision.  
 Question: How do they encourage understanding of literature as part of their objective function?
Output: Abstractive
In ConvE, only $v_h$ and $v_r$ are reshaped and then concatenated into an input matrix which is fed to the convolution layer 
 Question: Did the authors try stacking multiple convolutional layers?
Output: Yes-no
The policies trained with the NUS achieved an average success rate (SR) of 94.0% and of 96.6% when tested on the ABUS and the NUS, respectively. 
 Question: by how much did nus outperform abus?
Output: Abstractive
More than 2,100 texts were paired with 15 questions each, resulting in a total number of approx. 32,000 annotated questions. For 13% of the questions, the workers did not agree on one of the 4 categories with a 3 out of 5 majority, so we did not include these questions in our dataset.

The distribution of category labels on the remaining 87% is shown in Table TABREF10 . 14,074 (52%) questions could be answered. Out of the answerable questions, 10,160 could be answered from the text directly (text-based) and 3,914 questions required the use of commonsense knowledge (script-based). After removing 135 questions during the validation, the final dataset comprises 13,939 questions, 3,827 of which require commonsense knowledge (i.e. 27.4%). This ratio was manually verified based on a random sample of questions. 
 Question: what dataset statistics are provided?
Output: Abstractive
As indicated in its name, Recurrent Deep Stacking Network stacks and concatenates the outputs of previous frames into the input features of the current frame. 
 Question: What does recurrent deep stacking network do?
Output: Abstractive
Subsequently, an LSTM decoder generates the characters in the output word form using encoder states and an attention mechanism. 
 Question: What architecture does the decoder have?
Output: Extractive
Our unsupervised ranking model outperforms the supervised IMS system by 1.02% on the CoNLL F1 score, and achieves competitive performance with the latent tree model. Moreover, our approach considerably narrows the gap to other supervised systems listed in Table 3 . 
 Question: Is the model presented in the paper state of the art?
Output: Abstractive
Attention-base translation model We use the system of BIBREF6 , a convolutional sequence to sequence model. 
 Question: Which translation system do they use to translate to English?
Output: Abstractive
In order to achieve these configuration the TIMIT data was split. Fig. FIGREF12 illustrates the split of the data into 8 subsets (A-H). The TIMIT dataset contains speech from 462 speakers in training and 168 speakers in the test set, with 8 utterances for each speaker. The TIMIT training and test set are split into 8 blocks, where each block contains 2 utterances per speaker, randomly chosen. Thus each block A,B,C,D contains data from 462 speakers with 924 utterances taken from the training sets, and each block E,F,G,H contains speech from 168 test set speakers with 336 utterances.

For Task a training of embeddings and the classifier is identical, namely consisting of data from blocks (A+B+C+E+F+G). The test data is the remainder, namely blocks (D+H). For Task b the training of embeddings and classifiers uses (A+B+E+F) and (C+G) respectively, while again using (D+H) for test. Task c keeps both separate: embeddings are trained on (A+B+C+D), classifiers on (E+G) and tests are conducted on (F+H). Note that H is part of all tasks, and that Task c is considerably easier as the number of speakers to separate is only 168, although training conditions are more difficult. 
 Question: What TIMIT datasets are used for testing?
Output: Abstractive
machine comprehension  Nelufar  
 Question: What MC abbreviate for?
Output: Abstractive
We compare classification and regression approaches and show that classification produces better results than regression but the quality of the results depends on the approach followed to annotate the data labels. 
 Question: Did classification models perform better than previous regression one?
Output: Yes-no
In this paper, we introduce automatic `drunk-texting prediction' as a computational task. Given a tweet, the goal is to automatically identify if it was written by a drunk user.  
 Question: Do the authors equate drunk tweeting with drunk texting? 
Output: Yes-no
We present two use cases on which Seshat was developped: clinical interviews, and daylong child-centered recordings. 
 Question: Did they experiment with the tool?
Output: Yes-no
Results of the DQN-based agent are presented in fig: scenario comparison. Each plot depicts the average reward (across 5 seeds) of all representations methods. It can be seen that the NLP representation outperforms the other methods.  NLP representations remain robust to changes in the environment as well as task-nuisances in the state.  
 Question: What result from experiments suggest that natural language based agents are more robust?
Output: Abstractive
We compile three Chinese text corpora from online data for three domains, namely, “hotel", “mobile phone (mobile)", and “travel". All texts are about user reviews.  
 Question: What are the sources of the data?
Output: Abstractive
Posts in FBFans dataset are used for this analysis. We calculate the like statistics of each distinct author from these 32,595 posts. 
 Question: What is the size of the Chinese data?
Output: Extractive
In Figure FIGREF32 , we visualize the syntactic distance estimated by the Parsing Network, while reading three different sequences from the PTB test set. We observe that the syntactic distance tends to be higher between the last character of a word and a space, which is a reasonable breakpoint to separate between words.  The model autonomously discovered to avoid inter-word attention connection, and use the hidden states of space (separator) tokens to summarize previous information. This is strong proof that the model can understand the latent structure of data. 
 Question: How do they show their model discovers underlying syntactic structure?
Output: Abstractive
We first use state-of-the-art PDTB taggers for our baseline BIBREF13 , BIBREF12 for the evaluation of the causality prediction of our models ( BIBREF12 requires sentences extracted from the text as its input, so we used our parser to extract sentences from the message). 
 Question: What baselines did they consider?
Output: Extractive
The annotator carried out all annotation. 
 Question: How many annotators tagged each tweet?
Output: Abstractive
In future work we also intend to add these types of studies to the ERP predictions.

Discussion 
 Question: Which two pairs of ERPs from the literature benefit from joint training?
Output: Abstractive
We introduce our proposed diversity, density, and homogeneity metrics with their detailed formulations and key intuitions. 
 Question: Did they propose other metrics?
Output: Yes-no
Looking to other target tasks, the grammar-related CoLA task benefits dramatically from ELMo pretraining: The best result without language model pretraining is less than half the result achieved with such pretraining. In contrast, the meaning-oriented textual similarity benchmark STS sees good results with several kinds of pretraining, but does not benefit substantially from the use of ELMo. 
 Question: Do some pretraining objectives perform better than others for sentence level understanding tasks?
Output: Yes-no
The crowdworkers were located in the US and hired on the BIBREF22 platform. 
 Question: Who are the crowdworkers?
Output: Abstractive
As shown in Table TABREF3 , we collect Amazon review keywords for 2,896 e-books (publishers: Kiwi, Rowohlt, Fischer, and Droemer), which leads to 33,663 distinct review keywords and on average 30 keyword assignments per e-book. 
 Question: how large is the vocabulary?
Output: Extractive
We collected Japanese-Vietnamese parallel data from TED talks extracted from WIT3's corpus BIBREF15 .  
 Question: did they collect their own data?
Output: Yes-no
We describe our rules for WikiSQL here. Our rule for KBQA is simple without using a curated mapping dictionary. The pipeline of rules in SequentialQA is similar to that of WikiSQL. 
 Question: Are the rules dataset specific?
Output: Yes-no
Following previous studies BIBREF1, we collect event-related microposts from Twitter using 11 and 8 seed events (see Section SECREF2) for CyberAttack and PoliticianDeath, respectively. Unlabeled microposts are collected by using the keyword `hack' for CyberAttack, while for PoliticianDeath, we use a set of keywords related to `politician' and `death' (such as `bureaucrat', `dead' etc.) 
 Question: Do they report results only on English data?
Output: Yes-no
On the same architecture, our RCRN outperforms ablative baselines BiLSTM by INLINEFORM2 and 3L-BiLSTM by INLINEFORM3 on average across 16 datasets. 
 Question: By how much do they outperform BiLSTMs in Sentiment Analysis?
Output: Abstractive
For annotation purposes, we created snippets of conversations as the ones shown in Example 1 and Example 2 consisting of the parent of the suspected trolling event, the suspected trolling event comment, and all of the direct responses to the suspected trolling event. However, the trade off is that snippets like this allow us to make use of Amazon Mechanical Turk (AMT) to have the dataset annotated, because it is not a big burden for a “turker” to work on an individual snippet in exchange for a small pay, and expedites the annotation process by distributing it over dozens of people. Specifically, for each snippet, we requested three annotators to label the four aspects previously described. 
 Question: how was annotation done?
Output: Abstractive
Los resultados de la evaluación se presentan en la Tabla TABREF42, en la forma de promedios normalizados entre [0,1] y de su desviación estándar $\sigma $. 
 Question: What evaluation metrics did they look at?
Output: Abstractive
Data was collected from a 10% uniform sample of Twitter posts made during 2013, specifically the Gardenhose API. Control documents were also selected. These documents did not contain any of `caused', `causing', or `causes', nor any bidirectional words, and are further matched temporally to obtain the same number of control documents as causal documents in each fifteen-minute period during 2013. Control documents were otherwise selected randomly; causal synonyms may be present.  
 Question: How do they collect the control corpus?
Output: Abstractive
In order to evaluate the models trained on generated data, we manually annotated a named entities dataset comprising 53453 tokens and 2566 sentences selected from over 250 news texts from ilur.am. During annotation, we generally relied on categories and guidelines assembled by BBN Technologies for TREC 2002 question answering track. 
 Question: did they use a crowdsourcing platform for manual annotations?
Output: Yes-no
The Gallup survey asked for opinions on legality of “homosexual relations" until 2008, but then changed the wording to “gay and lesbian relations". This was likely because many people who identify as gay and lesbian find the word homosexual to be outdated and derogatory. 
 Question: Do they analyze specific derogatory words?
Output: Yes-no
For each qualifying diagnosis tweet we retrieve the timeline of the corresponding Twitter user using the Twitter user_timeline API endpoint . Subsequently, we remove all non-English tweets (Twitter API machine-detected“lang” field), all retweets, and tweets that contain “diagnos*” or “depress*”, but not a valid diagnosis statement. The resulting Depressed cohort contains 1,207 individuals and 1,759,644 tweets ranging from from May 2008 to September 2018. 
 Question: Do they report results only on English datasets?
Output: Yes-no
Three different datasets have been used to train our models: the Toronto book corpus, Wikipedia sentences and tweets.  Our Sent2Vec models also on average outperform or are at par with the C-PHRASE model, despite significantly lagging behind on the STS 2014 WordNet and News subtasks. This observation can be attributed to the fact that a big chunk of the data that the C-PHRASE model is trained on comes from English Wikipedia, helping it to perform well on datasets involving definition and news items.  
 Question: Do they report results only on English data?
Output: Yes-no
Secondly, a well-known problem of uneven yet unknown distribution of word senses is alleviated by modifying a naïve Bayesian classifier. Thanks to this correction, the classifier is no longer biased towards senses that have more training data. Finally, the aggregated feature values corresponding to target word senses are used to build a naïve Bayesian classifier adjusted to a situation of unknown a priori probabilities. 
 Question: How do they deal with unknown distribution senses?
Output: Abstractive
The question is encoded with a 1024-dimensional LSTM model that takes in a one-hot descriptor of each word in the question. The image is described with the 4096-dimensional output from the last fully connected layer of the Convolutional Neural Network (CNN), VGG16 BIBREF25 . The system performs an element-wise multiplication of the image and question features, after linearly transforming the image descriptor to 1024 dimensions. The final layer of the architecture is a softmax layer. 
 Question: What is the model architecture used?
Output: Abstractive
We asked for two distinct paraphrases of each sentence because we believe that a good sentence embedding should put paraphrases close together in vector space.

Several modification types were specifically selected to constitute a thorough test of embeddings. In different meaning, the annotators should create a sentence with some other meaning using the same words as the original sentence. Other transformations which should be difficult for embeddings include minimal change, in which the sentence meaning should be significantly changed by using only very small modification, or nonsense, in which words of the source sentence should be shuffled so that it is grammatically correct, but without any sense. 
 Question: What annotations are available in the dataset?
Output: Abstractive
The overall Total Accuracy score reported in table TABREF19 using the entire feature set is 549.  
 Question: Do they experiment with the dataset?
Output: Yes-no
While it is obvious that our embeddings can be used as features for new predictive models, it is also very easy to incorporate our learned Dolores embeddings into existing predictive models on knowledge graphs. The only requirement is that the model accepts as input, an embedding layer (for entities and relations). If a model fulfills this requirement (which a large number of neural models on knowledge graphs do), we can just use Dolores embeddings as a drop-in replacement. We just initialize the corresponding embedding layer with Dolores embeddings. In our evaluation below, we show how to improve several state-of-the-art models on various tasks simply by incorporating Dolores as a drop-in replacement to the original embedding layer. 
 Question: How are meaningful chains in the graph selected?
Output: Yes-no
BLEU: We use the Bilingual Evaluation Understudy (BLEU) BIBREF34 metric which is commonly used in machine translation tasks. The BLEU metric can be used to evaluate dialogue generation models as in BIBREF5, BIBREF35. The BLEU metric is a word-overlap metric which computes the co-occurrence of N-grams in the reference and the generated response and also applies the brevity penalty which tries to penalize far too short responses which are usually not desired in task-oriented chatbots. We compute the BLEU score using all generated responses of our systems.

Per-turn Accuracy: Per-turn accuracy measures the similarity of the system generated response versus the target response. Eric and Manning eric2017copy used this metric to evaluate their systems in which they considered their response to be correct if all tokens in the system generated response matched the corresponding token in the target response. This metric is a little bit harsh, and the results may be low since all the tokens in the generated response have to be exactly in the same position as in the target response.

Per-Dialogue Accuracy: We calculate per-dialogue accuracy as used in BIBREF8, BIBREF5. For this metric, we consider all the system generated responses and compare them to the target responses. A dialogue is considered to be true if all the turns in the system generated responses match the corresponding turns in the target responses. Note that this is a very strict metric in which all the utterances in the dialogue should be the same as the target and in the right order.

F1-Entity Score: Datasets used in task-oriented chores have a set of entities which represent user preferences. For example, in the restaurant domain chatbots common entities are meal, restaurant name, date, time and the number of people (these are usually the required entities which are crucial for making reservations, but there could be optional entities such as location or rating). Each target response has a set of entities which the system asks or informs the user about. Our models have to be able to discern these specific entities and inject them into the generated response. To evaluate our models we could use named-entity recognition evaluation metrics BIBREF36. The F1 score is the most commonly used metric used for the evaluation of named-entity recognition models which is the harmonic average of precision and recall of the model. We calculate this metric by micro-averaging over all the system generated responses. 
 Question: Is human evaluation performed?
Output: Yes-no
In addition, in order to estimate annotation reliability and provide for better evaluation, every question in the test set is answered by at least two additional experts. 
 Question: Are the answers double (and not triple) annotated?
Output: Yes-no
We carried out our experiments on the Universal Dependencies v1.2 treebanks BIBREF21 , hereafter UD1.2, from which morphosyntactically annotated corpora can be trivially extracted.  We considered UD1.2 corpora for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish.  
 Question: which datasets did they experiment with?
Output: Abstractive
The main reason is that when the controllability is strong, the change of selection will directly affect the text realization so that a tiny error of content selection might lead to unrealistic text. If the selector is not perfectly trained, the fluency will inevitably be influenced. When the controllability is weaker, like in RS, the fluency is more stable because it will not be affected much by the selection mask. 
 Question: Does the performance necessarily drop when more control is desired?
Output: Yes-no
We use standard Precision, Recall and F1 at mention level (Micro) and at the document level (Macro) as measurements. We can see the average linking precision (Micro) of WW is lower than that of TAC2010, and NCEL outperforms all baseline methods in both easy and hard cases. 
 Question: How do they verify generalization ability?
Output: Abstractive
Practical evaluation of GTD is currently only possible on synthetic data. We construct a range of datasets designed for image captioning evaluation. We call this diagnostic evaluation benchmark ShapeWorldICE (ShapeWorld for Image Captioning Evaluation). We illustrate the evaluation of specific image captioning models on ShapeWorldICE. 
 Question: Are the images from a specific domain?
Output: Yes-no
Further, FSDM has a new module called response slot binary classifier that adds extra supervision to generate the slots that will be present in the response more precisely before generating the final textual agent response (see Section "Methodology" for details). 
 Question: How do slot binary classifiers improve performance?
Output: Abstractive
Given a news Twitter account, we read its tweets from the account's timeline. Then we sort the tweets by the posting date in ascending way and we split them into $N$ chunks. Each chunk consists of a sorted sequence of tweets labeled by the label of its corresponding account. 
 Question: Was the approach used in this work to detect fake news fully supervised?
Output: Yes-no
For each model, we examined word-level perplexity, R@3 in next-word prediction, latency (ms/q), and energy usage (mJ/q). To explore the perplexity-recall relationship, we collected individual perplexity and recall statistics for each sentence in the test set. 
 Question: What aspects have been compared between various language models?
Output: Abstractive
Table TABREF15 shows a comparison of the results on SimCluster versus K-means algorithm. Here our SimCluster algorithm improves the F1-scores from 0.412 and 0.417 in the two domains to 0.442 and 0.441. The ARI scores also improve from 0.176 and 0.180 to 0.203 and 0.204. 
 Question: Do they use the same distance metric for both the SimCluster and K-means algorithm?
Output: Yes-no
We consider two types of architectures: a bi-directional language model to augment the sequence to sequence encoder and a uni-directional model to augment the decoder. Both use self-attention BIBREF16 and the uni-directional model contains INLINEFORM0 transformer blocks, followed by a word classifier to predict the next word on the right. bi-directional language model to augment the sequence to sequence encoder 
 Question: What language model architectures are used?
Output: Extractive
We extracted 200 sentence pairs from BIBREF3 's dataset and provided each pair with a document context consisting of a preceding and a following sentence, as in the following example. 
 Question: What document context was added?
Output: Abstractive
The GENIA Corpus BIBREF3 contains biomedical abstracts from the PubMed database. We use GENIA technical term annotations 3.02, which cover linguistic expressions to entities of interest in molecular biology, e.g. proteins, genes and cells. CoNLL2003 BIBREF14 is a standard NER dataset based on the Reuters RCV-1 news corpus. It covers named entities of type person, location, organization and misc.

For testing the overall annotation performance, we utilize CoNLL2003-testA and a 50 document split from GENIA.  
 Question: what standard dataset were used?
Output: Extractive
The incomplete dataset used for training is composed of lower-cased incomplete data obtained by manipulating the original corpora. 
 Question: Do they test their approach on a dataset without incomplete data?
Output: Yes-no
These actions consist of the following components:

[leftmargin=*]

Co-Reference Resolution: To support multi-turn interactions, it is sometimes necessary to use co-reference resolution techniques for effective retrieval. Query Generation: This component generates a query based on the past user-system interactions. Retrieval Model: This is the core ranking component that retrieves documents or passages from a large collection. Result Generation: The retrieved documents can be too long to be presented using some interfaces. 
 Question: What are the different modules in Macaw?
Output: Extractive
We evaluated our models on 3 different datasets:

CSAT dataset for CSAT prediction, consisting of spoken transcripts (automatic via ASR).

20 newsgroups for topic identification task, consisting of written text;

Fisher Phase 1 corpus for topic identification task, consisting of spoken transcripts (manual); 
 Question: What datasets did they use for evaluation?
Output: Extractive
The straight line in figures FIGREF39, FIGREF43 and FIGREF51 is the result of a supervised LDA algorithm which is used as a baseline.  
 Question: Was performance of the weakly-supervised model compared to the performance of a supervised model?
Output: Yes-no
We trained word embeddings using either GloVe BIBREF11 or SGNS BIBREF12 on a small or a large corpus. 
 Question: What types of word representations are they evaluating?
Output: Abstractive
The basic concept is to use a phone-discriminative model to produce frame-level phonetic features, and then use these features to enhance RNN LID systems that were originally built with raw acoustic features. The initial step is therefore feature combination, with the phonetic feature used as auxiliary information to assist acoustic RNN LID. This is improved further, as additional research identified that a simpler model using only the phonetic feature as the RNN LID input provides even better performance. We call this RNN model based on phonetic features the phonetic temporal neural LID approach, or PTN LID. As well as having a simplified model structure, the PTN offers deeper insight into the LID task by rediscovering the value of the phonetic temporal property in language discrimination.  
 Question: What is the main contribution of the paper? 
Output: Abstractive
SIGHAN Bakeoff defines two types of evaluation settings, closed test limits all the data for learning should not be beyond the given training set, while open test does not take this limitation BIBREF21. 
 Question: What is meant by closed test setting?
Output: Extractive
MTMSN BIBREF4 is the first, and only model so far, that specifically tried to tackle the multi-span questions of DROP. Their approach consisted of two parts. The first was to train a dedicated categorical variable to predict the number of spans to extract. The second was to generalize the single-span head method of extracting a span, by utilizing the non-maximum suppression (NMS) algorithm BIBREF7 to find the most probable set of non-overlapping spans. The number of spans to extract was determined by the aforementioned categorical variable 
 Question: What approach did previous models use for multi-span questions?
Output: Abstractive
We collect more human-written scenes for each concept-set in dev and test set through crowd-sourcing via the Amazon Mechanical Turk platform. Each input concept-set is annotated by at least three different humans. The annotators are also required to give sentences as the rationales, which further encourage them to use common sense in creating their scenes. 
 Question: Are the rationales generated after the sentences were written?
Output: Yes-no
We also run a manual evaluation which shows for the En-It task a slight quality degradation in exchange of a statistically significant reduction in the average length ratio, from 1.05 to 1.01. 
 Question: Do they conduct any human evaluation?
Output: Yes-no
We employ BIBREF73 's weighted lexicon of terms that uses the dataset of 75,394 Facebook users who shared their status, age and gender. Utilizing these two weighted lexicon of terms, we are predicting the demographic information (age or gender) of INLINEFORM0 (denoted by INLINEFORM1 ) using following equation: INLINEFORM2

where INLINEFORM0 is the lexicon weight of the term, and INLINEFORM1 represents the frequency of the term in the user generated INLINEFORM2 , and INLINEFORM3 measures total word count in INLINEFORM4 . 
 Question: How do this framework facilitate demographic inference from social media?
Output: Abstractive
In natural language processing (NLP), taking into account the morphological complexity inherent to each language could be important for improving or adapting the existing methods, since the amount of semantic and grammatical information encoded at the word level, may vary significantly from language to language. This information could be useful for linguistic analysis and for measuring the impact of different word form normalization tools depending of the language. 
 Question: what is the practical application for this paper?
Output: Abstractive
Moreover, to be able to make use of multiple answer styles within a single system, our model introduces an artificial token corresponding to the target style at the beginning of the answer sentence ( $y_1$ ), like BIBREF14 . At test time, the user can specify the first token to control the answer styles. 
 Question: Does their model also take the expected answer style as input?
Output: Yes-no
Our approach improves LR by 5.17% (Accuracy) and 18.38% (AUC), and MLP by 10.71% (Accuracy) and 30.27% (AUC) on average. Such significant improvements clearly demonstrate that our approach is effective at improving model performance. 
 Question: How are the accuracy merits of the approach demonstrated?
Output: Extractive