niryuu/sni-each-converted huggingface dataset:
Probing Methodology and Modeling ::: Task Definition and Modeling ::: Baselines and Sanity Checks.
When creating synthetic datasets, it is important to ensure that systematic biases, or annotation artifacts BIBREF41, are not introduced into the resulting probes and that the target datasets are sufficiently challenging (or good, in the sense of BIBREF42). To test for this, we use several of the MCQA baseline models first introduced in BIBREF0, which take inspiration from the LSTM-based models used in BIBREF43 for NLI and various partial-input baselines based on these models. 
 Question: How do they control for annotation artificats?
Output:  we use several of the MCQA baseline models first introduced in BIBREF0
We collect data by using the Twitter API for saved data, available for public use. For our experiments we collect 3200 tweets filtered by keywords like “fire”, “earthquake”, “theft”, “robbery”, “drunk driving”, “drunk driving accident” etc. Later, we manually label tweets with and labels for classification as stage one. Our dataset contains 1313 tweet with positive label and 1887 tweets with a negative label . We create another dataset with the positively labeled tweets and provide them with category labels like “fire”, “accident”, “earthquake” etc. 
 Question: Are the tweets specific to a region?
Output: No
Hence we have determined our targets as Galatasaray (namely Target-1) and Fenerbahçe (namely, Target-2) which are two of the most popular football clubs in Turkey. 
 Question: Which sports clubs are the targets?
Output: Galatasaray Fenerbahçe
However, a significant bottleneck that has been overlooked by previous efforts is the input vocabulary size and its corresponding word embedding matrix, often accounting for a significant proportion of all model parameters. For instance, the embedding table of the BERTBASE model, comprising over 30K WordPiece tokens BIBREF14, accounts for over $21\%$ of the model size. While there has been existing work on reducing NLP model vocabulary sizes BIBREF15, distillation techniques cannot utilize these, since they require the student and teacher models to share the same vocabulary and output space. This profoundly limits their potential to further reduce model sizes. 
 Question: Why are prior knowledge distillation techniques models are ineffective in producing student models with vocabularies different from the original teacher models?  
Output: While there has been existing work on reducing NLP model vocabulary sizes BIBREF15, distillation techniques cannot utilize these, since they require the student and teacher models to share the same vocabulary and output space. This profoundly limits their potential to further reduce model sizes.
The whole dataset consists of 81,826 samples annotated by native English speakers. 80% of them are used as training set. 10% of them are used as validation set while the rest is used as test set. 3000 hard samples are selected from the test set. 
 Question: What is the difference between the full test set and the hard test set?
Output: 3000 hard samples are selected from the test set
To identify legally sound answers, we recruit seven experts with legal training to construct answers to Turker questions.  
 Question: Who were the experts used for annotation?
Output: Individuals with legal training
Table IV shows the result of our model for sentiment classification against other models. We compare our model performance with the approaches of BIBREF0 BIBREF5 on STS Corpus. BIBREF0 reported the results of Maximum Entropy (MaxEnt), NB, SVM on STS Corpus having good performance in previous time. The model of BIBREF5 is a state-of-the-art so far by using a CharSCNN. As can be seen, 86.63 is the best prediction accuracy of our model so far for the STS Corpus.

For Sanders and HCR datasets, we compare results with the model of BIBREF14 that used a ensemble of multiple base classifiers (ENS) such as NB, Random Forest (RF), SVM and Logistic Regression (LR). The ENS model is combined with bag-of-words (BoW), feature hashing (FH) and lexicons. The model of BIBREF14 is a state-of-the-art on Sanders and HCR datasets. Our models outperform the model of BIBREF14 for the Sanders dataset and HCR dataset.

 
 Question: What was the baseline?
Output: We compare our model performance with the approaches of BIBREF0 BIBREF5 on STS Corpus. BIBREF0 reported the results of Maximum Entropy (MaxEnt), NB, SVM on STS Corpus having good performance in previous time. The model of BIBREF5 is a state-of-the-art so far by using a CharSCNN. we compare results with the model of BIBREF14 that used a ensemble of multiple base classifiers (ENS) such as NB, Random Forest (RF), SVM and Logistic Regression (LR). The ENS model is combined with bag-of-words (BoW), feature hashing (FH) and lexicons. The model of BIBREF14 is a state-of-the-art on Sanders and HCR datasets.
As fig:fit shows, estimated test accuracy is highly correlated with actual test accuracy for various datasets, with worst-case values $\mu <1\%$ and $\sigma <5\%$ . Note that the number of free parameters is small ($||\le 6$) compared to the number of points (42–49 model-data configurations), demonstrating the appropriateness of the proposed function for modeling the complex error landscape. 
 Question: What is proof that proposed functional form approximates well generalization error in practice?
Output: estimated test accuracy is highly correlated with actual test accuracy for various datasets appropriateness of the proposed function for modeling the complex error landscape
Second, we compute three metrics on the extracted information:

$\bullet $ Relation Generation (RG) estimates how well the system is able to generate text containing factual (i.e., correct) records. We measure the precision and absolute number (denoted respectively RG-P% and RG-#) of unique relations $r$ extracted from $\hat{y}_{1:T}$ that also appear in $s$.

$\bullet $ Content Selection (CS) measures how well the generated document matches the gold document in terms of mentioned records. We measure the precision and recall (denoted respectively CS-P% and CS-R%) of unique relations $r$ extracted from $\hat{y}_{1:T}$ that are also extracted from $y_{1:T}$.

$\bullet $ Content Ordering (CO) analyzes how well the system orders the records discussed in the description. We measure the normalized Damerau-Levenshtein distance BIBREF36 between the sequences of records extracted from $\hat{y}_{1:T}$ that are also extracted from $y_{1:T}$. 
 Question: Which qualitative metric are used for evaluation?
Output:  Relation Generation (RG)  Content Selection (CS)  Content Ordering (CO)
In Table TABREF1, we summarize the quantitative results of the above previous studies. In Table TABREF1, we summarize the quantitative results of the above previous studies. 
 Question: What is the accuracy reported by state-of-the-art methods?
Output: Answer with content missing: (Table 1)
Previous state-of-the art on same dataset: ResNet50 89% (6 languages), SVM-HMM 70% (4 languages)
This dataset was collected through Mechanical Turk using 100 simulated environments with a corresponding topological map and, to the best of our knowledge, it is the first of its kind for behavioral navigation.  We created a new dataset for the problem of following navigation instructions under the behavioral navigation framework of BIBREF5 . This dataset was created using Amazon Mechanical Turk and 100 maps of simulated indoor environments, each with 6 to 65 rooms. To the best of our knowledge, this is the first benchmark for comparing translation models in the context of behavioral robot navigation.

As shown in Table TABREF16 , the dataset consists of 8066 pairs of free-form natural language instructions and navigation plans for training. This training data was collected from 88 unique simulated environments, totaling 6064 distinct navigation plans (2002 plans have two different navigation instructions each; the rest has one). The dataset contains two test set variants:

While the dataset was collected with simulated environments, no structure was imposed on the navigation instructions while crowd-sourcing data. Thus, many instructions in our dataset are ambiguous. Moreover, the order of the behaviors in the instructions is not always the same. For instance, a person said “turn right and advance” to describe part of a route, while another person said “go straight after turning right” in a similar situation. The high variability present in the natural language descriptions of our dataset makes the problem of decoding instructions into behaviors not trivial. See Appendix A of the supplementary material for additional details on our data collection effort.

 
 Question: How were the navigation instructions collected?
Output: using Amazon Mechanical Turk using simulated environments with topological maps
This work presented MVCNN, a novel CNN architecture for sentence classification. It combines multichannel initialization – diverse versions of pretrained word embeddings are used – and variable-size filters – features of multigranular phrases are extracted with variable-size convolution filters.  
 Question: How is MVCNN compared to CNN?
Output: MVCNN, a novel CNN architecture for sentence classification. It combines multichannel initialization – diverse versions of pretrained word embeddings are used – and variable-size filters – features of multigranular phrases are extracted with variable-size convolution filters.
In our experiments, we used WordNet 3.0 BIBREF9 as our external knowledge base INLINEFORM0 . For word embeddings, we experimented with two popular models: (1) GloVe embeddings trained by BIBREF10 on Wikipedia and Gigaword 5 (vocab: 400K, dim: 300), and (2) w2v-gn, Word2vec BIBREF5 trained on the Google News dataset (vocab: 3M, dim: 300). Our coverage enhancement starts by transforming the knowledge base INLINEFORM0 into a vector space representation that is comparable to that of the corpus-based space INLINEFORM1 . To this end, we use two techniques for learning low-dimensional feature spaces from knowledge graphs: DeepWalk and node2vec. DeepWalk uses a stream of short random walks in order to extract local information for a node from the graph. By treating these walks as short sentences and phrases in a special language, the approach learns latent representations for each node. Similarly, node2vec learns a mapping of nodes to continuous vectors that maximizes the likelihood of preserving network neighborhoods of nodes. Thanks to a flexible objective that is not tied to a particular sampling strategy, node2vec reports improvements over DeepWalk on multiple classification and link prediction datasets. For both these systems we used the default parameters and set the dimensionality of output representation to 100. Also, note than nodes in the semantic graph of WordNet represent synsets. Hence, a polysemous word would correspond to multiple nodes. In our experiments, we use the MaxSim assumption of BIBREF11 in order to map words to synsets. 
 Question: What other embedding models are tested?
Output: GloVe embeddings trained by BIBREF10 on Wikipedia and Gigaword 5 (vocab: 400K, dim: 300) w2v-gn, Word2vec BIBREF5 trained on the Google News dataset (vocab: 3M, dim: 300) DeepWalk  node2vec
We use 300 Dimensional Glove Common Crawl Embeddings (840B Token) BIBREF11 and fine tune them for the task. 
 Question: What data are the embeddings trained on?
Output: Common Crawl
Table TABREF14 shows the effectiveness of our model (multi-task transformer) over the baseline transformer BIBREF8 .  
 Question: what are the baselines?
Output: the baseline transformer BIBREF8
Email classifier using machine learning ::: Machine learning approach ::: Feature selection
Ngrams are a continuous sequence of n items from a given sample of text. From title, body and OCR text words are selected. Ngrams of 3 nearby words are extracted with Term Frequency-Inverse Document Frequency (TF-IDF) vectorizing, then features are filtered using chi squared the feature scoring method. Email classifier using machine learning ::: Machine learning approach ::: Random forest
Random Forest is a bagging Algorithm, an ensemble learning method for classification that operates by constructing a multitude of decision trees at training time and outputting the class that has highest mean majority vote of the classesBIBREF14. Email classifier using machine learning ::: Machine learning approach ::: XGBoost
XGBoost is a decision-tree-based ensemble Machine Learning algorithm that uses a gradient boosting framework. It is used commonly in the classification problems involving unstructured dataBIBREF5. Email classifier using machine learning ::: Machine learning approach ::: Hierarchical Model
Since the number of target labels are high, achieving the higher accuracy is difficult, while keeping all the categories under same feature selection method. Some categories performs well with lower TF-IDF vectorizing range and higher n grams features even though they showed lower accuracy in the overall single model. Therefore, hierarchical machine learning models are built to classify 31 categories in the first classification model and remaining categories are named as low-accu and predicted as one category. In the next model, predicted low-accu categories are again classified into 47 categories. Comparatively this hierarchical model works well since various feature selection methods are used for various categoriesBIBREF5. 
 Question: What are all machine learning approaches compared in this work?
Output: Feature selection Random forest XGBoost Hierarchical Model
To test if the learned representations can separate phonetic categories, we use a minimal pair ABX discrimination task BIBREF19 , BIBREF20 . For the within-speaker task, all the phones triplets belong to the same speaker (e.g. $x$3 ) Finally the scores for every pair of central phones are averaged and subtracted from 1 to yield the reported within-talker ABX error rate. For the across-speaker task, $x$4 and $x$5 belong to the same speaker, and $x$6 to a different one (e.g. $x$7 ). The scores for a given minimal pair are first averaged across all of the pairs of speakers for which this contrast can be made. As above, the resulting scores are averaged over all contexts over all pairs of central phones and converted to an error rate. 
 Question: What is the metric that is measures in this paper?
Output: error rate in a minimal pair ABX discrimination task
The utterance is concatenated with a special symbol marking the end of the sequence. We initialize our word embeddings using 300-dimensional GloVe BIBREF30 and then fine-tune them during training. 
 Question: Do they use pretrained word vectors for dialogue context embedding?
Output: Yes
In this study, we concentrate our effort on re-assessing keyphrase extraction performance on three increasingly sophisticated levels of document preprocessing described below. 
 Question: what keyphrase extraction models were reassessed?
Output: Answer with content missing: (LVL1, LVL2, LVL3) 
- Stanford CoreNLP
- Optical Character Recognition (OCR) system, ParsCIT 
- further abridge the input text from level 2 preprocessed documents to the following: title, headers, abstract, introduction, related work, background and conclusion.
One-Vs-The-Rest strategy was adopted for the task of multi-class classification and I reported F-score, micro-F, macro-F and weighted-F scores using 10-fold cross-validation.  
 Question: What metrics are considered?
Output: F-score micro-F macro-F weighted-F
We used state-of-the-art features that have shown to be useful in ID: some of them are language-independent (e.g., punctuation marks, positive and negative emoticons, quotations, personal pronouns, tweet's length, named entities) while others are language-dependent relying on dedicated lexicons (e.g., negation, opinion lexicons, opposition words). 
 Question: What text-based features are used?
Output: language-independent (e.g., punctuation marks, positive and negative emoticons, quotations, personal pronouns, tweet's length, named entities)  language-dependent relying on dedicated lexicons (e.g., negation, opinion lexicons, opposition words)
Co-Reference Resolution: To support multi-turn interactions, it is sometimes necessary to use co-reference resolution techniques for effective retrieval. In Macaw, we identify all the co-references from the last request of user to the conversation history. The same co-reference resolution outputs can be used for different query generation components. This can be a generic or action-specific component.

Query Generation: This component generates a query based on the past user-system interactions. The query generation component may take advantage of co-reference resolution for query expansion or re-writing.

Retrieval Model: This is the core ranking component that retrieves documents or passages from a large collection. Macaw can retrieve documents from an arbitrary document collection using the Indri python interface BIBREF9, BIBREF10. We also provide the support for web search using the Bing Web Search API. Macaw also allows multi-stage document re-ranking.

Result Generation: The retrieved documents can be too long to be presented using some interfaces. Result generation is basically a post-processing step ran on the retrieved result list. In case of question answering, it can employ answer selection or generation techniques, such as machine reading comprehension models. For example, Macaw features the DrQA model BIBREF11 for question answering. 
 Question: What functionality does Macaw provide?
Output: Co-Reference Resolution Query Generation Retrieval Model Result Generation
Paraphrases can be obtained by translating an English string into a foreign language and then back-translating it into English.  
 Question: It looks like learning to paraphrase questions, a neural scoring model and a answer selection model cannot be trained end-to-end. How are they trained?
Output: using multiple pivot sentences
The NLG model is a seq2seq model with attention as described in section SECREF2. 
 Question: Do they use attention?
Output: Yes
Our method is purely text-based, and ignores the publication date and the source of the article. It combines task-specific embeddings, produced by a two-level attention-based deep neural network model, with manually crafted features (stylometric, lexical, grammatical, and semantic), into a kernel-based SVM classifier. 
 Question: what types of features were used?
Output: stylometric, lexical, grammatical, and semantic
This paper used the real-time method to randomly collect 10% of publicly available English tweets using several pre-defined DDEO-related queries (Table TABREF6 ) within a specific time frame.  
 Question: Do they evaluate only on English data?
Output: Yes
We compare the results of our model to a baseline multi-task architecture inspired by yang2016multi. In our baseline model there are no explicit connections between tasks - the only shared parameters are in the hidden layer. 
 Question: What is the baseline?
Output: The baseline is a multi-task architecture inspired by another paper.
Further, we employ BLEU score to evaluate the accuracy of generations BIBREF21, and the number of distinct n-gram to evaluate the diversity of generations BIBREF6.  
 Question: How do they measure the diversity of inferences?
Output: by number of distinct n-grams
We decided to explore the remaining space for improvement on the CBT by testing humans on a random subset of 50 named entity and 50 common noun validation questions that the psr ensemble could not answer correctly. 
 Question: How do they show there is space for further improvement?
Output:  by testing humans on a random subset of 50 named entity and 50 common noun validation questions that the psr ensemble could not answer correctly
Given a set of evaluation documents, each having a known correct label from a closed set of labels (often referred to as the “gold-standard”), and a predicted label for each document from the same set, the document-level accuracy is the proportion of documents that are correctly labeled over the entire evaluation collection. This is the most frequently reported metric and conveys the same information as the error rate, which is simply the proportion of documents that are incorrectly labeled (i.e. INLINEFORM0 ). There are two distinct ways in which results are generally summarized per-language: (1) precision, in which documents are grouped according to their predicted language; and (2) recall, in which documents are grouped according to what language they are actually written in. It is also common practice to report an F-score INLINEFORM0 , which is the harmonic mean of precision and recall. 
 Question: what evaluation methods are discussed?
Output: document-level accuracy precision recall F-score
We propose a novel problem of relationship recommendation (RSR). Different from the reciprocal recommendation problem on DSNs, our RSR task operates on regular social networks (RSN), estimating long-term and serious relationship compatibility based on social posts such as tweets. 
 Question: Is this a task other people have worked on?
Output: No
Tourist: I can't go straight any further.

Guide: ok. turn so that the theater is on your right.

Guide: then go straight

Tourist: That would be going back the way I came

Guide: yeah. I was looking at the wrong bank

Tourist: I'll notify when I am back at the brooks brothers, and the bank.

Tourist: ACTION:TURNRIGHT

Guide: make a right when the bank is on your left

Tourist: ACTION:FORWARD ACTION:FORWARD ACTION:TURNRIGHT

Tourist: Making the right at the bank.

Tourist: ACTION:FORWARD ACTION:FORWARD

Tourist: I can't go that way.

Tourist: ACTION:TURNLEFT

Tourist: Bank is ahead of me on the right

Tourist: ACTION:FORWARD ACTION:FORWARD ACTION:TURNLEFT

Guide: turn around on that intersection

Tourist: I can only go to the left or back the way I just came.

Tourist: ACTION:TURNLEFT

Guide: you're in the right place. do you see shops on the corners?

Guide: If you're on the corner with the bank, cross the street

Tourist: I'm back where I started by the shop and the bank.

Tourist: ACTION:TURNRIGHT 
 Question: What language do the agents talk in?
Output: English
Inflectional realization defines the inflected forms of a lexeme/lemma. As a computational task, often referred to as simply “morphological inflection," inflectional realization is framed as a mapping from the pairing of a lemma with a set of morphological tags to the corresponding word form. For example, the inflectional realization of SJQ Chatino verb forms entails a mapping of the pairing of the lemma lyu1 `fall' with the tag-set 1;SG;PROG to the word form nlyon32. Morphological analysis is the task of creating a morphosyntactic description for a given word. It can be framed in a context-agnostic manner (as in our case) or within a given context, as for instance for the SIGMORPHON 2019 second shared task BIBREF11. 
 Question: How does morphological analysis differ from morphological inflection?
Output: Morphological analysis is the task of creating a morphosyntactic description for a given word  inflectional realization is framed as a mapping from the pairing of a lemma with a set of morphological tags to the corresponding word form
Our pool of annotators is selected after several short training rounds, with up to 15 predicates per round, in which they received extensive personal feedback. 
 Question: How are workers trained?
Output: extensive personal feedback
We used the Reuters-8 dataset without stop words from BIBREF27 aiming at single-label classification, which is a preprocessed format of the Reuters-21578. 
 Question: Which dataset has been used in this work?
Output: Reuters-8 dataset without stop words
The RelNet model achieves a mean error of 0.285% across tasks which is better than the results of the EntNet model BIBREF17 . The RelNet model is able to achieve 0% test error on 11 of the tasks, whereas the EntNet model achieves 0% error on 7 of the tasks. The RelNet model achieves a mean error of 0.285% across tasks which is better than the results of the EntNet model BIBREF17 . The RelNet model is able to achieve 0% test error on 11 of the tasks, whereas the EntNet model achieves 0% error on 7 of the tasks. 
 Question: What are the relative improvements observed over existing methods?
Output: The RelNet model achieves a mean error of 0.285% across tasks which is better than the results of the EntNet model BIBREF17 . The RelNet model is able to achieve 0% test error on 11 of the tasks, whereas the EntNet model achieves 0% error on 7 of the tasks.
the input fusion layer to allow interactions between input facts and a novel attention based GRU that allows for logical reasoning over ordered inputs. 
 Question: How does the model circumvent the lack of supporting facts during training?
Output: the input fusion layer to allow interactions between input facts and a novel attention based GRU that allows for logical reasoning over ordered inputs.
We leveraged an existing, annotated Twitter dataset that was constructed based on a hierarchical model of depression-related symptoms BIBREF12 , BIBREF13 . The dataset contains 9,473 annotations for 9,300 tweets. 
 Question: What dataset is used for this study?
Output: BIBREF12 , BIBREF13
For example, the performance of even the best QA models degrades substantially on our hyponym probes (by 8-15%) when going from 1-hop links to 2-hops. Further, the accuracy of even our best models on the WordNetQA probe drops by 14-44% under our cluster-based analysis, which assesses whether a model knows several facts about each individual concept, rather than just being good at answering isolated questions.  
 Question: After how many hops does accuracy decrease?
Output: 1-hop links to 2-hops
The initial analyses of the embedding matrices for both the UK and France revealed that in general, words were grouped by context or influence on the electricity consumption. For instance, we observed that winter words were together and far away from summer ones.  
 Question: Is there any example where geometric property is visible for context similarity between words?
Output: Yes
Out of 43 teams our system ranked 421st with an official F1-score of 0.2905 on the test set. Although our model outperforms the baseline in the validation set in terms of F1-score, we observe important drops for all metrics compared to the test set, showing that the architecture seems to be unable to generalize well. 
 Question: What were their results on the test set?
Output: an official F1-score of 0.2905 on the test set
The baseline is a hierarchical phrase-based system BIBREF29 with a 4-gram language model, with feature weights tuned using MIRA BIBREF30 . Therefore, an appropriate additional baseline would be to mark translation rules with these indicator functions but without the scores, akin to identifying rules with phrases in them (Baseline + SegOn). 
 Question: Which translation systems do they compare against?
Output: hierarchical phrase-based system BIBREF29 appropriate additional baseline would be to mark translation rules with these indicator functions but without the scores, akin to identifying rules with phrases in them (Baseline + SegOn)
While $\beta $-VAE offers regularizing the ELBO via an additional coefficient $\beta \in {\rm I\!R}^+$, a simple extension BIBREF16 of its objective function incorporates an additional hyperparameter $C$ to explicitly control the magnitude of the KL term,

where $C\!\! \in \!\! {\rm I\!R}^+$ and $| . |$ denotes the absolute value. 
 Question: How does explicit constraint on the KL divergence term that authors propose looks like?
Output: Answer with content missing: (Formula 2) Formula 2 is an answer: 
\big \langle\! \log p_\theta({x}|{z}) \big \rangle_{q_\phi({z}|{x})}  -  \beta |D_{KL}\big(q_\phi({z}|{x}) || p({z})\big)-C|
To evaluate the proposed LRC framework and the AutoJudge model, we carry out a series of experiments on the divorce proceedings, a typical yet complex field of civil cases.  
 Question: what civil field is the dataset about?
Output: divorce
Table 1 shows an example of difficult and easy examples according to our definition of difficulty. The underlined text demarcates the (consensus) reference label provided by domain experts. In the difficult examples, crowd workers marked text distinct from these reference annotations; whereas in the easy cases they reproduced them with reasonable fidelity. The difficult sentences usually exhibit complicated structure and feature jargon. 
 Question: Is an instance a sentence or an IE tuple?
Output: sentence
We collect more human-written scenes for each concept-set in dev and test set through crowd-sourcing via the Amazon Mechanical Turk platform. Each input concept-set is annotated by at least three different humans. The annotators are also required to give sentences as the rationales, which further encourage them to use common sense in creating their scenes. 
 Question: Are the rationales generated after the sentences were written?
Output: Yes
Since our objective is to compare character-based and word-based approaches, we have also implemented a simple word-level encoder for tweets. The input tweet is first split into tokens along white-spaces. A more sophisticated tokenizer may be used, but for a fair comparison we wanted to keep language specific preprocessing to a minimum. The encoder is essentially the same as tweet2vec, with the input as words instead of characters. A lookup table stores word vectors for the $V$ (20K here) most common words, and the rest are grouped together under the `UNK' token. 
 Question: what is the word level baseline they compare to?
Output: a simple word-level encoder with the input as words instead of characters
We employ a multi-class Naive Bayes classifier as the second stage classification mechanism, for categorizing tweets appropriately, depending on the type of emergencies they indicate. 
 Question: What classifier is used for emergency categorization?
Output: multi-class Naive Bayes
We evaluated SDNet on CoQA dataset 
 Question: Is the model evaluated on other datasets?
Output: No
One of the several formats into which FHIR can be serialized is RDF. However, because RDF was designed as an abstract information model and FHIR was designed for operational use in a healthcare setting, there is the potential for a slight mismatch between the models 
 Question: What are the differences between FHIR and RDF?
Output: One of the several formats into which FHIR can be serialized is RDF there is the potential for a slight mismatch between the models
We collect utterances from the $\mathbf {C}$hinese $\mathbf {A}$rtificial $\mathbf {I}$ntelligence $\mathbf {S}$peakers (CAIS), and annotate them with slot tags and intent labels. The training, validation and test sets are split by the distribution of intents, where detailed statistics are provided in the supplementary material. Since the utterances are collected from speaker systems in the real world, intent labels are partial to the PlayMusic option. We adopt the BIOES tagging scheme for slots instead of the BIO2 used in the ATIS, since previous studies have highlighted meaningful improvements with this scheme BIBREF30 in the sequence labeling field. 
 Question: What is the domain of their collected corpus?
Output: speaker systems in the real world
Our search space consists of two stackable cells, one for the model encoder and one for the decoder Each cell contains NASNet-style blocks, which receive two hidden state inputs and produce new hidden states as outputs  Our search space contains five branch-level search fields (input, normalization, layer, output dimension and activation), one block-level search field (combiner function) and one cell-level search field (number of cells). 
 Question: What is in the model search space?
Output: Our search space consists of two stackable cells, one for the model encoder and one for the decoder  Each cell contains NASNet-style blocks, which receive two hidden state inputs and produce new hidden states as outputs Our search space contains five branch-level search fields (input, normalization, layer, output dimension and activation), one block-level search field (combiner function) and one cell-level search field (number of cells).
More specifically, we explore three regularization terms to address the problem: (1) a regularization term associated with neutral features; (2) the maximum entropy of class distribution regularization term; and (3) the KL divergence between reference and predicted class distribution. 
 Question: What are the three regularization terms?
Output: a regularization term associated with neutral features the maximum entropy of class distribution regularization term the KL divergence between reference and predicted class distribution
As a convention, the metric of joint goal accuracy is used to compare our model to previous work. 
 Question: What are the performance metrics used?
Output: joint goal accuracy
In Sec. "Materials and Methods" we discuss our materials and methods, including the dataset we studied, how we preprocessed that data and extracted a `causal' corpus and a corresponding `control' corpus, and the details of the statistical and language analysis tools we studied these corpora with. 
 Question: Do they use expert annotations, crowdsourcing, or only automatic methods to analyze the corpora?
Output: Only automatic methods
We thus also conducted human studies on Amazon MTurk to evaluate the generated responses with pairwise comparison for dialogue quality. We compare our models with an advanced decoding algorithm MMI BIBREF2 and two models, namely LSTM BIBREF0 and VHRED BIBREF7, both with additive attention. To our best knowledge, LSTM and VHRED were the primary models with which F1's were reported on the Ubuntu dataset. Following BIBREF5 (BIBREF5), we employ two criteria: Plausibility and Content Richness. The first criterion measures whether the response is plausible given the context, while the second gauges whether the response is diverse and informative.  
 Question: How is human evaluation performed, what was the criteria?
Output: Through Amazon MTurk annotators to determine plausibility and content richness of the response
Compared to using external models for confidence modeling, an advantage of the proposed method is that the base model does not change: the binary classification loss just provides additional supervision. For supervised open IE systems, the confidence score of an assertion is typically computed based on its extraction likelihood given by the model BIBREF3 , BIBREF5  We follow the evaluation metrics described by Stanovsky:2016:OIE2016: area under the precision-recall curve (AUC) and F1 score. 
 Question: How does this compare to traditional calibration methods like Platt Scaling?
Output: No reliability diagrams are provided and no explicit comparison is made between confidence scores or methods.
Thus, to limit the number of different inputs to the classifier, we wish to reduce the number of distinct word recognition outputs that an attacker can induce, not just the number of words on which the model is “fooled”. We denote this property of a model as its sensitivity. 
 Question: What does the "sensitivity" quantity denote?
Output: the number of distinct word recognition outputs that an attacker can induce
We compare our best F1 score with other state-of-the-art approaches in table TABREF39 , which shows our model has competitive advantage in dealing with drug-drug interaction extraction. 
 Question: What is the performance of their model?
Output: Answer with content missing: (Table II) Proposed model has F1 score of  0.7220.
With the above hyperparameter setting, the hybrid NER model achieved a F1 score of $0.995$ on synthesized queries and $0.948$ on clinical notes while the i2b2 NER model achieved a F1 score of $0.441$ on synthesized queries and $0.927$ on clinical notes (See Table TABREF23). 
 Question: what were their performance results?
Output:  the hybrid NER model achieved a F1 score of $0.995$ on synthesized queries and $0.948$ on clinical notes while the i2b2 NER model achieved a F1 score of $0.441$ on synthesized queries and $0.927$ on clinical notes
Posts in FBFans dataset are used for this analysis. We calculate the like statistics of each distinct author from these 32,595 posts. 
 Question: What is the size of the Chinese data?
Output: 32,595 posts
To this goal, we select a diverse set of eight source languages from different language families – Basque, French, German, Hungarian, Italian, Navajo, Turkish, and Quechua – and three target languages – English, Spanish and Zulu.  
 Question: What are the tree target languages studied in the paper?
Output: English, Spanish and Zulu
jiant System Overview ::: jiant Components
Tasks: Tasks have references to task data, methods for processing data, references to classifier heads, and methods for calculating performance metrics, and making predictions. 
 Question: Does jiant involve datasets for the 50 NLU tasks?
Output: Yes
Hashtag prediction for social media has been addressed earlier, for example in BIBREF15 , BIBREF16 . BIBREF15 also use a neural architecture, but compose text embeddings from a lookup table of words. 
 Question: Is this hashtag prediction task an established task, or something new?
Output: established task
Results of the DQN-based agent are presented in fig: scenario comparison. Each plot depicts the average reward (across 5 seeds) of all representations methods. It can be seen that the NLP representation outperforms the other methods.  NLP representations remain robust to changes in the environment as well as task-nuisances in the state.  
 Question: What result from experiments suggest that natural language based agents are more robust?
Output: Average reward across 5 seeds show that NLP representations are robust to changes in the environment as well task-nuisances
We train an LSTM with and without attention on the training set. After training, we take the best model in terms of BLEU score BIBREF16 on the development set and calculate the BLEU score on the test set.  
 Question: How is the generative model evaluated?
Output: Comparing BLEU score of model with and without attention
Next, we compared our VTN model with an RNN-based seq2seq VC model called ATTS2S BIBREF8. This model is based on the Tacotron model BIBREF32 with the help of context preservation loss and guided attention loss to stabilize training and maintain linguistic consistency after conversion. We followed the configurations in BIBREF8 but used mel spectrograms instead of WORLD features. 
 Question: What is the baseline model?
Output: a RNN-based seq2seq VC model called ATTS2S based on the Tacotron model
To report the result for answer selection, MAP and MRR are used; however, the answer triggering task is evaluated by F1-score. The result for MULT Method is reported in Table. 1. 
 Question: Is accuracy the only metric they used to compare systems?
Output: No
By choosing English (En) as the pivot language, we perform pivot alignments for identical English segments on Europarl Fr-En and En-De parallel corpora BIBREF18 , constructing a multi-parallel corpus of Fr-En-De. Then each of the Fr*-De and Fr-De* pseudo parallel corpora is established from the multi-parallel data by applying the pivot language-based translation described in the previous section. 
 Question: How do they align the synthetic data?
Output: By choosing English (En) as the pivot language, we perform pivot alignments for identical English segments on Europarl Fr-En and En-De parallel corpora BIBREF18 , constructing a multi-parallel corpus of Fr-En-De. Then each of the Fr*-De and Fr-De* pseudo parallel corpora is established from the multi-parallel data by applying the pivot language-based translation described in the previous section.
Despite the greater similarity between our task and the 2013 ShARe/CLEF Task 1, we use the clinical notes from the CE task in 2010 i2b2/VA on account of 1) the data from 2010 i2b2/VA being easier to access and parse, 2) 2013 ShARe/CLEF containing disjoint entities and hence requiring more complicated tagging schemes. 
 Question: where did they obtain the annotated clinical notes from?
Output: clinical notes from the CE task in 2010 i2b2/VA
We address this issue by We address this issue by examining and reviewing publicly available datasets for abusive content detection, which we provide access to on a new dedicated website, hatespeechdata.com. We address this issue by examining and reviewing publicly available datasets for abusive content detection, which we provide access to on a new dedicated website, hatespeechdata.com. 
 Question: What is open website for cataloguing abusive language data?
Output: hatespeechdata.com
For each test set, we use the corresponding training questions $Q_\mathit {tr}$ to retrieve domain-relevant sentences from S. Specifically, for each multiple-choice question $(q,A) \in Q_\mathit {tr}$ and each choice $a \in A$ , we use all non-stopword tokens in $q$ and $a$ as an ElasticSearch query against S. We take the top 200 hits, run Open IE v4, and aggregate the resulting tuples over all $a \in A$ and over all questions in $Q_\mathit {tr}$ to create the tuple KB (T). 
 Question: What method was used to generate the OpenIE extractions?
Output: for each multiple-choice question $(q,A) \in Q_\mathit {tr}$ and each choice $a \in A$ , we use all non-stopword tokens in $q$ and $a$ as an ElasticSearch query against S take the top 200 hits, run Open IE v4, and aggregate the resulting tuples over all $a \in A$ and over all questions in $Q_\mathit {tr}$
We build the iSQuAD and iNewsQA datasets based on SQuAD v1.1 BIBREF0 and NewsQA BIBREF1. iMRC: Making MRC Interactive ::: Evaluation Metric
Since iMRC involves both MRC and RL, we adopt evaluation metrics from both settings. First, as a question answering task, we use $\text{F}_1$ score to compare predicted answers against ground-truth, as in previous works. When there exist multiple ground-truth answers, we report the max $\text{F}_1$ score. Second, mastering multiple games remains quite challenging for RL agents. Therefore, we evaluate an agent's performance during both its training and testing phases. During training, we report training curves averaged over 3 random seeds. During test, we follow common practice in supervised learning tasks where we report the agent's test performance corresponding to its best validation performance . 
 Question: What are the models evaluated on?
Output: They evaluate F1 score and agent's test performance on their own built interactive datasets (iSQuAD and iNewsQA)
Visual Question Generation (VQG) is another emerging topic which aims to ask questions given an image. Motivated by this, BIBREF10 proposed open-ended VQG that aims to generate natural and engaging questions about an image. 
 Question: Do they survey visual question generation work?
Output: Yes
To test the proposed methods ability to generate unsupervised words, it was necessary to devise a method of measuring word relevance. Topic modeling was used based on the assumption that words found in the same topic are more relevant to one another then words from different topics BIBREF14. The complete 200k headline dataset BIBREF11 was modeled using a Naïve Bayes Algorithm BIBREF15 to create a word-category co-occurrence model. The top 200 most relevant words were then found for each category and used to create the topic table SECREF12. It was assumed that each category represented its own unique topic.

The number of relevant output words as a function of the headline’s category label were measured, and can be found in figure SECREF4. The results demonstrate that the proposed method could correctly identify new words relevant to the input topic at a signal to noise ratio of 4 to 1. 
 Question: How do they determine similarity between predicted word and topics?
Output: number of relevant output words as a function of the headline’s category label
BIBREF6 introduce the KG-A2C, which uses a knowledge graph based state-representation to aid in the section of actions in a combinatorially-sized action-space—specifically they use the knowledge graph to constrain the kinds of entities that can be filled in the blanks in the template action-space. They test their approach on Zork1, showing the combination of the knowledge graph and template action selection resulted in improvements over existing methods. They note that their approach reaches a score of 40 which corresponds to a bottleneck in Zork1 where the player is eaten by a “grue” (resulting in negative reward) if the player has not first lit a lamp. 
 Question: What are the baselines?
Output: a score of 40
In this work, we use MLP (modified to handle our data representation) as the base classifier. 
 Question: Which classification algorithm do they use for s2sL?
Output: MLP
We show the final results for opinion recommendation, comparing our proposed model with the following state-of-the-art baseline systems:

RS-Average is the widely-adopted baseline (e.g., by Yelp.com), using the averaged review scores as the final score.

RS-Linear estimates the rating score that a user would give by INLINEFORM0 BIBREF49 , where INLINEFORM1 and INLINEFORM2 are the the training deviations of the user INLINEFORM3 and the product INLINEFORM4 , respectively.

RS-Item applies INLINEFORM0 NN to estimate the rating score BIBREF50 . We choose the cosine similarity between INLINEFORM1 to measure the distance between product.

RS-MF is a state-of-the-art recommendation model, which uses matrix factorisation to predict rating score BIBREF8 , BIBREF41 , BIBREF25 .

Sum-Opinosis uses a graph-based framework to generate abstractive summarisation given redundant opinions BIBREF51 .

Sum-LSTM-Att is a state-of-the-art neural abstractive summariser, which uses an attentional neural model to consolidate information from multiple text sources, generating summaries using LSTM decoding BIBREF44 , BIBREF3 .

All the baseline models are single-task models, without considering rating and summarisation prediction jointly. The results are shown in Table TABREF46 .  
 Question: What are the baselines?
Output: RS-Average  RS-Linear RS-Item RS-MF Sum-Opinosis Sum-LSTM-Att
Stanford - Twitter Sentiment Corpus (STS Corpus): STS Corpus contains 1,600K training tweets collected by a crawler from BIBREF0 . BIBREF0 constructed a test set manually with 177 negative and 182 positive tweets. The Stanford test set is small. However, it has been widely used in different evaluation tasks BIBREF0 BIBREF5 BIBREF13 .

Sanders - Twitter Sentiment Corpus: This dataset consists of hand-classified tweets collected by using search terms: INLINEFORM0 , #google, #microsoft and #twitter. We construct the dataset as BIBREF14 for binary classification.

Health Care Reform (HCR): This dataset was constructed by crawling tweets containing the hashtag #hcr BIBREF15 . Task is to predict positive/negative tweets BIBREF14 . 
 Question: Which three Twitter sentiment classification datasets are used for experiments?
Output: Stanford - Twitter Sentiment Corpus (STS Corpus) Sanders - Twitter Sentiment Corpus Health Care Reform (HCR)
According to the context given, the raters were instructed to evaluate the quality of the responses based on three criteria: (1) grammatical correctness—whether or not the response is fluent and free of grammatical mistakes; (2) contextual coherence—whether or not the response is context sensitive to the previous dialog history; (3) emotional appropriateness—whether or not the response conveys the right emotion and feels as if it had been produced by a human. For each criterion, the raters gave scores of either 0, 1 or 2, where 0 means bad, 2 means good, and 1 indicates neutral. 
 Question: How is human evaluation performed?
Output: (1) grammatical correctness (2) contextual coherence (3) emotional appropriateness
To address our next question, we assessed our pipeline when learning relations independently (i.e., individually) versus learning relations jointly within the RDN, displayed in Table TABREF22 . 
 Question: What do they learn jointly?
Output: relations