niryuu/sni-each-converted huggingface dataset:
I first measured the effectiveness of the embeddings at the skip-gram's training task, predicting context words INLINEFORM0 given input words INLINEFORM1 . This task measures the methods' performance for predictive language modeling. I used four datasets of sociopolitical, scientific, and literary interest: the corpus of NIPS articles from 1987 – 1999 ( INLINEFORM2 million), the U.S. presidential state of the Union addresses from 1790 – 2015 ( INLINEFORM3 ), the complete works of Shakespeare ( INLINEFORM4 ; this version did not contain the Sonnets), and the writings of black scholar and activist W.E.B. Du Bois, as digitized by Project Gutenberg ( INLINEFORM5 ). For each dataset, I held out 10,000 INLINEFORM6 pairs uniformly at random, where INLINEFORM7 , and aimed to predict INLINEFORM8 given INLINEFORM9 (and optionally, INLINEFORM10 ). Since there are a large number of classes, I treat this as a ranking problem, and report the mean reciprocal rank. The experiments were repeated and averaged over 5 train/test splits. 
 Question: What is MRR?
Output: mean reciprocal rank
Machine-machine Interaction A related line of work explores simulation-based dialogue generation, where the user and system roles are simulated to generate a complete conversation flow, which can then be converted to natural language using crowd workers BIBREF1. Such a framework may be cost-effective and error-resistant since the underlying crowd worker task is simpler, and semantic annotations are obtained automatically. It is often argued that simulation-based data collection does not yield natural dialogues or sufficient coverage, when compared to other approaches such as Wizard-of-Oz. We argue that simulation-based collection is a better alternative for collecting datasets like this owing to the factors below. 
 Question: How did they gather the data?
Output: simulation-based dialogue generation, where the user and system roles are simulated to generate a complete conversation flow, which can then be converted to natural language using crowd workers
In this section we describe a number of experiments targeted to compare the performance of popular named entity recognition algorithms on our data. We trained and evaluated Stanford NER, spaCy 2.0, and a recurrent model similar to BIBREF13 , BIBREF14 that uses bidirectional LSTM cells for character-based feature extraction and CRF, described in Guillaume Genthial's Sequence Tagging with Tensorflow blog post BIBREF15 . Stanford NER is conditional random fields (CRF) classifier based on lexical and contextual features such as the current word, character-level n-grams of up to length 6 at its beginning and the end, previous and next words, word shape and sequence features BIBREF16 . spaCy 2.0 uses a CNN-based transition system for named entity recognition. The main model that we focused on was the recurrent model with a CRF top layer, and the above-mentioned methods served mostly as baselines.  
 Question: what ner models were evaluated?
Output: Stanford NER spaCy 2.0  recurrent model with a CRF top layer
We suspect the absence of decoding in maximum likelihood estimation as a cause behind this inconsistency, and suggest investigating sequence-level learning as an alternative in the future. Inconsistency may arise from the lack of decoding in solving this optimization problem. Maximum likelihood learning fits the model $p_{\theta }$ using the data distribution, whereas a decoded sequence from the trained model follows the distribution $q_{\mathcal {F}}$ induced by a decoding algorithm. Based on this discrepancy, we make a strong conjecture: we cannot be guaranteed to obtain a good consistent sequence generator using maximum likelihood learning and greedy decoding. 
 Question: Is infinite-length sequence generation a result of training with maximum likelihood?
Output: There are is a strong conjecture that it might be the reason but it is not proven.
In order to evaluate the precision of the retrieved documents in each experiment, we used "TREC_Eval" tool [3]. TREC_Eval is a standard tool for evaluation of IR tasks and its name is a short form of Text REtrieval Conference (TREC) Evaluation tool. The Mean Average Precision (MAP) reported by TREC_Eval was 27.99% without query expansion and 37.10% with query expansion which shows more than 9 percent improvement. 
 Question: Which evaluation metric has been measured?
Output: Mean Average Precision
We evaluate our model on two public datasets, namely, Penn Treebank (PTB) BIBREF9 and the end-to-end (E2E) text generation corpus BIBREF10, which have been used in a number of previous works for text generation BIBREF0, BIBREF5, BIBREF11, BIBREF12. PTB consists of more than 40,000 sentences from Wall Street Journal articles whereas the E2E dataset contains over 50,000 sentences of restaurant reviews. The statistics of these two datasets are summarised in Table TABREF11. 
 Question: Which dataset do they use for text modelling?
Output: Penn Treebank (PTB) end-to-end (E2E) text generation corpus
With the goal of building a generalizable sentiment analysis model, we used three different training sets as provided in Table TABREF5 . One of these three datasets (Amazon reviews BIBREF23 , BIBREF24 ) is larger and has product reviews from several different categories including book reviews, electronics products reviews, and application reviews. The other two datasets are to make the model more specialized in the domain. In this paper we focus on restaurant reviews as our domain and use Yelp restaurant reviews dataset extracted from Yelp Dataset Challenge BIBREF25 and restaurant reviews dataset as part of a Kaggle competition BIBREF26 . 
 Question: what dataset was used for training?
Output: Amazon reviews Yelp restaurant reviews restaurant reviews
We asked medical doctors experienced in extracting knowledge related to medical entities from texts to annotate the entities described above. Initially, we asked four annotators to test our guidelines on two texts. Subsequently, identified issues were discussed and resolved. Following this pilot annotation phase, we asked two different annotators to annotate two case reports according to our guidelines. The same annotators annotated an overall collection of 53 case reports. The annotation was performed using WebAnno BIBREF7, a web-based tool for linguistic annotation. The annotators could choose between a pre-annotated version or a blank version of each text. The pre-annotated versions contained suggested entity spans based on string matches from lists of conditions and findings synonym lists. 
 Question: How was annotation performed?
Output: Experienced medical doctors used a linguistic annotation tool to annotate entities.
Experimental results on public datasets show that our CRU model could substantially outperform various systems by a large margin, and set up new state-of-the-art performances on related datasets. 
 Question: Are there some results better than state of the art on these tasks?
Output: Yes
Our model adopts the recently introduced biaffine attention BIBREF14 to enhance our role scorer.  
 Question: What is the biaffine scorer?
Output: biaffine attention BIBREF14
In this paper, we provide a brief survey of existing datasets and describe the CRWIZ framework for pairing crowdworkers and having half of them acting as Wizards by limiting their dialogue options only to relevant and plausible ones, at any one point in the interaction Dialogue structure: we introduced structured dialogues through a Finite State Machine (FSM) that controls the current dialogue state and offers multiple suitable and relevant state transitions (actions) to the Wizard depending on the point in the interaction, the state of the world and the history. A graph of dialogue states, transitions and utterances is loaded when the system is initialised, and each chat room has its own dialogue state, which changes through actions. 
 Question: How is dialogue guided to avoid interactions that breach procedures and processes only known to experts?
Output: pairing crowdworkers and having half of them acting as Wizards by limiting their dialogue options only to relevant and plausible ones, at any one point in the interaction
We also evaluate all five models on downstream tasks from the VecEval suite BIBREF13 , using only the tasks for which training and evaluation data is freely available: chunking, sentiment and question classification, and natural language identification (NLI). The default settings from the suite are used, but we run only the fixed settings, where the embeddings themselves are not tunable parameters of the models, forcing the system to use only the information already in the embeddings.

Finally, we use LV-N, LV-M, and FT to generate OOV word representations for the following words: 1) “hellooo”: a greeting commonly used in instant messaging which emphasizes a syllable. 2) “marvelicious”: a made-up word obtained by merging “marvelous” and “delicious”. 3) “louisana”: a misspelling of the proper name “Louisiana”. 4) “rereread”: recursive use of prefix “re”. 5) “tuzread”: made-up prefix “tuz”. 
 Question: How do they evaluate their resulting word embeddings?
Output: We also evaluate all five models on downstream tasks from the VecEval suite BIBREF13 , using only the tasks for which training and evaluation data is freely available: chunking, sentiment and question classification, and natural language identification (NLI). The default settings from the suite are used, but we run only the fixed settings, where the embeddings themselves are not tunable parameters of the models, forcing the system to use only the information already in the embeddings.
In this work, the experiments are conducted using the BABEL speech corpus collected from the IARPA babel program.  
 Question: What data do they train the language models on?
Output:  BABEL speech corpus
The corpus of supervisor assessment has 26972 sentences. The summary statistics about the number of words in a sentence is: min:4 max:217 average:15.5 STDEV:9.2 Q1:9 Q2:14 Q3:19. 
 Question: What is the average length of the sentences?
Output: 15.5
For our study, we consider that a tweet went viral if it was retweeted more than 1000 times. 
 Question: What is their definition of tweets going viral?
Output: Viral tweets are the ones that are retweeted more than 1000 times
There are several caveats in our work: first, tweet sentiment is rarely binary (this work could be extended to a multinomial or continuous model). Second, our results are constrained to Twitter users, who are known to be more negative than the general U.S. population BIBREF9 . Third, we do not take into account the aggregate effects of continued natural disasters over time. Going forward, there is clear demand in discovering whether social networks can indicate environmental metrics in a “nowcasting" fashion. As climate change becomes more extreme, it remains to be seen what degree of predictive power exists in our current model regarding climate change sentiments with regards to natural disasters. 
 Question: Do the authors mention any confounds to their study?
Output: No
To test the applicability of the proposed MWA attention, we choose three publicly available Chinese pre-trained models as the basic encoder: BERT, ERNIE, and BERT-wwm. In order to make a fair comparison, we keep the same hyper-parameters (such maximum length, warm-up steps, initial learning rate, etc) as suggested in BERT-wwm BIBREF13 for both baselines and our method on each dataset. 
 Question: What pre-trained models did they compare to?
Output: BERT, ERNIE, and BERT-wwm
We first evaluate our converted embedding models on multi-language lexical similarity and relatedness tasks, as a sanity check, to make sure the word sense induction process did not hurt the general performance of the embeddings. Then, we test the sense embeddings on WSD task. 
 Question: Was any extrinsic evaluation carried out?
Output: Yes
We treat each individual language as a task and train a joint model for all the tasks. We seek to learn a set of shared character embeddings for taggers in both languages together through optimization of a joint loss function that combines the high-resource tagger and the low-resource one. 
 Question: How are character representations from various languages joint?
Output: shared character embeddings for taggers in both languages together through optimization of a joint loss function
We have evaluated our framework using a synthetic literal set (S-Lite) and a real literal set (R-Lite) from DBpedia BIBREF0 .  
 Question: What KB is used in this work?
Output: DBpedia
On Figure FIGREF1 one can see that the outcomes for every single rerun differ significantly. Namely, accuracy can change up to 5 percentage points, whereas BLEU can vary up to 8 points. 
 Question: How much does the standard metrics for style accuracy vary on different re-runs?
Output: accuracy can change up to 5 percentage points, whereas BLEU can vary up to 8 points
To have an estimation about human performance in each metric, we iteratively treat every reference sentence in dev/test data as the prediction to be compared with all references (including itself). That is, if a model has the same reasoning ability with average performance of our crowd workers, its results should exceed this “human bound”. 
 Question: What measures were used for human evaluation?
Output: To have an estimation about human performance in each metric, we iteratively treat every reference sentence in dev/test data as the prediction to be compared with all references (including itself).
Our Pointer-Gen+ARL-SEN model achieves the best performance of 60.8%. This is an absolute improvement of 18.2% over the Pointer-Gen baseline. 
 Question: What is this method improvement over the best performing state-of-the-art?
Output: absolute improvement of 18.2% over the Pointer-Gen baseline
To demonstrate the generality of NSA, we further present video captioning, machine translation, and visual question answering experiments on the VATEX, WMT 2014 English-to-German, and VQA-v2 datasets, respectively.  
 Question: What datasets are used for experiments on three other tasks?
Output: VATEX, WMT 2014 English-to-German, and VQA-v2 datasets
A part of the ILCI English-Hindi Tourism parallel corpus (1500 sentences) was used for training the classifiers.  
 Question: What is the size of the parallel corpus used to train the model constraints?
Output: 1500 sentences
Table 2 lists the accuracies of the all methods on two classifier architectures. The results show that, for various datasets on different classifier architectures, our conditional BERT contextual augmentation improves the model performances most. BERT can also augments sentences to some extent, but not as much as conditional BERT does. 
 Question: Do the authors report performance of conditional bert on tasks without data augmentation?
Output: Yes
In this paper, we analyze the task of automatically correcting run-on sentences. We develop two methods: a conditional random field model (roCRF) and a Seq2Seq attention model (roS2S) and show that they outperform models from the sister tasks of punctuation restoration and whole-sentence grammatical error correction. 
 Question: Which machine learning models do they use to correct run-on sentences?
Output: conditional random field model Seq2Seq attention model
We initialized the embeddings of these words with 300 dimensional Glove embeddings BIBREF31 .  
 Question: Do they use pretrained embeddings?
Output: Yes
We took inspiration from these works to design our experiments to solve the CSKS task. 
 Question: What problem do they apply transfer learning to?
Output: CSKS task
In this section, we describe our data collection for these 3 components. We first describe a series of pilot studies that we conducted in order to collect commonsense inference questions (Section SECREF4 ). In Section SECREF5 , we discuss the resulting data collection of questions, texts and answers via crowdsourcing on Amazon Mechanical Turk (henceforth MTurk). Section SECREF17 gives information about some necessary postprocessing steps and the dataset validation.  
 Question: how was the data collected?
Output: The data was collected using 3 components: describe a series of pilot studies that were conducted to collect commonsense inference questions, then discuss the resulting data collection of questions, texts and answers via crowdsourcing on Amazon Mechanical Turk and gives information about some necessary postprocessing steps and the dataset validation.
On Twitter we can see results that are consistent with the RCV results for the left-to-center political spectrum. The exception, which clearly stands out, is the right-wing groups ENL and EFDD that seem to be the most cohesive ones. This is the direct opposite of what was observed in the RCV data. We speculate that this phenomenon can be attributed to the fact that European right-wing groups, on a European but also on a national level, rely to a large degree on social media to spread their narratives critical of European integration. 
 Question: Do the authors mention any possible confounds in their study?
Output: Yes
It is worth mentioning that the collected texts contain a large quantity of errors of several types: orthographic, syntactic, code-switched words (i.e. words not in the required language), jokes, etc. Hence, the original written sentences have been processed in order to produce “cleaner” versions, in order to make the data usable for some research purposes (e.g. to train language models, to extract features for proficiency assessment, ...). 
 Question: Are any of the utterances ungrammatical?
Output: Yes
After averaging over all concepts, we lose information on the lexical variation that each concept presents but on the other hand one can now investigate which regions show similar geolectal variation, yielding well defined linguistic varieties. Those cells that have similar colors in either figure FIGREF16 or figure FIGREF17 are expected to be ascribed to the same dialect zone. Thus, we can distinguish two main regions or clusters in the maps. The purple background covers most of the map and represents rural regions with small, scattered population. Our analysis shows that this group of cells possesses more specific words in their lexicon. In contrast, the green and yellow cells form a second cluster that is largely concentrated on the center and along the coastline, which correspond to big cities and industrialized areas. In these cells, the use of standard Spanish language is widespread due probably to school education, media, travelers, etc. The character of its vocabulary is more uniform as compared with the purple group. While the purple cluster prefer particular utterances, the lexicon of the urban group includes most of the keywords.  
 Question: What are the characteristics of the rural dialect?
Output: It uses particular forms of a concept rather than all of them uniformly
Finally, the decoder task, which predicts the target sequence probability at time INLINEFORM3 based on previous output and context information INLINEFORM4 , can be formulated as: DISPLAYFORM0 
 Question: How does their decoder generate text?
Output: decoder task, which predicts the target sequence probability at time INLINEFORM3 based on previous output and context information
A growing body of evidence shows that state-of-the-art models learn to exploit spurious statistical patterns in datasets BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, BIBREF17, instead of learning meaning in the flexible and generalizable way that humans do. Given this, human annotators—be they seasoned NLP researchers or non-experts—might easily be able to construct examples that expose model brittleness. 
 Question: What are the weaknesses found by non-expert annotators of current state-of-the-art NLI models?
Output: state-of-the-art models learn to exploit spurious statistical patterns in datasets human annotators—be they seasoned NLP researchers or non-experts—might easily be able to construct examples that expose model brittleness
In this study, we investigate the performance of our proposed models on WSJ BIBREF5 .  
 Question: Which dataset do they use?
Output: WSJ
We feed the above-described hand-crafted features together with the task-specific embeddings learned by the deep neural neural network (a total of 1,892 attributes combined) into a Support Vector Machines (SVM) classifier BIBREF37 . 
 Question: what classifiers were used in this paper?
Output: Support Vector Machines (SVM) classifier
When we annotate dialogues, we should read dialogues from begin to the end. For each utterance, we should find its one parent node at least from all its previous utterances. We assume that the discourse structure is a connected graph and no utterance is isolated. We propose three questions for eache dialogue and annotate the span of answers in the input dialogue. As we know, our dataset is the first corpus for multi-party dialogues reading comprehension.

We construct following questions and answers for the dialogue in Example 1:

Q1: When does Bdale leave?

A1: Fri morning

Q2: How to get people love Mark in Mjg59's opinion.

A2: Hire people to work on reverse-engineering closed drivers.

On the other hand, to improve the difficulty of the task, we propose $ \frac{1}{6}$ to $ \frac{1}{3}$ unanswerable questions in our dataset. We annotate unanswerable questions and their plausible answers (PA). Each plausible answer comes from the input dialogue, but is not the answer for the plausible question.

Q1: Whis is the email of daniels?

PA: +61 403 505 896 
 Question: Is annotation done manually?
Output: Yes
Baselines. As none of the existing KBC methods can solve the OKBC problem, we choose various versions of LiLi as baselines.

Single: Version of LiLi where we train a single prediction model INLINEFORM0 for all test relations.

Sep: We do not transfer (past learned) weights for initializing INLINEFORM0 , i.e., we disable LL.

F-th): Here, we use a fixed prediction threshold 0.5 instead of relation-specific threshold INLINEFORM0 .

BG: The missing or connecting links (when the user does not respond) are filled with “@-RelatedTo-@" blindly, no guessing mechanism.

w/o PTS: LiLi does not ask for additional clues via past task selection for skillset improvement. 
 Question: What baseline is used in the experiments?
Output: versions of LiLi
Filter 1: The string match filter deletes all KB triples where the correct answer (e.g., Apple) is a case-insensitive substring of the subject entity name (e.g., Apple Watch). Filter 2: Of course, entity names can be revealing in ways that are more subtle. As illustrated by our French actor example, a person's name can be a useful prior for guessing their native language and by extension, their nationality, place of birth, etc. Our person name filter uses cloze-style questions to elicit name associations inherent in BERT, and deletes KB triples that correlate with them. 
 Question: How is it determined that a fact is easy-to-guess?
Output:  filter deletes all KB triples where the correct answer (e.g., Apple) is a case-insensitive substring of the subject entity name (e.g., Apple Watch) person name filter uses cloze-style questions to elicit name associations inherent in BERT, and deletes KB triples that correlate with them
Participants who were shown the definition were more likely to suggest to ban the tweet. In fact, participants in group one very rarely gave different answers to questions one and two (18 of 500 instances or 3.6%). This suggests that participants in that group aligned their own opinion with the definition. 
 Question: How did the authors demonstrate that showing a hate speech definition caused annotators to partially align their own opinion with the definition?
Output: participants in group one very rarely gave different answers to questions one and two (18 of 500 instances or 3.6%)
It is an order of magnitude more efficient in terms of training time. The model is complex, both in terms of implementation and run-time. Indeed, this model requires pre-training and mutual-learning and requires days of training time, whereas the simple architecture we propose requires on the order of an hour (and is easy to implement). MGNC-CNN is usually better than MG-CNN. And on the Subj dataset, MG-CNN actually achieves slightly better results than BIBREF11 , with far less complexity and required training time (MGNC-CNN performs comparably, although no better, here). 
 Question: How much faster is training time for MGNC-CNN over the baselines?
Output: It is an order of magnitude more efficient in terms of training time. his model requires pre-training and mutual-learning and requires days of training time, whereas the simple architecture we propose requires on the order of an hour
Human Judgments
Following BIBREF11 , BIBREF12 and the vast amount of previous work on semantic similarity, we ask nine undergraduate subjects to assess the similarity of 360 pairs of relations from a subset of Wikidata BIBREF8 that are chosen to cover from high to low levels of similarity. In our experiment, subjects were asked to rate an integer similarity score from 0 (no similarity) to 4 (perfectly the same) for each pair.  
 Question: How do they gather human judgements for similarity between relations?
Output: By assessing similarity of 360 pairs of relations from a subset of Wikidata using an integer similarity score from 0 to 4
UTD aims to automatically identify and cluster repeated terms (e.g. words or phrases) from speech. To circumvent the exhaustive DTW-based search limited by INLINEFORM0 time BIBREF6 , we exploit the scalable UTD framework in the Zero Resource Toolkit (ZRTools) BIBREF7 , which permits search in INLINEFORM1 time. 
 Question: How is the vocabulary of word-like or phoneme-like units automatically discovered?
Output: Zero Resource Toolkit (ZRTools) BIBREF7
These powerful ideas from the NLP community have been adapted by researchers from other domains to tasks beyond representation of words, including relational entities BIBREF1 , BIBREF2 , general text-based attributes BIBREF3 , descriptive text of images BIBREF4 , nodes in graph structure of networks BIBREF5 , and queries BIBREF6 , to name a few. 
 Question: What domains are considered that have such large vocabularies?
Output: relational entities general text-based attributes descriptive text of images nodes in graph structure of networks queries
We considered UD1.2 corpora for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish. 
 Question: which languages are explored?
Output: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish
o sufficiently utilize the large dataset $\mathcal {A}$ and $\mathcal {M}$, the model is pre-trained on CTC-based ASR task and MT task in the pre-training stage. 
 Question: What is the attention module pretrained on?
Output: the model is pre-trained on CTC-based ASR task and MT task in the pre-training stage.
Looking to other target tasks, the grammar-related CoLA task benefits dramatically from ELMo pretraining: The best result without language model pretraining is less than half the result achieved with such pretraining. In contrast, the meaning-oriented textual similarity benchmark STS sees good results with several kinds of pretraining, but does not benefit substantially from the use of ELMo. 
 Question: Do some pretraining objectives perform better than others for sentence level understanding tasks?
Output: Yes
Methods ::: Length Token Method
Our first approach to control the length is inspired by target forcing in multilingual NMT BIBREF15, BIBREF16. We first split the training sentence pairs into three groups according to the target/source length ratio (in terms of characters). Ideally, we want a group where the target is shorter than the source (short), one where they are equally-sized (normal) and a last group where the target is longer than the source (long). In practice, we select two thresholds $t_\text{min}$ and $t_\text{max}$ according to the length ratio distribution. All the sentence pairs with length ratio between $t_\text{min}$ and $t_\text{max}$ are in the normal group, the ones with ratio below $t_\text{min}$ in short and the remaining in long. At training time we prepend a length token to each source sentence according to its group ($<$short$>$, $<$normal$>$, or $<$long$>$), in order to let a single network to discriminate between the groups (see Figure FIGREF2). At inference time, the length token is used to bias the network to generate a translation that belongs to the desired length group. 
 Question: How do they condition the output to a given target-source class?
Output: They use three groups short/normal/long translation classes to learn length token, which is in inference used to bias network to generate desired length group.
It was developed by querying the Followerwonk Web service API with location-neutral seed words known to be used by gang members across the U.S. in their Twitter profiles. 
 Question: Do the authors report on English datasets only?
Output: Yes
We evaluate the quality of the document embeddings learned by MPAD on 10 document classification datasets, covering the topic identification, coarse and fine sentiment analysis and opinion mining, and subjectivity detection tasks. We briefly introduce the datasets next. Their statistics are reported in Table TABREF21.

(1) Reuters. This dataset contains stories collected from the Reuters news agency in 1987. Following common practice, we used the ModApte split and considered only the 10 classes with the highest number of positive training examples. We also removed documents belonging to more than one class and then classes left with no document (2 classes).

(2) BBCSport BIBREF30 contains documents from the BBC Sport website corresponding to 2004-2005 sports news articles.

(3) Polarity BIBREF31 features positive and negative labeled snippets from Rotten Tomatoes.

(4) Subjectivity BIBREF32 contains movie review snippets from Rotten Tomatoes (subjective sentences), and Internet Movie Database plot summaries (objective sentences).

(5) MPQA BIBREF33 is made of positive and negative phrases, annotated as part of the summer 2002 NRRC Workshop on Multi-Perspective Question Answering.

(6) IMDB BIBREF34 is a collection of highly polarized movie reviews from IMDB (positive and negative). There are at most 30 reviews for each movie.

(7) TREC BIBREF35 consists of questions that are classified into 6 different categories.

(8) SST-1 BIBREF36 contains the same snippets as Polarity. The authors used the Stanford Parser to parse the snippets and split them into multiple sentences. They then used Amazon Mechanical Turk to annotate the resulting phrases according to their polarity (very negative, negative, neutral, positive, very positive).

(9) SST-2 BIBREF36 is the same as SST-1 but with neutral reviews removed and snippets classified as positive or negative.

(10) Yelp2013 BIBREF26 features reviews obtained from the 2013 Yelp Dataset Challenge. 
 Question: Which datasets are used?
Output: Reuters  BBCSport Polarity Subjectivity MPQA IMDB TREC SST-1 SST-2 Yelp2013
For the human evaluation, we follow the standard approach in evaluating machine translation systems BIBREF23 , as used for question generation by BIBREF9 . We asked three workers to rate 300 generated questions between 1 (poor) and 5 (good) on two separate criteria: the fluency of the language used, and the relevance of the question to the context document and answer. 
 Question: What human evaluation metrics were used in the paper?
Output: rating questions on a scale of 1-5 based on fluency of language used and relevance of the question to the context
The annotation projection method, which we follow in this work, is one way to address this problem. It was introduced for POS tagging, base noun phrase bracketing, NER tagging, and inflectional morphological analysis BIBREF29 but it has also been used for dependency parsing BIBREF30 , role labeling BIBREF31 , BIBREF32 and semantic parsing BIBREF26 . 
 Question: Do the authors test their annotation projection techniques on tasks other than AMR?
Output: No
However, recent work has found that many NLI datasets contain biases, or annotation artifacts, i.e., features present in hypotheses that enable models to perform surprisingly well using only the hypothesis, without learning the relationship between two texts BIBREF2 , BIBREF3 , BIBREF4 . For instance, in some datasets, negation words like “not” and “nobody” are often associated with a relationship of contradiction. As a ramification of such biases, models may not generalize well to other datasets that contain different or no such biases. 
 Question: Is such bias caused by bad annotation?
Output: No
To study the lexical and semantic diversities of responses, we performed three analyses. First, we aggregated all worker responses to a particular question into a single list corresponding to that question. Second, we compared the diversity of individual responses between Control and AUI for each question. To measure diversity for a question, we computed the number of responses divided by the number of unique responses to that question. We call this the response density.  
 Question: How was lexical diversity measured?
Output: By computing number of unique responses and number of responses divided by the number of unique responses to that question for each of the questions
This work also contributes a new dataset of INLINEFORM0 pairs of free-form natural language instructions and high-level navigation plans. This dataset was collected through Mechanical Turk using 100 simulated environments with a corresponding topological map and, to the best of our knowledge, it is the first of its kind for behavioral navigation.  
 Question: Did the authors use a crowdsourcing platform?
Output: Yes
Our dataset is constructed via distant supervision from Twitter. 
 Question: Where did they get the data for this project?
Output: Twitter
In our experiments, we compared about 60 elementary metrics, which can be organised as follows:

MT metrics

BLEU, ROUGE, METEOR, TERp

Variants of BLEU: BLEU_1gram, BLEU_2gram, BLEU_3gram, BLEU_4gram and seven smoothing methods from NLTK BIBREF32 .

Intermediate components of TERp inspired by BIBREF18 : e.g. number of insertions, deletions, shifts...

Readability metrics and other sentence-level features: FKGL and FRE, numbers of words, characters, syllables...

Metrics based on the baseline QuEst features (17 features) BIBREF28 , such as statistics on the number of words, word lengths, language model probability and INLINEFORM0 -gram frequency.

Metrics based on other features: frequency table position, concreteness as extracted from BIBREF33 's BIBREF33 list, language model probability of words using a convolutional sequence to sequence model from BIBREF34 , comparison methods using pre-trained fastText word embeddings BIBREF35 or Skip-thought sentence embeddings BIBREF36 . 
 Question: what approaches are compared?
Output: MT metrics Readability metrics and other sentence-level features Metrics based on the baseline QuEst features Metrics based on other features
In order to verify the reliability of our technique in coverage expansion for infrequent words we did a set of experiments on the Rare Word similarity dataset BIBREF6 . The dataset comprises 2034 pairs of rare words, such as ulcerate-change and nurturance-care, judged by 10 raters on a [0,10] scale. Table TABREF15 shows the results on the dataset for three pre-trained word embeddings (cf. § SECREF2 ), in their initial form as well as when enriched with additional words from WordNet. 
 Question: How are rare words defined?
Output: judged by 10 raters on a [0,10] scale
The ICSI Meeting Corpus BIBREF11 is a collection of meeting recordings that has been thoroughly annotated, including annotations for involvement hot spots BIBREF12, linguistic utterance units, and word time boundaries based on forced alignment. The dataset is comprised of 75 meetings and about 70 hours of real-time audio duration, with 6 speakers per meeting on average. Most of the participants are well-acquainted and friendly with each other. Hot spots were originally annotated with 8 levels and degrees, ranging from `not hot' to `luke warm' to `hot +'. Every utterance was labeled with one of these discrete labels by a single annotator. Hightened involvement is rare, being marked on only 1% of utterances. 
 Question: How big is ICSI meeting corpus?
Output:  75 meetings and about 70 hours of real-time audio duration
Turning to semantics, using visualizations of the activations created by different pieces of text, we show suggestive evidence that BERT distinguishes word senses at a very fine level.  We apply attention probes to the task of identifying the existence and type of dependency relation between two words. 
 Question: How were the feature representations evaluated?
Output: attention probes using visualizations of the activations created by different pieces of text
Table TABREF44 shows average results of our automatic and human evaluations. 
 Question: How big is the difference in performance between proposed model and baselines?
Output: Metric difference between Aloha and best baseline score:
Hits@1/20: +0.061 (0.3642 vs 0.3032)
MRR: +0.0572(0.5114 vs 0.4542)
F1: -0.0484 (0.3901 vs 0.4385)
BLEU: +0.0474 (0.2867 vs 0.2393)
Following lebret2016neural, we used BLEU-4, NIST-4 and ROUGE-4 as the evaluation metrics. 
 Question: What metrics are used for evaluation?
Output: BLEU-4 NIST-4 ROUGE-4
Similar trends are seen in the performance of other two state-of-the-art approaches BIBREF9 , BIBREF8 . 
 Question: What are the state of the art models?
Output: BIBREF9  BIBREF8
We use the $\textsc {BERT}_{\textsc {BASE}}$ ensemble from BIBREF3 as the single-hop QA model.  
 Question: What off-the-shelf QA model was used to answer sub-questions?
Output: $\textsc {BERT}_{\textsc {BASE}}$ ensemble from BIBREF3
As a first experiment, we compare the quality of fastText embeddings trained on (high-quality) curated data and (low-quality) massively extracted data for Twi and Yorùbá languages. The huge ambiguity in the written Twi language motivates the exploration of different approaches to word embedding estimations. In this work, we compare the standard fastText methodology to include sub-word information with the character-enhanced approach with position-based clustered embeddings (CWE-LP as introduced in Section SECREF17). 
 Question: What two architectures are used?
Output: fastText CWE-LP
Although techniques for constructing this knowledge base are outside the scope of this paper, we briefly mention them. Tables were constructed using a mixture of manual and semi-automatic techniques 
 Question: How is the semi-structured knowledge base created?
Output: using a mixture of manual and semi-automatic techniques
We constructed our seed lexicon consisting of 15 positive words and 15 negative words, as shown in Section SECREF27.  
 Question: How big is seed lexicon used for training?
Output: 30 words
The character sets of these 7 languages have little overlap except that (i) they all include common basic Latin alphabet, and (ii) both Hindi and Marathi use Devanagari script. We took the union of 7 character sets therein as the multilingual grapheme set (Section SECREF2), which contained 432 characters. 
 Question: How much of the ASR grapheme set is shared between languages?
Output: Little overlap except common basic Latin alphabet and that Hindi and Marathi languages use same script.
Last but not least, ethics and fairness are important considerations, that deserve to be studied. In that sense, detection of individual and global bias should be prioritized in order to give useful feedbacks to practitioners. Furthermore we are considering using adversarial learning as in BIBREF33 in order to ensure fairness during the training process. 
 Question: Do they analyze if their system has any bias?
Output: No
We consider the following metrics for automatic evaluation of different submissions. Joint goal accuracy has been used as the primary metric to rank the submissions.

Active Intent Accuracy: The fraction of user turns for which the active intent has been correctly predicted.

Requested Slot F1: The macro-averaged F1 score for requested slots over all eligible turns. Turns with no requested slots in ground truth and predictions are skipped.

Average Goal Accuracy: For each turn, we predict a single value for each slot present in the dialogue state. This is the average accuracy of predicting the value of a slot correctly.

Joint Goal Accuracy: This is the average accuracy of predicting all slot assignments for a given service in a turn correctly. 
 Question: How are the models evaluated?
Output: Active Intent Accuracy Requested Slot F1 Average Goal Accuracy Joint Goal Accuracy
As we only extracted references to other judicial decisions, we obtained 471,319 references from Supreme Court decisions, 167,237 references from Supreme Administrative Court decisions and 264,463 references from Constitutional Court Decisions. These are numbers of text spans identified as references prior the further processing described in Section SECREF3. 
 Question: How big is the dataset?
Output: 903019 references
Therefore, we used the crowdsourcing platform CrowdFlower (CF) for our data collection. 
 Question: How was this data collected?
Output: CrowdFlower
For the embeddings, we relied on $AraVec$ BIBREF30 for Arabic, FastText BIBREF31 for French, and Word2vec Google News BIBREF32 for English .  
 Question: What monolingual word representations are used?
Output: AraVec for Arabic, FastText for French, and Word2vec Google News for English.
We built 16 models of word embeddings using the implementation of CBOW and Skip-gram methods in the FastText tool BIBREF9 . 
 Question: What embedding algorithm is used to build the embeddings?
Output: CBOW and Skip-gram methods in the FastText tool BIBREF9
The data used for this task is drawn from the Switchboard conversational English corpus BIBREF31 . 
 Question: Which dataset do they use?
Output: Switchboard conversational English corpus
We use the LSTM network as follows. The $x$ vector is fed through the LSTM network which outputs a vector $\overrightarrow{h_i}$ for each time step $i$ from 0 to $n-1$. This is the forward LSTM. As we have access to the complete vector $x$, we can process a backward LSTM as well. This is done by computing a vector $\overleftarrow{h_i}$ for each time step $i$ from $n-1$ to 0. Finally, we concatenate the backward LSTM with the forward LSTM:

Both $\overrightarrow{h_i}$ and $\overleftarrow{h_i}$ have a dimension of $l$, which is an optimized hyperparameter. The BiLSTM output $h$ thus has dimension $2l\times n$. 
 Question: Is the LSTM bidirectional?
Output: Yes
For the Russian language, with its rich morphology, lemmatizing the training and testing data for ELMo representations yields small but consistent improvements in the WSD task.  
 Question: What other examples of morphologically-rich languages do the authors give?
Output: Russian
Starting with this list, we can locate the profile page for a user, and subsequently extract additional information, which includes fields such as name, email, occupation, industry, and so forth. We also generate two maps that delineate the gender distribution in the dataset. Our dataset provides mappings between location, profile information, and language use, which we can leverage to generate maps that reflect demographic, linguistic, and psycholinguistic properties of the population represented in the dataset. 
 Question: Which demographic dimensions of people do they obtain?
Output: occupation industry profile information language use gender
We evaluate the adversarially trained models, as shown in Table TABREF18.

After adversarial training, the performance of all the target models raises significantly, while that on the original examples remain comparable. 
 Question: How much in experiments is performance improved for models trained with generated adversarial examples?
Output: Answer with content missing: (Table 1) The performance of all the target models raises significantly, while that on the original
examples remain comparable (e.g. the overall accuracy of BERT on modified examples raises from 24.1% to 66.0% on Quora)
While it is obvious that our embeddings can be used as features for new predictive models, it is also very easy to incorporate our learned Dolores embeddings into existing predictive models on knowledge graphs. The only requirement is that the model accepts as input, an embedding layer (for entities and relations). If a model fulfills this requirement (which a large number of neural models on knowledge graphs do), we can just use Dolores embeddings as a drop-in replacement. We just initialize the corresponding embedding layer with Dolores embeddings. In our evaluation below, we show how to improve several state-of-the-art models on various tasks simply by incorporating Dolores as a drop-in replacement to the original embedding layer. 
 Question: How are meaningful chains in the graph selected?
Output: No
The second trend is that by using less than 50% of the data available the model tends to overfit the data, as indicated by the consistent increase in the validation loss after about 15 epochs (check dashed lines in right side of Figure FIGREF28 ). 
 Question: What experimental results suggest that using less than 50% of the available training examples might result in overfitting?
Output: consistent increase in the validation loss after about 15 epochs
Experimental Studies ::: Comparison with State-of-the-art Methods
Since BERT has already achieved the state-of-the-art performance of question-answering, in this section we compare our proposed model with state-of-the-art question answering models (i.e. QANet BIBREF39) and BERT-Base BIBREF26. As BERT has two versions: BERT-Base and BERT-Large, due to the lack of computational resource, we can only compare with BERT-Base model instead of BERT-Large. 
 Question: What baselines is the proposed model compared against?
Output: BERT-Base QANet
Understanding a user's intent and sentiment is of utmost importance for current intelligent chatbots to respond appropriately to human requests. However, current systems are not able to perform to their best capacity when presented with incomplete data, meaning sentences with missing or incorrect words. This scenario is likely to happen when one considers human error done in writing. In fact, it is rather naive to assume that users will always type fully grammatically correct sentences.  
 Question: How do the authors define or exemplify 'incorrect words'?
Output: typos in spellings or ungrammatical words
The evaluation of chatbots remains an open problem in the field. Recent work BIBREF25 has shown that the automatic evaluation metrics borrowed from machine translation such as BLEU score BIBREF26 tend to align poorly with human judgement. Therefore, in this paper, we mainly adopt human evaluation, along with perplexity, following the existing work. 
 Question: Is some other metrics other then perplexity measured?
Output: No
We use the morphological tagging datasets provided by the Universal Dependencies (UD) treebanks (the concatenation of the $4^\text{th}$ and $6^\text{th}$ columns of the file format) BIBREF13 . 
 Question: On which dataset is the experiment conducted?
Output: We use the morphological tagging datasets provided by the Universal Dependencies (UD) treebanks (the concatenation of the $4^\text{th}$ and $6^\text{th}$ columns of the file format) BIBREF13 .
A causal attribution dataset is a collection of text pairs that reflect cause-effect relationships proposed by humans (for example, “virus causes sickness”). These written statements identify the nodes of the network (see also our graph fusion algorithm for dealing with semantically equivalent statements) while cause-effect relationships form the directed edges (“virus” $\rightarrow $ “sickness”) of the causal attribution network. 
 Question: What are causal attribution networks?
Output: networks where nodes represent causes and effects, and directed edges represent cause-effect relationships proposed by humans
We train MaLOPa on the concantenation of training sections of all seven languages. 
 Question: How many languages have this parser been tried on?
Output: seven