niryuu/sni-each-converted huggingface dataset:
Annotator votes for one of the seven emotions, namely Ekman’s six basic emotions BIBREF1, plus the neutral. 
 Question: What labels does the dataset have?
Output: Ekman’s six basic emotions  neutral
Compared to reading comprehension based QA (RCQA) setup where the answers to a question is usually found in the given small paragraph, in the OpenBookQA setup the open book part is much larger (than a small paragraph) and is not complete as additional common knowledge may be required.  
 Question: How is OpenBookQA different from other natural language QA?
Output: in the OpenBookQA setup the open book part is much larger the open book part is much larger (than a small paragraph) and is not complete as additional common knowledge may be required
The average classification accuracy results are summarised in Table TABREF9. 
 Question: What evaluation metric is used?
Output: average classification accuracy
It is true that the above-mentioned associated caption sentences for each concept-set are human-written and do describe scenes that cover all given concepts. However, they are created under specific contexts (i.e. an image or a video) and thus might be less representative for common sense. To better measure the quality and interpretability of generative reasoners, we need to evaluate them with scenes and rationales created by using concept-sets only as the signals for annotators.

We collect more human-written scenes for each concept-set in dev and test set through crowd-sourcing via the Amazon Mechanical Turk platform. Each input concept-set is annotated by at least three different humans. The annotators are also required to give sentences as the rationales, which further encourage them to use common sense in creating their scenes. 
 Question: Are the sentences in the dataset written by humans who were shown the concept-sets?
Output: Yes
We have evaluated our models considering the F1 Score, which is the harmonic mean of precision and recall. We have run ten times the experiment for each model and considered the average F1 Score. The results are mentioned in Table TABREF11. Considering F1 Macro the models that include the multi-attention mechanism outperform the others and particularly the one with the Projected Layer has the highest performance. In three out of four pairs of models, the ones with the Projected Layer achieved better performance, so in most cases the addition of the Projected Layer had a significant enhancement.

 
 Question: Which variation provides the best results on this dataset?
Output: the model with multi-attention mechanism and a projected layer
Based on the best results (BLSTM-CNN-CRF), error analysis is performed based on five types of errors (No extraction, No annotation, Wrong range, Wrong tag, Wrong range and tag), in a way similar to BIBREF10, but we analyze on both gold labels and predicted labels (more detail in figure 1 and 2). 
 Question: What type of errors were produced by the BLSTM-CNN-CRF system?
Output: No extraction, No annotation, Wrong range, Wrong tag, Wrong range and tag
Following the formula, we can calculate the contribution of every input word makes to every output word, forming a contribution matrix of size $M \times N$, where $N$ is the output sentence length. Given the contribution matrix, we can obtain the word importance of each input word to the entire output sentence. To this end, for each input word, we first aggregate its contribution values to all output words by the sum operation, and then normalize all sums through the Softmax function. 
 Question: How do their models decide how much improtance to give to the output words?
Output: Given the contribution matrix, we can obtain the word importance of each input word to the entire output sentence.
We use two different unsupervised approaches for word sense disambiguation.  The second, called `dense model', represents synsets and contexts in a dense, low-dimensional space by averaging word embeddings. In the synset embeddings model approach, we follow SenseGram BIBREF14 and apply it to the synsets induced from a graph of synonyms.  We observe that the SenseGram-based approach for word sense disambiguation yields substantially better results in every case (Table TABREF25 ). The primary reason for that is the implicit handling of similar words due to the averaging of dense word vectors for semantically related words.  
 Question: Do the authors offer any hypothesis about why the dense mode outperformed the sparse one?
Output: Yes
The resource is composed of data from two different surveys. In both surveys subjects were asked to draw on a map (displayed under a Mercator projection) a polygon representing a given geographical descriptor, in the context of the geography of Galicia in Northwestern Spain (see Fig. FIGREF1 ). The first survey was run in order to obtain a high number of responses to be used as an evaluation testbed for modeling algorithms. It was answered by 15/16 year old students in a high school in Pontevedra (located in Western Galicia). 99 students provided answers for a list of 7 descriptors (including cardinal points, coast, inland, and a proper name). The second survey was addressed to meteorologists in the Galician Weather Agency BIBREF12 . 
 Question: Which two datasets does the resource come from?
Output: two surveys by two groups - school students and meteorologists to draw on a map a polygon representing a given geographical descriptor
We compare FSDM with four baseline methods and two ablations.

NDM BIBREF7 proposes a modular end-to-end trainable network. It applies de-lexicalization on user utterances and responses.

LIDM BIBREF9 improves over NDM by employing a discrete latent variable to learn underlying dialogue acts. This allows the system to be refined by reinforcement learning.

KVRN BIBREF13 adopts a copy-augmented Seq2Seq model for agent response generation and uses an attention mechanism on the KB. It does not perform belief state tracking.

TSCP/RL BIBREF10 is a two-stage CopyNet which consists of one encoder and two copy-mechanism-augmented decoders for belief state and response generation. 
 Question: What baselines have been used in this work?
Output: NDM, LIDM, KVRN, and TSCP/RL
We evaluate the sequence-to-sequence QRNN architecture described in SECREF5 on a challenging neural machine translation task, IWSLT German–English spoken-domain translation, applying fully character-level segmentation.  
 Question: What languages pairs are used in machine translation?
Output: German–English
The final embedding dimensionality is equal to the number of unique word labels in the training set, which is 1061. 
 Question: Which dimensionality do they use for their embeddings?
Output: 1061
Although humor is a universal construct, there is a wide variety between what each individual may find humorous. We attempt to focus on a subset of the population where we can quantitatively measure reactions: the popular Reddit r/Jokes thread. This forum is highly popular - with tens of thousands of jokes being posted monthly and over 16 million members. Although larger joke datasets exist, the r/Jokes thread is unparalleled in the amount of rated jokes it contains. To the best of our knowledge there is no comparable source of rated jokes in any other language. These Reddit posts consist of the body of the joke, the punchline, and the number of reactions or upvotes. Although this type of humor may only be most enjoyable to a subset of the population, it is an effective way to measure responses to jokes in a large group setting. 
 Question: What kind of humor they have evaluated?
Output: a subset of the population where we can quantitatively measure reactions: the popular Reddit r/Jokes thread These Reddit posts consist of the body of the joke, the punchline, and the number of reactions or upvotes.
Despite its usefulness, linked entities extracted from ELS's have issues because of low precision rates BIBREF11 and design challenges in training datasets BIBREF12 . First, the extracted entities may be ambiguous. Second, the linked entities may also be too common to be considered an entity.  
 Question: Why are current ELS's not sufficiently effective?
Output: Linked entities may be ambiguous or too common
Language Model (LM): We train a two layer recurrent neural language model with GRU cells of hidden size 512.

 Sequence-to-Sequence Attention Model (S2S): We train a two layer neural sequence to sequence model equipped with bi-linear attention function with GRU cells of hidden size 512.  Linear Dynamical System (LDS): We also train a linear dynamical system as discussed in Section SECREF1 as one of our baselines for fair comparisons. Semi-Supervised SLDS (SLDS-X%): To gauge the usability of semi-supervision, we also train semi-supervised SLDS models with varying amount of labelled sentiment tags unlike the original model which uses 100% tagged data.  
 Question: What baselines are used?
Output: a two layer recurrent neural language model with GRU cells of hidden size 512 a two layer neural sequence to sequence model equipped with bi-linear attention function with GRU cells of hidden size 512 a linear dynamical system semi-supervised SLDS models with varying amount of labelled sentiment tags
Irony Classifier: We implement a CNN classifier trained with our irony dataset. Sentiment Classifier for Irony: We first implement a one-layer LSTM network to classify ironic sentences in our dataset into positive and negative ironies. Sentiment Classifier for Non-irony: Similar to the training process of the sentiment classifier for irony, we first implement a one-layer LSTM network trained with the dataset for the sentiment analysis of common twitters to classify the non-ironies into positive and negative non-ironies. In this section, we describe some additional experiments on the transformation from ironic sentences to non-ironic sentences. 
 Question: What experiments are conducted?
Output: Irony Classifier Sentiment Classifier for Irony Sentiment Classifier for Non-irony transformation from ironic sentences to non-ironic sentences
We compare the performance with the Recurrent Entity Networks model (EntNet) BIBREF17 .  The RelNet model achieves a mean error of 0.285% across tasks which is better than the results of the EntNet model BIBREF17 . The RelNet model is able to achieve 0% test error on 11 of the tasks, whereas the EntNet model achieves 0% error on 7 of the tasks. 
 Question: What methods is RelNet compared to?
Output: We compare the performance with the Recurrent Entity Networks model (EntNet) BIBREF17
To assess the consistency of annotations and also eliminate coincidental annotations, we used agreement rates, which is calculated by dividing the number of senses under each category where the annotators annotate consistently by the total number of each kind of sense. And considering the potential impact of unbalanced distribution of senses, we also used the Kappa value. 
 Question: Which inter-annotator metric do they use?
Output: agreement rates Kappa value
Women represent 33.16% of the speakers, confirming the figures given by the GMMP report BIBREF0. 
 Question: How big is imbalance in analyzed corpora?
Output: Women represent 33.16% of the speakers
In this paper, we propose to represent each word with an expressive multimodal distribution, for multiple distinct meanings, entailment, heavy tailed uncertainty, and enhanced interpretability. For example, one mode of the word `bank' could overlap with distributions for words such as `finance' and `money', and another mode could overlap with the distributions for `river' and `creek'. It is our contention that such flexibility is critical for both qualitatively learning about the meanings of words, and for optimal performance on many predictive tasks. 
 Question: How does this compare to contextual embedding methods?
Output:  represent each word with an expressive multimodal distribution, for multiple distinct meanings, entailment, heavy tailed uncertainty, and enhanced interpretability. For example, one mode of the word `bank' could overlap with distributions for words such as `finance' and `money', and another mode could overlap with the distributions for `river' and `creek'.
Bag-of-words feature vectors were used to train a multinomial logistic regression model. Let INLINEFORM0 be the true label, where INLINEFORM1 is the total number of labels and INLINEFORM2 is the concatenation of the weight vectors INLINEFORM3 associated with the INLINEFORM4 th party then DISPLAYFORM0 
 Question: What model are the text features used in to provide predictions?
Output:  multinomial logistic regression
In this paper we focus on the skipgram approach with random negative examples proposed in BIBREF0 .  
 Question: Do they use skipgram version of word2vec?
Output: Yes
We perform our experiments using ActivityNet Captions dataset BIBREF2 that is considered as the standard benchmark for dense video captioning task. The dataset contains approximately 20k videos from YouTube and split into 50/25/25 % parts for training, validation, and testing, respectively.  
 Question: What domain does the dataset fall into?
Output: YouTube videos
Probing into the assorted empirical results may help us discover some interesting phenomenons:

The advantage of pre-training gradually diminishes with the increase of labeled data BIBREF14, BIBREF17, BIBREF18.

Fixed representations yield better results than fine-tuning in some cases BIBREF24.

Overall, pre-training the Seq2Seq encoder outperforms pre-training the decoder BIBREF24, BIBREF17, BIBREF15, BIBREF16. 
 Question: What experimental phenomena are presented?
Output: The advantage of pre-training gradually diminishes with the increase of labeled data Fixed representations yield better results than fine-tuning in some cases pre-training the Seq2Seq encoder outperforms pre-training the decoder
Even though this corpus has incorrect sentences and their emotional labels, they lack their respective corrected sentences, necessary for the training of our model. In order to obtain this missing information, we outsource native English speakers from an unbiased and anonymous platform, called Amazon Mechanical Turk (MTurk) BIBREF19, which is a paid marketplace for Human Intelligence Tasks (HITs). We use this platform to create tasks for native English speakers to format the original incorrect tweets into correct sentences. Some examples are shown in Table TABREF12. The dataset used to evaluate the models' performance is the Chatbot Natural Language Unerstanding (NLU) Evaluation Corpus, introduced by Braun et al. BIBREF20 to test NLU services. It is a publicly available benchmark and is composed of sentences obtained from a German Telegram chatbot used to answer questions about public transport connections. The dataset has two intents, namely Departure Time and Find Connection with 100 train and 106 test samples, shown in Table TABREF18. Even though English is the main language of the benchmark, this dataset contains a few German station and street names. 
 Question: Do they report results only on English datasets?
Output: Yes
A maximum of three contributors will listen to any audio clip. If an $<$audio,transcript$>$ pair first receives two up-votes, then the clip is marked as valid. If instead the clip first receives two down-votes, then it is marked as invalid. A contributor may switch between recording and validation as they wish. Only clips marked as valid are included in the official training, development, and testing sets for each language. Clips which did not recieve enough votes to be validated or invalidated by the time of release are released as “other”. The train, test, and development sets are bucketed such that any given speaker may appear in only one. This ensures that contributors seen at train time are not seen at test time, which would skew results. Additionally, repetitions of text sentences are removed from the train, test, and development sets of the corpus. 
 Question: How is validation of the data performed?
Output: A maximum of three contributors will listen to any audio clip. If an $<$audio,transcript$>$ pair first receives two up-votes, then the clip is marked as valid. If instead the clip first receives two down-votes, then it is marked as invalid.
his reduces significantly the needed training time: SBERT can be tuned in less than 20 minutes, while yielding better results than comparable sentence embedding methods. 
 Question: How much time takes its training?
Output: 20 minutes
As detailed before, we adapted the cluster-linking dataset from rupnik2016news to evaluate our online crosslingual clustering approach. 
 Question: What are the sources of the datasets?
Output: rupnik2016news
The memory mechanism is adopted in order to enable the model to look beyond localized features and have access to the entire sequence. 
 Question: Which features do they use?
Output: beyond localized features and have access to the entire sequence
We use UltraSuite: a repository of ultrasound and acoustic data gathered from child speech therapy sessions BIBREF15 . We used all three datasets from the repository: UXTD (recorded with typically developing children), and UXSSD and UPX (recorded with children with speech sound disorders). In total, the dataset contains 13,815 spoken utterances from 86 speakers, corresponding to 35.9 hours of recordings. The utterances have been categorised by the type of task the child was given, and are labelled as: Words (A), Non-words (B), Sentence (C), Articulatory (D), Non-speech (E), or Conversations (F).  
 Question: Do they annotate their own dataset or use an existing one?
Output: Use an existing one
Adaptation to food domain captioning Consequently, the food dataset has 3,806 images for training and 1,775 for validation. The non-food dataset has 78,976 images for training and 38,749 for validation. 
 Question: How many examples are there in the target domain?
Output: the food dataset has 3,806 images for training
We obtain word vectors of size 300 from the learned word embeddings. To represent a Twitter profile, we retrieve word vectors for all the words that appear in a particular profile including the words appear in tweets, profile description, words extracted from emoji, cover and profile images converted to textual formats, and words extracted from YouTube video comments and descriptions for all YouTube videos shared in the user's timeline. Those word vectors are combined to compute the final feature vector for the Twitter profile. 
 Question: How in YouTube content translated into a vector format?
Output: words extracted from YouTube video comments and descriptions for all YouTube videos shared in the user's timeline
Our goal is to decouple the content selection from the decoder by introducing an extra content selector. The most intuitive way is training the content selector to target some heuristically extracted contents. For example, we can train the selector to select overlapped words between the source and target BIBREF6 , sentences with higher tf-idf scores BIBREF20 or identified image objects that appear in the caption BIBREF21 . INLINEFORM0 as Latent Variable: Another way is to treat INLINEFORM1 as a latent variable and co-train selector and generator by maximizing the marginal data likelihood. By doing so, the selector has the potential to automatically explore optimal selecting strategies best fit for the corresponding generator component. Reinforce-select (RS) BIBREF24 , BIBREF9 utilizes reinforcement learning to approximate the marginal likelihood. We propose Variational Reinforce-Select (VRS) which applies variational inference BIBREF10 for variance reduction. 
 Question: How is the model trained to do only content selection?
Output: target some heuristically extracted contents treat INLINEFORM1 as a latent variable and co-train selector and generator by maximizing the marginal data likelihood reinforcement learning to approximate the marginal likelihood  Variational Reinforce-Select (VRS) which applies variational inference BIBREF10 for variance reduction
We modeled the relationship between word count and the two metrics of user engagement (overall rating, mean number of turns) in separate linear regressions. 
 Question: What are all the metrics to measure user engagement?
Output: overall rating mean number of turns
We describe our rules for WikiSQL here. We first detect WHERE values, which exactly match to table cells. After that, if a cell appears at more than one column, we choose the column name with more overlapped words with the question, with a constraint that the number of co-occurred words is larger than 1. By default, a WHERE operator is INLINEFORM0 , except for the case that surrounding words of a value contain keywords for INLINEFORM1 and INLINEFORM2 . Then, we deal with the SELECT column, which has the largest number of co-occurred words and cannot be same with any WHERE column. By default, the SELECT AGG is NONE, except for matching to any keywords in Table TABREF8 . The coverage of our rule on training set is 78.4%, with execution accuracy of 77.9%. Our rule for KBQA is simple without using a curated mapping dictionary. The pipeline of rules in SequentialQA is similar to that of WikiSQL. Compared to the grammar of WikiSQL, the grammar of SequentialQA has additional actions including copying the previous-turn logical form, no greater than, no more than, and negation. 
 Question: How many rules had to be defined?
Output: WikiSQL - 2 rules (SELECT, WHERE)
SimpleQuestions - 1 rule
SequentialQA - 3 rules (SELECT, WHERE, COPY)
First, many news articles begin with reporter names, media agencies, dates or other contents irrelevant to the content, e.g. “New York (CNN) –”, “Jones Smith, May 10th, 2018:”. We therefore apply simple regular expressions to remove these prefixes.

Second, to ensure that the summary is concise and the article contains enough salient information, we only keep articles with 10-150 words in the top three sentences and 150-1200 words in the rest, and that contain at least 6 sentences in total. In this way, we filter out i) articles with excessively long content to reduce memory consumption; ii) very short leading sentences with little information which are unlikely to be a good summary. To encourage the model to generate abstrative summaries, we also remove articles where any of the top three sentences is exactly repeated in the rest of the article.

Third, we try to remove articles whose top three sentences may not form a relevant summary. 
 Question: What does the data cleaning and filtering process consist of?
Output: many news articles begin with reporter names, media agencies, dates or other contents irrelevant to the content to ensure that the summary is concise and the article contains enough salient information, we only keep articles with 10-150 words in the top three sentences and 150-1200 words in the rest, and that contain at least 6 sentences in total we try to remove articles whose top three sentences may not form a relevant summary
In BIBREF0 , BIBREF1 , BIBREF2 , Pasca et al. firstly extract potential class-attribute pairs using linguistically motivated patterns from unstructured text including query logs and query sessions, and then score the attributes using the Bayes model. In BIBREF3 , Rahul Rai proposed to identify product attributes from customer online reviews using part-of-speech(POS) tagging patterns, and to evaluate their importance with several different frequency metrics. In BIBREF4 , Lee et al. developed a system to extract concept-attribute pairs from multiple data sources, such as Probase, general web documents, query logs and external knowledge base, and aggregate the weights from different sources into one consistent typicality score using a Ranking SVM model. In BIBREF5 , Li et al. introduced the OntoRank algorithm for ranking the importance of semantic web objects at three levels of granularity: document, terms and RDF graphs. The algorithm is based on the rational surfer model, successfully used in the Swoogle semantic web search engine. 
 Question: What are the traditional methods to identifying important attributes?
Output: automated attribute-value extraction score the attributes using the Bayes model evaluate their importance with several different frequency metrics aggregate the weights from different sources into one consistent typicality score using a Ranking SVM model OntoRank algorithm
For comparison, we also report the performance of the error detection system by Rei2016, trained using the same FCE dataset. 
 Question: What was the baseline used?
Output: error detection system by Rei2016
The question is encoded with a 1024-dimensional LSTM model that takes in a one-hot descriptor of each word in the question. The image is described with the 4096-dimensional output from the last fully connected layer of the Convolutional Neural Network (CNN), VGG16 BIBREF25 . The system performs an element-wise multiplication of the image and question features, after linearly transforming the image descriptor to 1024 dimensions. The final layer of the architecture is a softmax layer. 
 Question: What is the model architecture used?
Output: LSTM to encode the question, VGG16 to extract visual features. The outputs of LSTM and VGG16 are multiplied element-wise and sent to a softmax layer.
For the entailment classifier we compare Decomposable Attention BIBREF2 , BIBREF3 as implemented in the official baseline, ESIM BIBREF4 , and a transformer network with pre-trained weights BIBREF5 .  
 Question: Which pre-trained transformer do they use?
Output: BIBREF5
To investigate the trade-off between speed and performance, we compare our technique to standard models with and without attention on a Sequence Copy Task of varying length like in BIBREF14 . For this purpose we use 4 large machine translation datasets of WMT'17 on the following language pairs: English-Czech (en-cs, 52M examples), English-German (en-de, 5.9M examples), English-Finish (en-fi, 2.6M examples), and English-Turkish (en-tr, 207,373 examples). 
 Question: Which datasets are used in experiments?
Output: Sequence Copy Task and WMT'17
We compare our model with two sets of baselines:

MemN2N BIBREF12 is an end-to-end trainable version of memory networks BIBREF9 . Attentive and Impatient Readers BIBREF6 use bidirectional LSTMs to encode question and evidence, and do classification over a large vocabulary based on these two encodings. 
 Question: What are the baselines?
Output: MemN2N BIBREF12 Attentive and Impatient Readers BIBREF6
Hausa language is the second most spoken indigenous language in Africa with over 40 million native speakers BIBREF20, and one of the three major languages in Nigeria, along with Igbo and Yorùbá. The language is native to the Northern part of Nigeria and the southern part of Niger, and it is widely spoken in West and Central Africa as a trade language in eight other countries: Benin, Ghana, Cameroon, Togo, Côte d'Ivoire, Chad, Burkina Faso, and Sudan. Yorùbá language is the third most spoken indigenous language in Africa after Swahilli and Hausa with over 35 million native speakers BIBREF20. The language is native to the South-western part of Nigeria and the Southern part of Benin, and it is also spoken in other countries like Republic of Togo, Ghana, Côte d'Ivoire, Sierra Leone, Cuba and Brazil. 
 Question: In which countries are Hausa and Yor\`ub\'a spoken?
Output: Nigeria Benin, Ghana, Cameroon, Togo, Côte d'Ivoire, Chad, Burkina Faso, and Sudan Republic of Togo, Ghana, Côte d'Ivoire, Sierra Leone, Cuba and Brazil
In order to derive the discourse features, an entity grid is constructed by feeding the document through an NLP pipeline to identify salient entities. Two flavors of discourse features are created by populating the entity grid with either (i) grammatical relations (GR) or (ii) RST discourse relations (RST).  
 Question: What discourse features are used?
Output: Entity grid with grammatical relations and RST discourse relations.
Our visual model is based on fine-tuning an Inception V3 model BIBREF1 over visual renderings of documents, while our textual model is based on a hierarchical biLSTM. We further combine the two into a joint model.  We applied neural network models to capture visual features given visual renderings of documents. Experimental results show that we achieve a 2.9% higher accuracy than state-of-the-art approaches based on textual features over Wikipedia, and performance competitive with or surpassing state-of-the-art approaches over arXiv. We further proposed a joint model, combining textual and visual representations, to predict the quality of a document.  
 Question: What kind of model do they use?
Output: visual model is based on fine-tuning an Inception V3 model BIBREF1 over visual renderings of documents, while our textual model is based on a hierarchical biLSTM. We further combine the two into a joint model.  neural network models
For training data, 3 million webpages from this corpus were processed with a CCG parser to produce logical forms BIBREF10 . We also used the test set created by Krishnamurthy and Mitchell, which contains 220 queries generated in the same fashion as the training data from a separate section of ClueWeb. However, as they did not release a development set with their data, we used this set as a development set. This final test set contains 307 queries. 
 Question: How big is their dataset?
Output: 3 million webpages processed with a CCG parser for training, 220 queries for development, and 307 queries for testing
Akkadian is divided into six dialects in the dataset: Old Babylonian, Middle Babylonian peripheral, Standard Babylonian, Neo Babylonian, Late Babylonian, and Neo Assyrian BIBREF14 . 
 Question: Which language is divided into six dialects in the task mentioned in the paper?
Output: Akkadian.
In cooperation with the ruling regime, Weibo sets strict control over the content published under its service BIBREF0. According to Zhu et al. zhu-etal:2013, Weibo uses a variety of strategies to target censorable posts, ranging from keyword list filtering to individual user monitoring. Among all posts that are eventually censored, nearly 30% of them are censored within 5–30 minutes, and nearly 90% within 24 hours BIBREF1. We hypothesize that the former are done automatically, while the latter are removed by human censors. 
 Question: Is is known whether Sina Weibo posts are censored by humans or some automatic classifier?
Output: No
The following handcrafted features were used for the model:

Bias feature

Token feature

Uppercase feature (y/n)

Titlecase feature (y/n)

Character trigram feature

Quotation feature (y/n)

Word suffix feature (last three characters)

POS tag (provided by spaCy utilities)

Word shape (provided by spaCy utilities)

Word embedding (see Table TABREF26) 
 Question: What are the handcrafted features used?
Output: Bias feature Token feature Uppercase feature (y/n) Titlecase feature (y/n) Character trigram feature Quotation feature (y/n) Word suffix feature (last three characters) POS tag (provided by spaCy utilities) Word shape (provided by spaCy utilities) Word embedding (see Table TABREF26)
Recurrent Neural Network (RNN) Language Models
are trained to output the probability distribution of the upcoming word given a context, without explicitly representing the structure of the context BIBREF9, BIBREF10. ActionLSTM
models the linearized bracketed tree structure of a sentence by learning to predict the next action required to construct a phrase-structure parse BIBREF16. Generative Recurrent Neural Network Grammars (RNNG)
jointly model the word sequence as well as the underlying syntactic structure BIBREF18. 
 Question: What are the baseline models?
Output: Recurrent Neural Network (RNN) ActionLSTM Generative Recurrent Neural Network Grammars (RNNG)
Experiments on two publicly available datasets (Camrest BIBREF11 and InCar Assistant BIBREF6) confirm the effectiveness of the KB-retriever. 
 Question: Which dialog datasets did they experiment with?
Output: Camrest InCar Assistant
Open-domain chatbots are more generic dialogue systems. An example is the Poly-encoder from BIBREF7 humeau2019real. It outperforms the Bi-encoder BIBREF8, BIBREF9 and matches the performance of the Cross-encoder BIBREF10, BIBREF11 while maintaining reasonable computation time. It performs strongly on downstream language understanding tasks involving pairwise comparisons, and demonstrates state-of-the-art results on the ConvAI2 challenge BIBREF12. Feed Yourself BIBREF13 is an open-domain dialogue agent with a self-feeding model. When the conversation goes well, the dialogue becomes part of the training data, and when the conversation does not, the agent asks for feedback. Lastly, Kvmemnn BIBREF14 is a key-value memory network with a knowledge base that uses a key-value retrieval mechanism to train over multiple domains simultaneously. We use all three of these models as baselines for comparison.  We compare against four dialogue system baselines: Kvmemnn, Feed Yourself, Poly-encoder, and a BERT bi-ranker baseline trained on the Persona-Chat dataset using the same training hyperparameters (including learning rate scheduler and length capping settings) described in Section SECREF20. 
 Question: What baseline models are used?
Output: the Poly-encoder from BIBREF7 humeau2019real Feed Yourself BIBREF13 is an open-domain dialogue agent with a self-feeding model Kvmemnn BIBREF14 is a key-value memory network with a knowledge base that uses a key-value retrieval mechanism to train over multiple domains simultaneously We compare against four dialogue system baselines: Kvmemnn, Feed Yourself, Poly-encoder, and a BERT bi-ranker baseline trained on the Persona-Chat dataset using the same training hyperparameters (including learning rate scheduler and length capping settings) described in Section SECREF20. a BERT bi-ranker
We see that A-gen performance improves significantly with the joint model: both F1 and EM increase by about 10 percentage points. 
 Question: How much improvement does jointly learning QA and QG give, compared to only training QA?
Output: We see that A-gen performance improves significantly with the joint model: both F1 and EM increase by about 10 percentage points.
We defined the intents with guidance from queries collected using a scoping crowdsourcing task, which prompted crowd workers to provide questions and commands related to topic domains in the manner they would interact with an artificially intelligent assistant. We manually grouped data generated by scoping tasks into intents.  
 Question: How was the dataset annotated?
Output: intents are annotated manually with guidance from queries collected using a scoping crowdsourcing task
S2-GORC is a large citation graph dataset which includes full texts of 8.1 million scientific documents. We select a subset of 154K computer science articles as our corpus. From these, we extract 622K citing sentences that link back to other documents in our corpus. We hold 2500 examples for each of the validation and test sets. 
 Question: What is the size of the corpus?
Output: 8.1 million scientific documents 154K computer science articles 622K citing sentences
Observe that in only about 12% (7 of the top 59) of the most cited areas of research, women received higher average citations than men. These include: sentiment analysis, information extraction, document summarization, spoken dialogue, cross lingual (research), dialogue, systems, language generation. (Of course, note that some of the 59 areas, as estimated using title term bigrams, are overlapping. Also, we did not include large scale in the list above because the difference in averages is very small and it is not really an area of research.) Thus, the citation gap is common across a majority of the high-citations areas within NLP. 
 Question: Which NLP area have the highest average citation for woman author?
Output: sentiment analysis, information extraction, document summarization, spoken dialogue, cross lingual (research), dialogue, systems, language generation
The input document/summary that may have unordered sentences is processed so that it will have sentences clustered together. To test our approach, we jumble the ordering of sentences in a document, process the unordered document and compare the similarity of the output document with the original document. To structure an unordered document is an essential task in many applications. It is a post-requisite for applications like multiple document extractive text summarization where we have to present a summary of multiple documents. It is a prerequisite for applications like question answering from multiple documents where we have to present an answer by processing multiple documents. 
 Question: What is an unordered text document, do these arise in real-world corpora?
Output: A unordered text document is one where sentences in the document are disordered or jumbled. It doesn't appear that unordered text documents appear in corpora, but rather are introduced as part of processing pipeline.
We used a set of global network indicators which allow us to encode each network layer by a tuple of features. Then we simply concatenated tuples as to represent each multi-layer network with a single feature vector. We used the following global network properties:

Number of Strongly Connected Components (SCC): a Strongly Connected Component of a directed graph is a maximal (sub)graph where for each pair of vertices $u,v$ there is a path in each direction ($u\rightarrow v$, $v\rightarrow u$).

Size of the Largest Strongly Connected Component (LSCC): the number of nodes in the largest strongly connected component of a given graph.

Number of Weakly Connected Components (WCC): a Weakly Connected Component of a directed graph is a maximal (sub)graph where for each pair of vertices $(u, v)$ there is a path $u \leftrightarrow v$ ignoring edge directions.

Size of the Largest Weakly Connected Component (LWCC): the number of nodes in the largest weakly connected component of a given graph.

Diameter of the Largest Weakly Connected Component (DWCC): the largest distance (length of the shortest path) between two nodes in the (undirected version of) largest weakly connected component of a graph.

Average Clustering Coefficient (CC): the average of the local clustering coefficients of all nodes in a graph; the local clustering coefficient of a node quantifies how close its neighbours are to being a complete graph (or a clique). It is computed according to BIBREF28.

Main K-core Number (KC): a K-core BIBREF13 of a graph is a maximal sub-graph that contains nodes of internal degree $k$ or more; the main K-core number is the highest value of $k$ (in directed graphs the total degree is considered).

Density (d): the density for directed graphs is $d=\frac{|E|}{|V||V-1|}$, where $|E|$ is the number of edges and $|N|$ is the number of vertices in the graph; the density equals 0 for a graph without edges and 1 for a complete graph.

Structural virality of the largest weakly connected component (SV): this measure is defined in BIBREF14 as the average distance between all pairs of nodes in a cascade tree or, equivalently, as the average depth of nodes, averaged over all nodes in turn acting as a root; for $|V| > 1$ vertices, $SV=\frac{1}{|V||V-1|}\sum _i\sum _j d_{ij}$ where $d_{ij}$ denotes the length of the shortest path between nodes $i$ and $j$. This is equivalent to compute the Wiener's index BIBREF29 of the graph and multiply it by a factor $\frac{1}{|V||V-1|}$. In our case we computed it for the undirected equivalent graph of the largest weakly connected component, setting it to 0 whenever $V=1$. 
 Question: What are the global network features which quantify different aspects of the sharing process?
Output: Number of Strongly Connected Components (SCC) Size of the Largest Strongly Connected Component (LSCC) Number of Weakly Connected Components (WCC) Size of the Largest Weakly Connected Component (LWCC) Diameter of the Largest Weakly Connected Component (DWCC) Average Clustering Coefficient (CC) Main K-core Number (KC) Density (d)
Building on these results, we take a somewhat more systematic approach to looking for interpretable hidden state dimensions, by using decision trees to predict individual hidden state dimensions (Figure 2 ). We visualize the overall dynamics of the hidden states by coloring the training data with the k-means clusters on the state vectors (Figures 3 , 3 ). In Figures 3 and 3 , we color-code the training data with the 10 HMM states. In Figures 3 and 3 , we apply k-means clustering to the LSTM state vectors, and color-code the training data with the clusters. 
 Question: Which methods do the authors use to reach the conclusion that LSTMs and HMMs learn complementary information?
Output: decision trees to predict individual hidden state dimensions apply k-means clustering to the LSTM state vectors, and color-code the training data with the clusters
Four datasets are used in our work, including CoNLL 2003 German BIBREF9 , CoNLL 2002 Spanish BIBREF10 , OntoNotes 4 BIBREF11 and Weibo NER BIBREF12 .  Table TABREF22 shows the results on Chinese OntoNotes 4.0.  
 Question: Which languages do they work with?
Output: German Spanish Chinese
We can see that how 1.10intelligence is used varies by field: in computer science the most similar words include 1.10artificial and 1.10ai; in finance, similar words include 1.10abilities and 1.10consciousness. 
 Question: Which words are used differently across ArXiv?
Output: intelligence
We call the model as a modified Toulmin's model. It contains five argument components, namely claim, premise, backing, rebuttal, and refutation. When annotating a document, any arbitrary token span can be labeled with an argument component; the components do not overlap.  
 Question: What argument components do the ML methods aim to identify?
Output: claim, premise, backing, rebuttal, and refutation
Los resultados de la evaluación se presentan en la Tabla TABREF42, en la forma de promedios normalizados entre [0,1] y de su desviación estándar $\sigma $. 
 Question: What evaluation metrics did they look at?
Output: accuracy with standard deviation
When evaluating classifiers it is common to use accuracy, precision and recall as well as Hamming loss.  It has been shown that when calculating precision and recall on multi-label classifiers, it can be advantageous to use micro averaged precision and recall BIBREF6 . The formulas for micro averaged precision are expressed as DISPLAYFORM0 DISPLAYFORM1 
 Question: what evaluation metrics are discussed?
Output: precision  recall  Hamming loss micro averaged precision and recall
Very recently, the success of deep neural networks in many natural language processing tasks ( BIBREF20 ) has inspired new work in abstractive summarization . BIBREF2 propose a neural attention model with a convolutional encoder to solve this task.  More recently, BIBREF4 extended BIBREF2 's work with an RNN decoder, and BIBREF8 proposed an RNN encoder-decoder architecture for summarization. Both techniques are currently the state-of-the-art on the DUC competition. 
 Question: What is the state-of-the art?
Output: neural attention model with a convolutional encoder with an RNN decoder and RNN encoder-decoder
Although we create new models for tasks such as sentiment analysis and gender detection as part of AraNet, our focus is more on putting together the toolkit itself and providing strong baselines that can be compared to. Hence, although we provide some baseline models for some of the tasks, we do not explicitly compare to previous research since most existing works either exploit smaller data (and so it will not be a fair comparison), use methods pre-dating BERT (and so will likely be outperformed by our models) . For many of the tasks we model, there have not been standard benchmarks for comparisons across models. This makes it difficult to measure progress and identify areas worthy of allocating efforts and budgets. 
 Question: What models did they compare to?
Output:  we do not explicitly compare to previous research since most existing works either exploit smaller data (and so it will not be a fair comparison), use methods pre-dating BERT (and so will likely be outperformed by our models)
A higher score indicates better step ordering (with a maximum score of 2). tab:coherencemetrics shows that our personalized models achieve average recipe-level coherence scores of 1.78-1.82, surpassing the baseline at 1.77. On average, human evaluators preferred personalized model outputs to baseline 63% of the time, confirming that personalized attention improves the semantic plausibility of generated recipes. 
 Question: What were their results on the new dataset?
Output: average recipe-level coherence scores of 1.78-1.82 human evaluators preferred personalized model outputs to baseline 63% of the time
We also are especially interested in seeing this model applied to different languages. 
 Question: Have the authors tried this approach on other languages?
Output: No
During each evaluation, target terms are masked, predicted, and then compared to the masked (known) value. 
 Question: Are the Transformers masked?
Output: Yes
We use the full-length ROUGE F1 and METEOR as our main evaluation metrics. 
 Question: What evaluation metrics do they use?
Output: ROUGE F1 METEOR
Detection: In hypernymy detection, the task is to classify whether pairs of words are in a hypernymy relation. Direction: In direction prediction, the task is to identify which term is broader in a given pair of words. Graded Entailment: In graded entailment, the task is to quantify the degree to which a hypernymy relation holds. 
 Question: What hypernymy tasks do they study?
Output: Detection Direction Graded Entailment
The current configuration of Aristo comprises of eight solvers, described shortly, each of which attempts to answer a multiple choice question. To study particular phenomena and develop solvers, the project has created larger datasets to amplify and study different problems, resulting in 10 new datasets and 5 large knowledge resources for the community.

The solvers can be loosely grouped into:

Statistical and information retrieval methods

Reasoning methods

Large-scale language model methods Over the life of the project, the relative importance of the methods has shifted towards large-scale language methods. The field of NLP has advanced substantially with the advent of large-scale language models such as ELMo (BID6), ULMFit (BID37), GPT (BID38), BERT (BID7), and RoBERTa (BID8). These models are trained to perform various language prediction tasks such as predicting a missing word or the next sentence, using large amounts of text (e.g., BERT was trained on Wikipedia + the Google Book Corpus of 10,000 books). They can also be fine-tuned to new language prediction tasks, such as question-answering, and have been remarkably successful in the few months that they have been available. We apply BERT to multiple choice questions by treating the task as classification: Given a question $q$ with answer options $a_{i}$ and optional background knowledge $K_{i}$, we provide it to BERT as:

[CLS] $K_i$ [SEP] $q$ [SEP] $a_{i}$ [SEP] The AristoBERT solver uses three methods to apply BERT more effectively. First, we retrieve and supply background knowledge along with the question when using BERT. This provides the potential for BERT to “read” that background knowledge and apply it to the question, although the exact nature of how it uses background knowledge is more complex and less interpretable. Second, we fine-tune BERT using a curriculum of several datasets, including some that are not science related. Finally, we ensemble different variants of BERT together. 
 Question: Is Aristo just some modern NLP model (ex. BERT) finetuned od data specific for this task?
Output: Yes