niryuu/sni-each-converted huggingface dataset:
We apply an off-the-shelf tool for emotion recognition (the manufacturer cannot be disclosed due to licensing restrictions). It delivers frame-by-frame scores ($\in [0;100]$) for discrete emotional states of joy, anger and fear. We extract the audio signal for the same sequence as described for facial expressions and apply an off-the-shelf tool for emotion recognition. The software delivers single classification scores for a set of 24 discrete emotions for the entire utterance. 
 Question: What are the emotion detection tools used for audio and face input?
Output: We apply an off-the-shelf tool for emotion recognition (the manufacturer cannot be disclosed due to licensing restrictions)
In this section we describe how to evaluate and compare the outcomes of algorithms which assign relevance scores to words (such as LRP or SA) through intrinsic validation. Furthermore, we propose a measure of model explanatory power based on an extrinsic validation procedure. 
 Question: Are the document vectors that the authors introduce evaluated in any way other than the new way the authors propose?
Output: Yes
We use micro-averaged F1 score on all labels as the evaluation metric. 
 Question: What metris are used for evaluation?
Output: micro-averaged F1 score
Table TABREF10 reports our main results. Our models are built on top of prior works with the additional Attention Supervision Module as described in Section SECREF3 . Specifically, we denote by Attn-* our adaptation of the respective model by including our Attention Supervision Module. We highlight that MCB model is the winner of VQA challenge 2016 and MFH model is the best single model in VQA challenge 2017. In Table TABREF10 , we can observe that our proposed model achieves a significantly boost on rank-correlation with respect to human attention. Furthermore, our model outperforms alternative state-of-art techniques in terms of accuracy in answer prediction. Specifically, the rank-correlation for MFH model increases by 36.4% when is evaluated in VQA-HAT dataset and 7.7% when is evaluated in VQA-X. This indicates that our proposed methods enable VQA models to provide more meaningful and interpretable results by generating more accurate visual grounding. 
 Question: By how much do they outperform existing state-of-the-art VQA models?
Output: the rank-correlation for MFH model increases by 36.4% when is evaluated in VQA-HAT dataset and 7.7% when is evaluated in VQA-X
Therefore, we decided to compile our own corpora based on English documents, which we crawled from different publicly accessible sources. 
 Question: Do they report results only on English data?
Output: Yes
In an offline step, we organize the content of each instance in a graph where each node represents a sentence from the passages and the question. Then, we add edges between nodes using the following topology:

we fully connect nodes that represent sentences from the same passage (dotted-black);

we fully connect nodes that represent the first sentence of each passage (dotted-red);

we add an edge between the question and every node for each passage (dotted-blue). 
 Question: How are some nodes initially connected based on text structure?
Output: we fully connect nodes that represent sentences from the same passage we fully connect nodes that represent the first sentence of each passage we add an edge between the question and every node for each passage
We compare our HR-VAE model with three strong baselines using VAE for text modelling:

VAE-LSTM-base: A variational autoencoder model which uses LSTM for both encoder and decoder. KL annealing is used to tackled the latent variable collapse issue BIBREF0;

VAE-CNN: A variational autoencoder model with a LSTM encoder and a dilated CNN decoder BIBREF7;

vMF-VAE: A variational autoencoder model using LSTM for both encoder and decoder where the prior distribution is the von Mises-Fisher (vMF) distribution rather than a Gaussian distribution BIBREF5. 
 Question: Do they compare against state of the art text generation?
Output: Yes
Although no perfect matches exist, we decided upon the Quora duplicate question dataset BIBREF22 as the best match. To study the embeddings, we computed the euclidean distance between the two questions using various embeddings, to study the distance between semantically similar and dissimilar questions. 
 Question: What existing corpus is used for comparison in these experiments?
Output: Quora duplicate question dataset BIBREF22
We choose the following three models as the baselines:

K-means is a well known data clustering algorithm, we implement the algorithm using sklearn toolbox, and represent documents using bag-of-words weighted by TF-IDF.

LEM BIBREF13 is a Bayesian modeling approach for open-domain event extraction. It treats an event as a latent variable and models the generation of an event as a joint distribution of its individual event elements. We implement the algorithm with the default configuration.

DPEMM BIBREF14 is a non-parametric mixture model for event extraction. It addresses the limitation of LEM that the number of events should be known beforehand. We implement the model with the default configuration. 
 Question: What baseline approaches does this approach out-perform?
Output: K-means LEM BIBREF13 DPEMM BIBREF14
Following BIBREF21, we sample $K$ captions for each image when applying REINFORCE: ${\hat{c}}_1 \ldots {\hat{c}}_K$, ${\hat{c}}_k \sim p_{\theta }(c|I)$,

The baseline for each sampled caption is defined as the average reward of the rest samples. 
 Question: What baseline function is used in REINFORCE algorithm?
Output: baseline for each sampled caption is defined as the average reward of the rest samples
Our systems are attentional encoder-decoder networks BIBREF0 . We base our implementation on the dl4mt-tutorial, which we enhanced with new features such as ensemble decoding and pervasive dropout. 
 Question: what are the baseline systems?
Output: attentional encoder-decoder networks BIBREF0
We test our proposed approach for binary classification on either sarcasm or irony, on seven benchmark datasets retrieved from different media sources. Below we describe each dataset, please see Table TABREF1 below for a summary. Reddit: BIBREF21 collected SARC, a corpus comprising of 600.000 sarcastic comments on Reddit. We use main subset, SARC 2.0, and the political subset, SARC 2.0 pol. Twitter: We use the Twitter dataset provided for the SemEval 2018 Task 3, Irony Detection in English Tweets BIBREF18 . The dataset was manually annotated using binary labels. We also use the dataset by BIBREF4 , which is manually annotated for sarcasm. Finally, we use the dataset by BIBREF20 , who collected a user self-annotated corpus of tweets with the #sarcasm hashtag. Online Dialogues: We utilize the Sarcasm Corpus V1 (SC-V1) and the Sarcasm Corpus V2 (SC-V2), which are subsets of the Internet Argument Corpus (IAC). Compared to other datasets in our selection, these differ mainly in text length and structure complexity BIBREF22 . 
 Question: What are the three different sources of data?
Output: Twitter Reddit Online Dialogues
The model is sequential in nature, consisting of the following steps: read text, process it into a dynamic relational memory and then attention conditioned on the question generates the answer. We model the dynamic memory in a fashion similar to Recurrent Entity Networks BIBREF17 and then equip it with an additional relational memory. 
 Question: What is the architecture of the neural network?
Output: extends memory-augmented neural networks with a relational memory to reason about relationships between multiple entities present within the text.  The model is sequential in nature, consisting of the following steps: read text, process it into a dynamic relational memory and then attention conditioned on the question generates the answer. We model the dynamic memory in a fashion similar to Recurrent Entity Networks BIBREF17 and then equip it with an additional relational memory.
We compare our system to a variety of baselines. (1) Rank by the number of times a citation is mentioned in the document. (2) Rank by the number of times the citation is cited in the literature (citation impact). (3) Rank using Google Scholar Related Articles. (4) Rank by the TF*IDF weighted cosine similarity. (5) Rank using a learning-to-rank model trained on text similarity rankings. The first two baseline systems are models where the values are ordered from highest to lowest to generate the ranking. The idea behind them is that the number of times a citation is mentioned in an article, or the citation impact may already be good indicators of their closeness. The text similarity model is trained using the same features and methods used by the annotation model, but trained using text similarity rankings instead of the author's judgments. 
 Question: what were the baselines?
Output: Rank by the number of times a citation is mentioned in the document  Rank by the number of times the citation is cited in the literature (citation impact).  Rank using Google Scholar Related Articles. Rank by the TF*IDF weighted cosine similarity.  ank using a learning-to-rank model trained on text similarity rankings
MTMSN BIBREF4 is the first, and only model so far, that specifically tried to tackle the multi-span questions of DROP.  
 Question: What is the previous model that attempted to tackle multi-span questions as a part of its design?
Output: MTMSN BIBREF4
We build a dataset of Twitter accounts based on two lists annotated in previous works. For the non-factual accounts, we rely on a list of 180 Twitter accounts from BIBREF1. On the other hand, for the factual accounts, we use a list with another 32 Twitter accounts from BIBREF19 that are considered trustworthy by independent third parties. 
 Question: How did they obtain the dataset?
Output: For the non-factual accounts, we rely on a list of 180 Twitter accounts from BIBREF1 we use a list with another 32 Twitter accounts from BIBREF19 that are considered trustworthy
We label each review vector with the car it reviews.  
 Question: What are labels in car speak language dataset?
Output: car
Where possible, we tried to restrict the content of each text to the abstract and conclusion of the original work. However, since in many cases these sections were too short, we also considered other parts of the original works such as introduction or discussion sections. To ensure that the extracted text portions are appropriate for the AV task, each original work was preprocessed manually. More precisely, we removed tables, formulas, citations, quotes and sentences that include non-language content such as mathematical constructs or specific names of researchers, systems or algorithms. 
 Question: What is a self-compiled corpus?
Output:  restrict the content of each text to the abstract and conclusion of the original work considered other parts of the original works such as introduction or discussion sections extracted text portions are appropriate for the AV task, each original work was preprocessed manually removed tables, formulas, citations, quotes and sentences that include non-language content such as mathematical constructs or specific names of researchers, systems or algorithms
We used the relation classification dataset of the SemEval 2010 task 8 BIBREF8 . It consists of sentences which have been manually labeled with 19 relations (9 directed relations and one artificial class Other). 8,000 sentences have been distributed as training set and 2,717 sentences served as test set. 
 Question: Which dataset do they train their models on?
Output: relation classification dataset of the SemEval 2010 task 8
Prior to the SAOKE data set, an annotated data set for OIE tasks with 3,200 sentences in 2 domains was released in BIBREF20 to evaluate OIE algorithms, in which the data set was said BIBREF20 “13 times larger than the previous largest annotated Open IE corpus”. 
 Question: What's the size of the previous largest OpenIE dataset?
Output: 3,200 sentences
We use the Yelp Challenge dataset BIBREF2 for our fake review generation.  
 Question: Which dataset do they use a starting point in generating fake reviews?
Output: the Yelp Challenge dataset
This study assumes that a robot does not have any vocabularies in advance but can recognize syllables or phonemes. 
 Question: Does their model start with any prior knowledge of words?
Output: No
Next, we analyze why they do not perform well in this task and with this data:

[noitemsep,leftmargin=*]

Noisy data. A major challenge of this task is the discrepancy between annotations due to subjective judgement. Although this affects also detection using only text, its repercussion is bigger in more complex tasks, such as detection using images or multimodal detection.

Complexity and diversity of multimodal relations. Hate speech multimodal publications employ a lot of background knowledge which makes the relations between visual and textual elements they use very complex and diverse, and therefore difficult to learn by a neural network.

Small set of multimodal examples. Fig. FIGREF5 shows some of the challenging multimodal hate examples that we aimed to detect. But although we have collected a big dataset of $150K$ tweets, the subset of multimodal hate there is still too small to learn the complex multimodal relations needed to identify multimodal hate. 
 Question: What is author's opinion on why current multimodal models cannot outperform models analyzing only text?
Output: Noisy data Complexity and diversity of multimodal relations Small set of multimodal examples
In Table TABREF26 we report the results of our model on the three datasets standardly used for the evaluation of emotion classification, which we have described in Section SECREF3 . 
 Question: What was their performance on emotion detection?
Output: Answer with content missing: (Table 3) Best author's model B-M average micro f-score is 0.409, 0.459, 0.411 on Affective, Fairy Tales and ISEAR datasets respectively.
We are using the dataset of the competition, which includes text from tweets having the aforementioned categories.  
 Question: What dataset is used for this work?
Output: Twitter dataset provided by the organizers
A frequency filter ratio INLINEFORM0 is set to filter out the low-frequency words (rare words) from the lookup table 
 Question: how are rare words defined?
Output: low-frequency words
The Logoscope retrieves newspaper articles from several RSS feeds in French on a daily basis. 
 Question: How often are the newspaper websites crawled daily?
Output: RSS feeds in French on a daily basis
Exemplars aim to provide appropriate context. To better understand the context, we experimented by analysing the questions generated through an exemplar. We observed that indeed a supporting exemplar could identify relevant tags (cows in Figure FIGREF3 ) for generating questions.

We improve use of exemplars by using a triplet network. This network ensures that the joint image-caption embedding for the supporting exemplar are closer to that of the target image-caption and vice-versa. 
 Question: How do the authors define exemplars?
Output: Exemplars aim to provide appropriate context. joint image-caption embedding for the supporting exemplar are closer to that of the target image-caption
o evaluate the quality of the generated sequences regarding both precision and recall, the evaluation metrics include BLEU and ROUGE (1, 2, L) scores with multiple references BIBREF22 . 
 Question: What evaluation metrics are used?
Output: the evaluation metrics include BLEU and ROUGE (1, 2, L) scores
Results in Table TABREF13 show ten probes. Again, we see the performance of baseline ELMo-transformer and mSynC are similar, with mSynC doing slightly worse on 7 out of 9 tasks. 
 Question: For how many probe tasks the shallow-syntax-aware contextual embedding perform better than ELMo’s embedding?
Output: performance of baseline ELMo-transformer and mSynC are similar, with mSynC doing slightly worse on 7 out of 9 tasks
Considering the improvements over the majority baseline achieved by the RNN model for both non-English (on the average 22.76% relative improvement; 15.82% relative improvement on Spanish, 72.71% vs. 84.21%, 30.53% relative improvement on Turkish, 56.97% vs. 74.36%, 37.13% relative improvement on Dutch, 59.63% vs. 81.77%, and 7.55% relative improvement on Russian, 79.60% vs. 85.62%) and English test sets (27.34% relative improvement), we can draw the conclusion that our model is robust to handle multiple languages. 
 Question: which non-english language had the best performance?
Output: Russian
Twitter provides well-documented API, which allows to request any information about Tweets, users and their profiles, with respect to rate limits. There is special type of API, called Streaming API, that provides a real-time stream of tweets. The key difference with regular API is that connection is kept alive as long as possible, and Tweets are sent in real-time to the client. There are three endpoints of Streaming API of our interest: “sample”, “filter” and “firehose”. The first one provides a sample (random subset) of the full Tweet stream. The second one allows to receive Tweets matching some search criteria: matching to one or more search keywords, produced by subset of users, or coming from certain geo location. The last one provides the full set of Tweets, although it is not available by default. In order to get Twitter “firehose” one can contact Twitter, or buy this stream from third-parties.

In our case the simplest approach would be to use “sample” endpoint, but it provides Tweets in all possible languages from all over the World, while we are concerned only about one language (Russian). In order to use this endpoint we implemented filtering based on language. The filter is simple: if Tweet does not contain a substring of 3 or more cyrillic symbols, it is considered non-Russian. Although this approach keeps Tweets in Mongolian, Ukrainian and other slavic languages (because they use cyrillic alphabet), the total amount of false-positives in this case is negligible. To demonstrate this we conducted simple experiment: on a random sample of 200 tweets only 5 were in a language different from Russian. In order not to rely on Twitter language detection, we chose to proceed with this method of language-based filtering.

However, the amount of Tweets received through “sample” endpoint was not satisfying. This is probably because “sample” endpoint always streams the same content to all its clients, and small portion of it comes in Russian language. In order to force mining of Tweets in Russian language, we chose "filter" endpoint, which requires some search query. We constructed heuristic query, containing some auxiliary words, specific to Russian language: conjunctions, pronouns, prepositions. The full list is as follows:

russian я, у, к, в, по, на, ты, мы, до, на, она, он, и, да. 
 Question: Which Twitter corpus was used to train the word vectors?
Output: They collected tweets in Russian language using a heuristic query specific to Russian
The base sentence embedding model is a BiLSTM over all words in the respective sequences with randomly initialised word embeddings, following BIBREF30 . We opt for this strong baseline sentence encoding model, as opposed to engineering sentence embeddings that work particularly well for this dataset, to showcase the dataset. We would expect pre-trained contextual encoding models, e.g. ELMO BIBREF33 , ULMFit BIBREF34 , BERT BIBREF35 , to offer complementary performance gains, as has been shown for a few recent papers BIBREF36 , BIBREF37 . 
 Question: What were the baselines?
Output: a BiLSTM over all words in the respective sequences with randomly initialised word embeddings, following BIBREF30
To reduce variance and boost accuracy, we ensemble 10 CNNs and 10 LSTMs together through soft voting. The models ensembled have different random weight initializations, different number of epochs (from 4 to 20 in total), different set of filter sizes (either INLINEFORM0 , INLINEFORM1 or INLINEFORM2 ) and different embedding pre-training algorithms (either Word2vec or FastText). 
 Question: How many CNNs and LSTMs were ensembled?
Output: 10 CNNs and 10 LSTMs
Defining Faithfulness ::: Assumption 1 (The Model Assumption).
Two models will make the same predictions if and only if they use the same reasoning process. Defining Faithfulness ::: Assumption 2 (The Prediction Assumption).
On similar inputs, the model makes similar decisions if and only if its reasoning is similar. Defining Faithfulness ::: Assumption 3 (The Linearity Assumption).
Certain parts of the input are more important to the model reasoning than others. Moreover, the contributions of different parts of the input are independent from each other. 
 Question: Which are three assumptions in current approaches for defining faithfulness?
Output: Two models will make the same predictions if and only if they use the same reasoning process. On similar inputs, the model makes similar decisions if and only if its reasoning is similar. Certain parts of the input are more important to the model reasoning than others. Moreover, the contributions of different parts of the input are independent from each other.
To account for this, we compute the balanced accuracy, i.e., the average of the three accuracies on each class. 
 Question: Is model explanation output evaluated, what metric was used?
Output: balanced accuracy, i.e., the average of the three accuracies on each class
Noticeably, despite being both based on and integrated into a BERT$_\text{base}$ model, our architecture even outperforms a standalone BERT$_\text{large}$ model by a large margin. 
 Question: What models other than standalone BERT is new model compared to?
Output: Only Bert base and Bert large are compared to proposed approach.
To verify the effectiveness of our proposed model, we conduct multiple experiments on three Chinese Machine Reading Comprehension datasets, namely CMRC-2017 BIBREF17 , People's Daily (PD) and Children Fairy Tales (CFT) BIBREF2  Besides, we also use the Children's Book Test (CBT) dataset BIBREF1 to test the generalization ability in multi-lingual case. 
 Question: which public datasets were used?
Output: CMRC-2017 People's Daily (PD) Children Fairy Tales (CFT)  Children's Book Test (CBT)
We use the annotations from experts for an abstract if it exists otherwise use crowd annotations.  
 Question: How do they match annotators to instances?
Output: Annotations from experts are used if they have already been collected.
The SnapCaptions dataset is composed of 10K user-generated image (snap) and textual caption pairs where named entities in captions are manually labeled by expert human annotators (entity types: PER, LOC, ORG, MISC). 
 Question: How large is their MNER SnapCaptions dataset?
Output: 10K user-generated image (snap) and textual caption pairs
Given the annotated tweets, we wanted to ascertain the distribution of: types of offensive language, genres where it is used, the dialects used, and the gender of users using such language. As the figure shows, sports and politics are most dominant for offensive language including vulgar and hate speech. As for dialect, we looked at MSA and four major dialects, namely Egyptian (EGY), Leventine (LEV), Maghrebi (MGR), and Gulf (GLF). Figure FIGREF14 shows that 71% of vulgar tweets were written in EGY followed by GLF, which accounted for 13% of vulgar tweets. MSA was not used in any of the vulgar tweets. As for offensive tweets in general, EGY and GLF were used in 36% and 35% of the offensive tweets respectively. Unlike the case of vulgar language where MSA was non-existent, 15% of the offensive tweets were in fact written in MSA. For hate speech, GLF and EGY were again dominant and MSA consistuted 21% of the tweets. This is consistent with findings for other languages such as English and Italian where vulgar language was more frequently associated with colloquial language BIBREF24, BIBREF25. Regarding the gender, Figure FIGREF15 shows that the vast majority of offensive tweets, including vulgar and hate speech, were authored by males. Female Twitter users accounted for 14% of offensive tweets in general and 6% and 9% of vulgar and hate speech respectively. Figure FIGREF16 shows a detailed categorization of hate speech types, where the top three include insulting groups based on their political ideology, origin, and sport affiliation. Religious hate speech appeared in only 15% of all hate speech tweets. 
 Question: How did they analyze which topics, dialects and gender are most associated with tweets?
Output: ascertain the distribution of: types of offensive language, genres where it is used, the dialects used, and the gender of users using such language
We call these techniques Recurrence over BERT (RoBERT) and Transformer over BERT (ToBERT). We observed that ToBERT outperforms RoBERT on pre-trained BERT features and fine-tuned BERT features for all our tasks.  
 Question: On top of BERT does the RNN layer work better or the transformer layer?
Output: Transformer over BERT (ToBERT)
In particular, we investigated attention to nouns, verbs, pronouns, subjects, objects, and negation words, and special BERT tokens across the tasks. 
 Question: What handcrafter features-of-interest are used?
Output: nouns verbs pronouns subjects objects negation words special BERT tokens
We approached the first and second challenges by using a Bayesian approach to learn which terms were associated with events, regardless of whether they are standard language, acronyms, or even a made-up word, so long as they match the events of interest. The third and fourth challenges are approached by using word-pairs, where we extract all the pairs of co-occurring words within each tweet. To find the words most associated with events, we search for the words that achieve the highest number of spikes matching the days of events.  
 Question: How are the keywords associated with events such as protests selected?
Output: By using a Bayesian approach  and by using word-pairs, where they extract all the pairs of co-occurring words within each tweet.  They search for the words that achieve the highest number of spikes matching the days of events.
We use standard Precision, Recall and F1 at mention level (Micro) and at the document level (Macro) as measurements. We can see the average linking precision (Micro) of WW is lower than that of TAC2010, and NCEL outperforms all baseline methods in both easy and hard cases. 
 Question: How do they verify generalization ability?
Output: By calculating Macro F1 metric at the document level.
This process gives a total of INLINEFORM0 10700 training examples with roughly 2000 to 3000 examples per class, with each speaker having an average of 185 examples. 
 Question: How many instances does their dataset have?
Output: 10700
Our contribution leverages pre-trained word embeddings (GloVe, trained on wikipedia+gigaword corpus), the DepecheMood affective lexicon, and convolutional neural networks. 
 Question: What is the source of their lexicon?
Output: DepecheMood
As before, the training and test sets contain some 30,000 and 5,000 sentence pairs, respectively 
 Question: How many samples did they generate for the artificial language?
Output: 70,000
For NMT, we used the KyotoNMT system BIBREF16 . The NMT training settings are the same as those of the best systems that participated in WAT 2016. The sizes of the source and target vocabularies, the source and target side embeddings, the hidden states, the attention mechanism hidden states, and the deep softmax output with a 2-maxout layer were set to 32,000, 620, 1000, 1000, and 500, respectively. We used 2-layer LSTMs for both the source and target sides. ADAM was used as the learning algorithm, with a dropout rate of 20% for the inter-layer dropout, and L2 regularization with a weight decay coefficient of 1e-6. The mini batch size was 64, and sentences longer than 80 tokens were discarded. 
 Question: What kinds of neural networks did they use in this paper?
Output: LSTMs
We build and test our MMT models on the Multi30K dataset BIBREF21 . Each image in Multi30K contains one English (EN) description taken from Flickr30K BIBREF22 and human translations into German (DE), French (FR) and Czech BIBREF23 , BIBREF24 , BIBREF25 . The dataset contains 29,000 instances for training, 1,014 for development, and 1,000 for test. We only experiment with German and French, which are languages for which we have in-house expertise for the type of analysis we present. In addition to the official Multi30K test set (test 2016), we also use the test set from the latest WMT evaluation competition, test 2018 BIBREF25 . 
 Question: Do they report results only on English dataset?
Output: No
To evaluate the effectiveness of the proposed framework for the ATE task, we conduct experiments over four benchmark datasets from the SemEval ABSA challenge BIBREF1 , BIBREF18 , BIBREF12 . Table TABREF24 shows their statistics. INLINEFORM0 (SemEval 2014) contains reviews of the laptop domain and those of INLINEFORM1 (SemEval 2014), INLINEFORM2 (SemEval 2015) and INLINEFORM3 (SemEval 2016) are for the restaurant domain. In these datasets, aspect terms have been labeled by the task organizer. 
 Question: Which dataset(s) do they use to train the model?
Output: INLINEFORM0 (SemEval 2014) contains reviews of the laptop domain and those of INLINEFORM1 (SemEval 2014), INLINEFORM2 (SemEval 2015) and INLINEFORM3 (SemEval 2016) are for the restaurant domain.
Crowd workers were asked to mark whether a triple was correct, namely, did the triple reflect the consequence of the sentence. 
 Question: What is the role of crowd-sourcing?
Output: Crowd workers were asked to mark whether a triple was correct, namely, did the triple reflect the consequence of the sentence.
Category II and III errors are harder to avoid and could be improved by applying reasoning BIBREF36 or irony detection methods BIBREF37. 
 Question: What recommendations are made to improve the performance in future?
Output: applying reasoning BIBREF36 or irony detection methods BIBREF37
Only the information concerned with the dictionary definitions are used there, discarding the polarity scores. However, when we utilise the supervised score (+1 or -1), words of opposite polarities (e.g. “happy" and “unhappy") get far away from each other as they are translated across coordinate regions. 
 Question: How are the supervised scores of the words calculated?
Output: (+1 or -1), words of opposite polarities (e.g. “happy" and “unhappy") get far away from each other
In the sentiment classification task, we tried our model on the following public datasets. MR Movie reviews with one sentence each. IMDB Movie reviews from IMDB website, where each movie review is labeled with binary classes, either positive or negative BIBREF19. SUBJ$^1$ Movie review labeled with subjective or objective BIBREF20. We also tested our CRU model in the cloze-style reading comprehension task. We carried out experiments on the public datasets: CBT NE/CN BIBREF25. 
 Question: What datasets are used for testing sentiment classification and reading comprehension?
Output: CBT NE/CN MR Movie reviews IMDB Movie reviews SUBJ
First, we can observe that the final model “Ours with Mask and Ordered Triplets” outperforms the Baseline and Ablation models on all metrics in previously seen environments. The difference in performance is particularly evident for the Exact Match and Goal Match metrics, with our model increasing accuracy by 35% and 25% in comparison to the Baseline and Ablation models, respectively. These results suggest that providing the behavioral navigation graph to the model and allowing it to process this information as a knowledge base in an end-to-end fashion is beneficial. Lastly, it is worth noting that our proposed model (last row of Table TABREF28 ) outperforms all other models in previously seen environments. In particular, we obtain over INLINEFORM0 increase in EM and GM between our model and the next best two models. 
 Question: By how much did their model outperform the baseline?
Output: increasing accuracy by 35% and 25% in comparison to the Baseline and Ablation models, respectively over INLINEFORM0 increase in EM and GM between our model and the next best two models
Initially, we experimented with a graph-based community detection algorithm that maximizes cluster modularity BIBREF20 , but we found no simple way to constrain this method to produce a specific number of equally-sized clusters. The brute force approach of enumerating all possible cluster assignments is intractable given the large search space ( INLINEFORM0 possible assignments). We devise a simple clustering algorithm to approximate this process. First, we initialize with random cluster assignments and define “cluster strength” to be the relative difference between “intra-group” Euclidean distance and “inter-group” Euclidean distance. Then, we iteratively propose random exchanges of memberships, only accepting these proposals when the cluster strength increases, until convergence. To evaluate the quality of the computationally-derived clusters against those of Calvino, we measure cluster purity BIBREF21 : given a set of predicted clusters INLINEFORM1 and ground-truth clusters INLINEFORM2 that both partition a set of INLINEFORM3 data points, INLINEFORM4 
 Question: Which clustering method do they use to cluster city description embeddings?
Output:  We devise a simple clustering algorithm to approximate this process. First, we initialize with random cluster assignments and define “cluster strength” to be the relative difference between “intra-group” Euclidean distance and “inter-group” Euclidean distance. Then, we iteratively propose random exchanges of memberships, only accepting these proposals when the cluster strength increases, until convergence.
At the moment defender can draw on methods from image area to text for improving the robustness of DNNs, e.g. adversarial training BIBREF107 , adding extra layer BIBREF113 , optimizing cross-entropy function BIBREF114 , BIBREF115 or weakening the transferability of adversarial examples. 
 Question: Which strategies show the most promise in deterring these attacks?
Output: At the moment defender can draw on methods from image area to text for improving the robustness of DNNs, e.g. adversarial training BIBREF107 , adding extra layer BIBREF113 , optimizing cross-entropy function BIBREF114 , BIBREF115 or weakening the transferability of adversarial examples.
As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in BIBREF3 . The perplexity scores are also better. On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time. 
 Question: What improvement does the MOE model make over the SOTA on machine translation?
Output: 1.34 and 1.12 BLEU score on top of the strong baselines in BIBREF3 perplexity scores are also better On the Google Production dataset, our model achieved 1.01 higher test BLEU score
the facts that are relevant for answering a particular question) are labeled during training. 
 Question: What does supporting fact supervision mean?
Output:  the facts that are relevant for answering a particular question) are labeled during training.
A base model, INLINEFORM0 , is trained on 190 hours ( INLINEFORM1 100K instances) of transcribed speech data. Then, it selects a subset of a 1,700-hour ( INLINEFORM2 1.1M instances) unlabeled dataset. We query labels for the selected subset and incorporate them into training. Learning rates are tuned on a small validation set of 2048 instances. The trained model is then tested on a 156-hour ( INLINEFORM3 100K instances) test set and we report CTC loss, Character Error Rate (CER) and Word Error Rate (WER). 
 Question: Which dataset do they use?
Output: 190 hours ( INLINEFORM1 100K instances)
It includes a total 708 hours of French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh) speeches, with French and German ones having the largest durations among existing public corpora. 
 Question: Which languages are part of the corpus?
Output: French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh)
As indicated in its name, Recurrent Deep Stacking Network stacks and concatenates the outputs of previous frames into the input features of the current frame. 
 Question: What does recurrent deep stacking network do?
Output: Stacks and joins outputs of previous frames with inputs of the current frame
For sentence encoding models, we chose a simple one-layer bidirectional LSTM with max pooling (BiLSTM-max) with the hidden size of 600D per direction, used e.g. in InferSent BIBREF17 , and HBMP BIBREF18 . For the other models, we have chosen ESIM BIBREF19 , which includes cross-sentence attention, and KIM BIBREF2 , which has cross-sentence attention and utilizes external knowledge. We also selected two model involving a pre-trained language model, namely ESIM + ELMo BIBREF20 and BERT BIBREF0 . 
 Question: Which models were compared?
Output: BiLSTM-max, HBMP, ESIM, KIM, ESIM + ELMo, and BERT
Guidelines for Evaluating Faithfulness
We propose the following guidelines for evaluating the faithfulness of explanations. These guidelines address common pitfalls and sub-optimal practices we observed in the literature.

Guidelines for Evaluating Faithfulness ::: Be explicit in what you evaluate.
Conflating plausability and faithfulness is harmful. You should be explicit on which one of them you evaluate, and use suitable methodologies for each one. Of course, the same applies when designing interpretation techniques—be clear about which properties are being prioritized.

Guidelines for Evaluating Faithfulness ::: Faithfulness evaluation should not involve human-judgement on the quality of interpretation.
We note that: (1) humans cannot judge if an interpretation is faithful or not: if they understood the model, interpretation would be unnecessary; (2) for similar reasons, we cannot obtain supervision for this problem, either. Therefore, human judgement should not be involved in evaluation for faithfulness, as human judgement measures plausability.

Guidelines for Evaluating Faithfulness ::: Faithfulness evaluation should not involve human-provided gold labels.
We should be able to interpret incorrect model predictions, just the same as correct ones. Evaluation methods that rely on gold labels are influenced by human priors on what should the model do, and again push the evaluation in the direction of plausability.

Guidelines for Evaluating Faithfulness ::: Do not trust “inherent interpretability” claims.
Inherent interpretability is a claim until proven otherwise. Explanations provided by “inherently interpretable” models must be held to the same standards as post-hoc interpretation methods, and be evaluated for faithfulness using the same set of evaluation techniques.

Guidelines for Evaluating Faithfulness ::: Faithfulness evaluation of IUI systems should not rely on user performance.
End-task user performance in HCI settings is merely indicative of correlation between plausibility and model performance, however small this correlation is. While important to evaluate the utility of the interpretations for some use-cases, it is unrelated to faithfulness. 
 Question: Which are key points in guidelines for faithfulness evaluation?
Output: Be explicit in what you evaluate. Faithfulness evaluation should not involve human-judgement on the quality of interpretation. Faithfulness evaluation should not involve human-provided gold labels. Do not trust “inherent interpretability” claims. Faithfulness evaluation of IUI systems should not rely on user performance.
Figure FIGREF29 shows the $\text{MPAs}$ of the proposed DP-LSTM and vanilla LSTM for comparison.  
 Question: Is the model compared against a linear regression baseline?
Output: No
For the shared task, a balanced dataset of 2,396 ironic and 2,396 non-ironic tweets is provided. The ironic corpus was constructed by collecting self-annotated tweets with the hashtags #irony, #sarcasm and #not. The tweets were then cleaned and manually checked and labeled, using a fine-grained annotation scheme BIBREF3 .  
 Question: What is the size of the dataset?
Output: a balanced dataset of 2,396 ironic and 2,396 non-ironic tweets is provided
We recruited 176 AMT workers to participate in our conceptualization task. 
 Question: What crowdsourcing platform was used?
Output: AMT
We select a large-scale Chinese dataset BIBREF0 with millions of real comments and a human-annotated test set to evaluate our model. The dataset is collected from Tencent News, which is one of the most popular Chinese websites for news and opinion articles. The dataset consists of 198,112 news articles. Each piece of news contains a title, the content of the article, and a list of the users' comments. 
 Question: What news comment dataset was used?
Output: Chinese dataset BIBREF0
We trained word embeddings using either GloVe BIBREF11 or SGNS BIBREF12 on a small or a large corpus. 
 Question: What types of word representations are they evaluating?
Output: GloVE; SGNS
The bossa-nova and jovem-guarda genres, which have few instances in the dataset, are among the most difficult ones to classify using the model. 
 Question: what genre was the most difficult to classify?
Output:  bossa-nova and jovem-guarda genres
For each user, we calculate the proportion of tweets scored positively by each LIWC category.  
 Question: How is LIWC incorporated into this system?
Output:  For each user, we calculate the proportion of tweets scored positively by each LIWC category.
For each object, $O_i$ in the input Open IE tuple $(S; P; O_1; O_2 \ldots )$ , we add a triple $(S; P; O_i)$ to this table. 
 Question: Are the OpenIE extractions all triples?
Output: No
We use the additional INLINEFORM0 training articles labeled by publisher as an unsupervised data set to further train the BERT model. We first investigate the impact of pre-training on BERT-BASE's performance.  On the same computer, fine tuning the model on the small training set took only about 35 minutes for sequence length 100.  
 Question: How are the two different models trained?
Output: They pre-train the models using 600000 articles as an unsupervised dataset and then fine-tune the models on small training set.
In our method, we take an image as input and generate a natural question as output. 
 Question: What is the input to the differential network?
Output: image
We compare with three baseline methods. $({1})$ SC-LSTM BIBREF3 is a canonical model and a strong baseline that uses an additional dialog act vector and a reading gate to guide the utterance generation. $({2})$ GPT-2 BIBREF6 is used to directly fine-tune on the domain-specific labels, without pre-training on the large-scale corpus of (dialog act, response) pairs. $({3})$ HDSA BIBREF7 is a state-of-the-art model on MultiWOZ. It leverages dialog act structures to enable transfer in the multi-domain setting, showing superior performance than SC-LSTM. 
 Question: What existing methods is SC-GPT compared to?
Output: $({1})$ SC-LSTM BIBREF3 $({2})$ GPT-2 BIBREF6  $({3})$ HDSA BIBREF7
Our prediction task is thus a straightforward binary classification task at the word level. We develop the following five groups of features to capture properties of how a word is used in the explanandum (see Table TABREF18 for the full list):

[itemsep=0pt,leftmargin=*,topsep=0pt]

Non-contextual properties of a word. These features are derived directly from the word and capture the general tendency of a word being echoed in explanations.

Word usage in an OP or PC (two groups). These features capture how a word is used in an OP or PC. As a result, for each feature, we have two values for the OP and PC respectively.

How a word connects an OP and PC. These features look at the difference between word usage in the OP and PC. We expect this group to be the most important in our task.

General OP/PC properties. These features capture the general properties of a conversation. They can be used to characterize the background distribution of echoing.

Table TABREF18 further shows the intuition for including each feature, and condensed $t$-test results after Bonferroni correction. Specifically, we test whether the words that were echoed in explanations have different feature values from those that were not echoed. In addition to considering all words, we also separately consider stopwords and content words in light of Figure FIGREF8. Here, we highlight a few observations:

[itemsep=0pt,leftmargin=*,topsep=0pt] 
 Question: What are their proposed features?
Output: Non-contextual properties of a word Word usage in an OP or PC (two groups) How a word connects an OP and PC. General OP/PC properties
In this paper, I introduce an interpretable word embedding model, and an associated topic model, which are designed to work well when trained on a small to medium-sized corpus of interest. I tested the performance of the representations as features for document categorization and regression tasks. The results are given in Table TABREF26 . For document categorization, I used three standard benchmark datasets: 20 Newsgroups (19,997 newsgroup posts), Reuters-150 newswire articles (15,500 articles and 150 classes), and Ohsumed medical abstracts on 23 cardiovascular diseases (20,000 articles). I I also analyzed the regression task of predicting the year of a state of the Union address based on its text information.  
 Question: What supervised learning tasks are attempted with these representations?
Output: document categorization regression tasks
As is common in the literature BIBREF4 , BIBREF8 , we use 300-dimensional vectors for our embeddings and all word vectors are normalized to unit length prior to evaluation. 
 Question: What dimensions of word embeddings do they produce using factorization?
Output: 300-dimensional vectors
The second algorithm incorporates recent ideas of data selection in machine translation BIBREF7 . We develop a novel re-weighting method to resolve these problems, using ideas inspired by data selection in machine translation BIBREF26 , BIBREF7 . 
 Question: What is the data selection paper in machine translation
Output: BIBREF7 BIBREF26
In multitask learning, a set of related tasks are learned (e.g., emotional activation), along with a primary task (e.g., emotional valence); both tasks share parts of the network topology and are hence jointly trained, as depicted in Figure FIGREF4 . 
 Question: What are the tasks in the multitask learning setup?
Output: set of related tasks are learned (e.g., emotional activation) primary task (e.g., emotional valence)
The corpus used for sentiment analysis is the IMDb dataset of movie reviews by BIBREF11 while that for NER is Groningen Meaning Bank (GMB) by BIBREF12, containing 47,959 sentence samples. 
 Question: What sentiment analysis dataset is used?
Output: IMDb dataset of movie reviews
Sentiment Classification
We first conduct a multi-task experiment on sentiment classification.

We use 16 different datasets from several popular review corpora used in BIBREF20 . These datasets consist of 14 product review datasets and two movie review datasets.

All the datasets in each task are partitioned randomly into training set, development set and testing set with the proportion of 70%, 10% and 20% respectively. Transferability of Shared Sentence Representation
With attention mechanism, the shared sentence encoder in our proposed models can generate more generic task-invariant representations, which can be considered as off-the-shelf knowledge and then be used for unseen new tasks. Introducing Sequence Labeling as Auxiliary Task
A good sentence representation should include its linguistic information. Therefore, we incorporate sequence labeling task (such as POS Tagging and Chunking) as an auxiliary task into the multi-task learning framework, which is trained jointly with the primary tasks (the above 16 tasks of sentiment classification). 
 Question: What tasks did they experiment with?
Output: Sentiment Classification Transferability of Shared Sentence Representation Introducing Sequence Labeling as Auxiliary Task