niryuu/sni-each-converted huggingface dataset:
Here is a list of architectures for which reference implementations and pretrained weights are currently provided in Transformers. These models fall into two main categories: generative models (GPT, GPT-2, Transformer-XL, XLNet, XLM) and models for language understanding (Bert, DistilBert, RoBERTa, XLM).

BERT (BIBREF13) is a bi-directional Transformer-based encoder pretrained with a linear combination of masked language modeling and next sentence prediction objectives.

RoBERTa (BIBREF5) is a replication study of BERT which showed that carefully tuning hyper-parameters and training data size lead to significantly improved results on language understanding.

DistilBERT (BIBREF32) is a smaller, faster, cheaper and lighter version BERT pretrained with knowledge distillation.

GPT (BIBREF34) and GPT2 (BIBREF9) are two large auto-regressive language models pretrained with language modeling. GPT2 showcased zero-shot task transfer capabilities on various tasks such as machine translation or reading comprehension.

Transformer-XL (BIBREF35) introduces architectural modifications enabling Transformers to learn dependency beyond a fixed length without disrupting temporal coherence via segment-level recurrence and relative positional encoding schemes.

XLNet (BIBREF4) builds upon Transformer-XL and proposes an auto-regressive pretraining scheme combining BERT's bi-directional context flow with auto-regressive language modeling by maximizing the expected likelihood over permutations of the word sequence.

XLM (BIBREF8) shows the effectiveness of pretrained representations for cross-lingual language modeling (both on monolingual data and parallel data) and cross-lingual language understanding.

We systematically release the model with the corresponding pretraining heads (language modeling, next sentence prediction for BERT) for adaptation using the pretraining objectives. Some models fine-tuned on downstream tasks such as SQuAD1.1 are also available. Overall, more than 30 pretrained weights are provided through the library including more than 10 models pretrained in languages other than English. Some of these non-English pretrained models are multi-lingual models (with two of them being trained on more than 100 languages) . 
 Question: What state-of-the-art general-purpose pretrained models are made available under the unified API? 
Output: BERT RoBERTa DistilBERT GPT GPT2 Transformer-XL XLNet XLM
In this work we train bilingual UWS models using the endangered language Mboshi as target and different well-resourced languages as aligned information. Results show that similar languages rank better in terms of segmentation performance, and that by combining the information learned by different models, segmentation is further improved. This might be due to the different language-dependent structures that are captured by using more than one language.  
 Question: How does the well-resourced language impact the quality of the output?
Output: Results show that similar languages rank better in terms of segmentation performance, and that by combining the information learned by different models, segmentation is further improved.
To discover topics from the collected tweets, we used a topic modeling approach that fuzzy clusters the semantically related words such as assigning “diabetes", “cancer", and “influenza" into a topic that has an overall “disease" theme BIBREF44 , BIBREF45 . Among topic models, Latent Dirichlet Allocation (LDA) BIBREF49 is the most popular effective model BIBREF50 , BIBREF19 as studies have shown that LDA is an effective computational linguistics model for discovering topics in a corpus BIBREF51 , BIBREF52 . We used the Mallet implementation of LDA BIBREF49 , BIBREF56 with its default settings to explore opinions in the tweets. 
 Question: How were topics of interest about DDEO identified?
Output: using topic modeling model Latent Dirichlet Allocation (LDA)
We present and evaluate a model that we have developed for this, called QuaSP+Zero, that modifies the QuaSP+ parser as follows: During decoding, at points where the parser is selecting which property to include in the LF (e.g., Figure FIGREF31 ), it does not just consider the question tokens, but also the relationship between those tokens and the properties INLINEFORM0 used in the qualitative model. This approach follows the entity-linking approach used by BIBREF11 Krishnamurthy2017NeuralSP, where the similarity between question tokens and (words associated with) entities - called the entity linking score - help decide which entities to include in the LF during parsing. 
 Question: How does the QuaSP+Zero model work?
Output: does not just consider the question tokens, but also the relationship between those tokens and the properties
Some authors use pretrained embeddings (especially when their data set is too small to train their own embeddings) or try to modify these embeddings and adjust to their set. But the biggest drawback of these approaches is that the corpus for training embeddings can be not related to the specific task where embeddings are utilized. A lot of medical concepts are not contained in well-known embeddings bases. Furthermore, the similarity of words may vary in different contexts. Then, we compute embeddings of concepts (by GloVe) for interview descriptions and for examination descriptions separately. 
 Question: Do they fine-tune the used word embeddings on their medical texts?
Output: No
Our work is further connected to the topic of interpretability and transparency of machine learning models BIBREF11, BIBREF35, BIBREF12, for which humans are increasingly involved, for instance for post-hoc evaluations of the model's interpretability.  
 Question: How are the interpretability merits of the approach demonstrated?
Output: By involving humans for post-hoc evaluation of model's interpretability
The final code-mixed tweets were forwarded to a group of three annotators who were university students and fluent in both English and Hindi. 
 Question: How many annotators tagged each text?
Output: three
In order to further analysis the embeddings learned by our method, we use t-SNE tool BIBREF27 to visualize the learned embedding. 
 Question: What further analysis is done?
Output: we use t-SNE tool BIBREF27 to visualize the learned embedding
The data set we evaluate on in this work is WMT English-French NewsTest2014, which has 380M words of parallel training data and a 3003 sentence test set. The NewsTest2013 set is used for validation. 
 Question: Do they only test on one dataset?
Output: Yes
We employ precision, recall, F1 and accuracy for evaluation metrics. 
 Question: what evaluation metrics are reported?
Output: precision, recall, F1 and accuracy
It is is now clear that some configurations used in some architectures (e.g., additional RNN layers) are actually irrelevant and can be removed altogether without harming accuracy. 
 Question: What are least important components identified in the the training of VQA models?
Output:  some configurations used in some architectures (e.g., additional RNN layers) are actually irrelevant
We start by crawling the content of a few notable debating websites: idebate.com, debatewise.org, procon.org. 
 Question: What debate websites did they look at?
Output: idebate.com debatewise.org procon.org
The baseline system for the SLC task is a very simple logistic regression classifier with default parameters, where we represent the input instances with a single feature: the length of the sentence.  The baseline for the FLC task generates spans and selects one of the 18 techniques randomly.  
 Question: What was the baseline for this task?
Output: The baseline system for the SLC task is a very simple logistic regression classifier with default parameters. The baseline for the FLC task generates spans and selects one of the 18 techniques randomly.
We use the Ultrax Typically Developing dataset (UXTD) from the publicly available UltraSuite repository BIBREF19 . This dataset contains synchronized acoustic and ultrasound data from 58 typically developing children, aged 5-12 years old (31 female, 27 male).  
 Question: How many speakers do they have in the dataset?
Output: 58
The second Turkish dataset is the Twitter corpus which is formed of tweets about Turkish mobile network operators. Those tweets are mostly much noisier and shorter compared to the reviews in the movie corpus. In total, there are 1,716 tweets. 973 of them are negative and 743 of them are positive. These tweets are manually annotated by two humans, where the labels are either positive or negative. 
 Question: What details are given about the Twitter dataset?
Output: Those tweets are mostly much noisier and shorter compared to the reviews in the movie corpus. In total, there are 1,716 tweets. 973 of them are negative and 743 of them are positive.
Information Gathering: At step $t$ of the information gathering phase, the agent can issue one of the following four actions to interact with the paragraph $p$, where $p$ consists of $n$ sentences and where the current observation corresponds to sentence $s_k,~1 \le k \le n$:

previous: jump to $ \small {\left\lbrace \begin{array}{ll} s_n & \text{if $k = 1$,}\\ s_{k-1} & \text{otherwise;} \end{array}\right.} $

next: jump to $ \small {\left\lbrace \begin{array}{ll} s_1 & \text{if $k = n$,}\\ s_{k+1} & \text{otherwise;} \end{array}\right.} $

Ctrl+F $<$query$>$: jump to the sentence that contains the next occurrence of “query”;

stop: terminate information gathering phase. 
 Question: What commands does their setup provide to models seeking information?
Output: previous next Ctrl+F $<$query$>$ stop
Our street view environment was integrated into ParlAI BIBREF6 and used to collect a large-scale dataset on Mechanical Turk involving human perception, action and communication. 
 Question: What data did they use?
Output:  dataset on Mechanical Turk involving human perception, action and communication
The CRF module achieved the best result on the Thai sentence segmentation task BIBREF8 ; therefore, we adopt the Bi-LSTM-CRF model as our baseline.  
 Question: Which deep learning architecture do they use for sentence segmentation?
Output: Bi-LSTM-CRF
To study how the multimodal context can boost the performance compared to an unimodal context we evaluate different models: a Feature Concatenation Model (FCM), a Spatial Concatenation Model (SCM) and a Textual Kernels Model (TKM). 
 Question: What different models for multimodal detection were proposed?
Output: Feature Concatenation Model (FCM) Spatial Concatenation Model (SCM) Textual Kernels Model (TKM)
Our goal is to supply an NMT system with knowledge regarding the speaker and interlocutor of first-person sentences, in order to produce the desired target-side morphology when the information is not available in the source sentence. The approach we take in the current work is that of black-box injection, in which we attempt to inject knowledge to the input in order to influence the output of a trained NMT system, without having access to its internals or its training procedure as proposed by vanmassenhove-hardmeier-way:2018:EMNLP. To verify this, we experiment with translating the sentences with the following variations: No Prefix—The baseline translation as returned by the GMT system. “He said:”—Signaling a male speaker. We expect to further skew the system towards masculine forms. “She said:”—Signaling a female speaker and unknown audience. As this matches the actual speaker's gender, we expect an improvement in translation of first-person pronouns and verbs with first-person pronouns as subjects. “I said to them:”—Signaling an unknown speaker and plural audience. “He said to them:”—Masculine speaker and plural audience. “She said to them:”—Female speaker and plural audience—the complete, correct condition. We expect the best translation accuracy on this setup. “He/she said to him/her”—Here we set an (incorrect) singular gender-marked audience, to investigate our ability to control the audience morphology. 
 Question: What are the components of the black-box context injection system?
Output: supply an NMT system with knowledge regarding the speaker and interlocutor of first-person sentences
We recruited 176 AMT workers to participate in our conceptualization task. 
 Question: What was the task given to workers?
Output: conceptualization task
We compare the performance of translation approaches based on four metrics:

[align=left,leftmargin=0em,labelsep=0.4em,font=]

As in BIBREF20 , EM is 1 if a predicted plan matches exactly the ground truth; otherwise it is 0.

The harmonic average of the precision and recall over all the test set BIBREF26 .

The minimum number of insertions, deletions or swap operations required to transform a predicted sequence of behaviors into the ground truth sequence BIBREF27 .

GM is 1 if a predicted plan reaches the ground truth destination (even if the full sequence of behaviors does not match exactly the ground truth). Otherwise, GM is 0. 
 Question: What evaluation metrics are used?
Output: exact match, f1 score, edit distance and goal match
This paper experimented with two different models and compared them against each other.  
 Question: what was the baseline?
Output: There is no baseline.
Even if the optional lemmatization step is applied, one can still aim at further reducing the graph complexity by merging similar vertices. This step is called meta vertex construction. The meta-vertex construction step works as follows. Let INLINEFORM0 represent the set of vertices, as defined above. A meta vertex INLINEFORM1 is comprised of a set of vertices that are elements of INLINEFORM2 , i.e. INLINEFORM3 . Let INLINEFORM4 denote the INLINEFORM5 -th meta vertex. We construct a given INLINEFORM6 so that for each INLINEFORM7 , INLINEFORM8 's initial edges (prior to merging it into a meta vertex) are rewired to the newly added INLINEFORM9 . Note that such edges connect to vertices which are not a part of INLINEFORM10 . Thus, both the number of vertices, as well as edges get reduced substantially. This feature is implemented via the following procedure: Meta vertex candidate identification. Edit distance and word lengths distance are used to determine whether two words should be merged into a meta vertex (only if length distance threshold is met, the more expensive edit distance is computed). The meta vertex creation. As common identifiers, we use the stemmed version of the original vertices and if there is more than one resulting stem, we select the vertex from the identified candidates that has the highest centrality value in the graph and its stemmed version is introduced as a novel vertex (meta vertex). 
 Question: How are meta vertices computed?
Output: Meta vertex candidate identification. Edit distance and word lengths distance are used to determine whether two words should be merged into a meta vertex (only if length distance threshold is met, the more expensive edit distance is computed). The meta vertex creation. As common identifiers, we use the stemmed version of the original vertices and if there is more than one resulting stem, we select the vertex from the identified candidates that has the highest centrality value in the graph and its stemmed version is introduced as a novel vertex (meta vertex).
We sparsify this graph to the union of a minimum spanning tree and a k-Nearest Neighbors (MST-kNN) graph BIBREF14, a geometric construction that removes less important similarities but preserves global connectivity for the graph and, hence, for the dataset. The MST-kNN graph is then analysed with Markov Stability BIBREF15, BIBREF16, BIBREF17, BIBREF18, a multi-resolution graph partitioning method that identifies relevant subgraphs (i.e., clusters of documents) at different levels of granularity. 
 Question: What cluster identification method is used in this paper?
Output: A combination of Minimum spanning trees, K-Nearest Neighbors and Markov Stability BIBREF15, BIBREF16, BIBREF17, BIBREF18
We use two datasets for experiments (Table TABREF29 ): (a) STAN INLINEFORM0 , created by BIBREF10 BansalBV15, which consists of 1,108 unique English hashtags from 1,268 randomly selected tweets in the Stanford Sentiment Analysis Dataset BIBREF36 along with their crowdsourced segmentations and our additional corrections; and (b) STAN INLINEFORM1 , our new expert curated dataset, which includes all 12,594 unique English hashtags and their associated tweets from the same Stanford dataset. 
 Question: Do the hashtag and SemEval datasets contain only English data?
Output: Yes
We evaluated baselines and our model using accuracy as the metric on the ROCStories dataset, and summarized these results in Table 2 . 
 Question: Which metrics are they evaluating with?
Output: accuracy
We use GraphParser without paraphrases as our baseline. This gives an idea about the impact of using paraphrases We compare our paraphrasing models with monolingual machine translation based model for paraphrase generation BIBREF24 , BIBREF36 . In particular, we use Moses BIBREF37 to train a monolingual phrase-based MT system on the Paralex corpus. Finally, we use Moses decoder to generate 10-best distinct paraphrases for the test questions. 
 Question: What are the baselines?
Output: GraphParser without paraphrases monolingual machine translation based model for paraphrase generation
The annotation process was a trial-and-error, with cycles composed of annotation, discussing confusing entities, updating the annotation guide schematic and going through the corpus section again to correct entities following guide changes. 
 Question: Did they experiment with the corpus?
Output: Yes
INSIGHT 1: Political handles are more likely to engage in profile changing behavior as compared to their followers.  We analyze the trends in Figure FIGREF12 and find that the political handles do not change their usernames at all. This is in contrast to the trend in Figure FIGREF15 where we see that there are a lot of handles that change their usernames multiple times.  INSIGHT 3: Political handles tend to make new changes related to previous attribute values. However, the followers make comparatively less related changes to previous attribute values. 
 Question: How do profile changes vary for influential leads and their followers over the social movement?
Output: Influential leaders are more likely to change their profile attributes than their followers; the leaders do not change their usernames, while their followers change their usernames a lot; the leaders  tend to make new changes related to previous attribute values, while the followers make comparatively less related changes to previous attribute values.
Moreover, this work also handles the cases of singular/plural, subject pronoun/object pronoun, etc. For instance, the pronoun “he" is used for the subject as “he" but is used for the object as “him". 
 Question: What else is tried to be solved other than 12 tenses, model verbs and negative form?
Output: cases of singular/plural, subject pronoun/object pronoun, etc.
For all the tasks in our experimental study, we use 36 millions English tweets collected between August and September 2017.  
 Question: Do they report results only on English datasets?
Output: Yes
$LS \cup KLD \cup CONN$ and $KLD \cup LS \cup LS_{inter}$ are the best systems with the highest recall and F1-score respectively. 
 Question: What was previous state-of-the-art approach?
Output: TextCNN, TextRNN, SASE, DPCNN, and BERT $LS \cup KLD \cup CONN$ and $KLD \cup LS \cup LS_{inter}$ are the best systems with the highest recall and F1-score respectively
To systematically evaluate the importance of the clinical sentiment values extracted from the free text in EHRs, we first build a baseline model using the structured features, which are similar to prior studies on readmission risk prediction BIBREF6. 
 Question: Do they compare to previous models?
Output: Yes
They perform data augmentation by back translation between English and Chinese, which seems to be one of the distinguishing factors resulting in a much higher accuracy. 
 Question: What data augmentation techniques are used?
Output: back translation between English and Chinese
We show the statistics of the dataset we use in tab:statistics, and the construction procedures will be introduced in this section. In Wikidata BIBREF8 , facts can be described as (Head item/property, Property, Tail item/property).  We first choose the most common 202 relations and 120000 entities from Wikidata as our initial data. ReVerb BIBREF9 is a program that automatically identifies and extracts binary relationships from English sentences. We use the extractions from running ReVerb on Wikipedia. FB15K BIBREF3 is a subset of freebase. TACRED BIBREF10 is a large supervised relation extraction dataset obtained via crowdsourcing. We directly use these two dataset, no extra processing steps were applied. 
 Question: Which knowledge bases do they use?
Output: Wikidata ReVerb FB15K TACRED
Machine translation finds use in cheminformatics in “translation" from one language (e.g. reactants) to another (e.g. products). The variational Auto-encoder (VAE) is another widely adopted text generation architecture BIBREF101. Generative Adversarial Network (GAN) models generate novel molecules by using two components: the generator network generates novel molecules, and the discriminator network aims to distinguish between the generated molecules and real molecules BIBREF107. 
 Question: Are this models usually semi/supervised or unsupervised?
Output: Both supervised and unsupervised, depending on the task that needs to be solved.
In addition, we propose a new standard experimental protocol for the IBM-UB-1 dataset BIBREF25 (Sec. SECREF50 ) to enable easier comparison between approaches in the future. The IAM-OnDB dataset BIBREF42 is probably the most used evaluation dataset for online handwriting recognition. It consists of 298 523 characters in 86 272 word instances from a dictionary of 11 059 words written by 221 writers. We use the standard IAM-OnDB dataset separation: one training set, two validations sets and a test set containing 5 363, 1 438, 1 518 and 3 859 written lines, respectively. We tune the decoder weights using the validation set with 1 438 items and report error rates on the test set. We provide an evaluation of our production system trained on our in-house datasets applied to a number of publicly available benchmark datasets from the literature. Note that for all experiments presented in this section we evaluate our current live system without any tuning specifec to the tasks at hand.

The ICDAR-2013 Competition for Online Handwriting Chinese Character Recognition BIBREF45 introduced a dataset for classifying the most common Chinese characters. We report the error rates in comparison to published results from the competition and more recent work done by others in Table TABREF56 . In the ICFHR2018 Competition on Vietnamese Online Handwritten Text Recognition using VNOnDB BIBREF50 , our production system was evaluated against other systems. The system used in the competition is the one reported and described in this paper. Due to licensing restrictions we were unable to do any experiments on the competition training data, or specific tuning for the competition, which was not the case for the other systems mentioned here. 
 Question: What datasets did they use?
Output: IBM-UB-1 dataset BIBREF25 IAM-OnDB dataset BIBREF42 The ICDAR-2013 Competition for Online Handwriting Chinese Character Recognition BIBREF45 ICFHR2018 Competition on Vietnamese Online Handwritten Text Recognition using VNOnDB BIBREF50
After excluding all the words that are not used by at least three separate users in our training set, we build our AllWords model by counting the frequencies of all the remaining words and training a multinomial Naive Bayes classifier.  
 Question: What model did they use for their system?
Output: AllWords model by counting the frequencies of all the remaining words and training a multinomial Naive Bayes classifier
N-grams, POS, Hierarchical features: A baseline bag-of-words model incorporating both tagged and untagged unigrams and bigams.  CNN: Kim BIBREF28 demonstrated near state-of-the-art performance on a number of sentence classification tasks (including TREC question classification) by using pre-trained word embeddings BIBREF40 as feature extractors in a CNN model. 
 Question: What previous methods is their model compared to?
Output: bag-of-words model CNN
We set up the baselines and proposed models as follows: LSTM with text embedding: CNNs and LSTMs are widely used to encode textual contents for sentiment analysis in BIBREF45 , BIBREF46 and many online tutorials. Here we select the standard LSTM with pre-trained word embedding as input, and add one fully-connected layer with sigmoid activation top of the LSTM encoder (same as all other models), denoted as T-LSTM. LSTM with emoji embedding: We consider the emoji as one special word and input both pre-trained text and emoji embeddings into the same LSTM network, namely E-LSTM. Similarly, we concatenate the pre-trained bi-sense emoji embedding as one special word to feed into the LSTM network. This model is called BiE-LSTM. Attention-based LSTM with emojis:We also use the word-emoji embedding to calculate the emoji-word attention following Equation EQREF20 and EQREF21 , and the only difference is that we replace the attention-derived senti-emoji embedding with the pre-trained word-emoji embedding by fasttext, denoted as ATT-E-LSTM. 
 Question: What is the baseline for experiments?
Output: LSTM with text embedding LSTM with emoji embedding Attention-based LSTM with emojis
Our baseline system (Baseline_1850K) is taken from BIBREF13 . It consists of a DNN trained to predict subword targets within the keywords. The input to the DNN consists of a sequence with INLINEFORM0 frames of left and INLINEFORM1 frames of right context; each with a stride of INLINEFORM2 . The topology consists of a 1-D convolutional layer with 92 filters (of shape 8x8 and stride 8x8), followed by 3 fully-connected layers with 512 nodes and a rectified linear unit activation each. A final softmax output predicts the 7 subword targets, obtained from the same forced alignment process described in SECREF5 . This results in the baseline DNN containing 1.7M parameters, and performing 1.8M multiply-accumulate operations per inference (every 30ms of streaming audio). A keyword spotting score between 0 and 1 is computed by first smoothing the posterior values, averaging them over a sliding window of the previous 100 frames with respect to the current INLINEFORM3 ; the score is then defined as the largest product of the smoothed posteriors in the sliding window as originally proposed in BIBREF6 . 
 Question: What previous approaches are considered?
Output: Our baseline system (Baseline_1850K) is taken from BIBREF13 .
Given the relatively small quest length for TextWorld games—games can be completed in as little as 5 steps—we generate 50 such games and partition them into train and test sets in a 4:1 ratio. We choose the game, 9:05 as our target task game due to similarities in structure in addition to the vocabulary overlap.  For the horror domain, we choose Lurking Horror to train the question-answering system on. The source and target task games are chosen as Afflicted and Anchorhead respectively. 
 Question: What games are used to test author's methods?
Output: Lurking Horror Afflicted Anchorhead 9:05 TextWorld games
The text and image encodings were combined by concatenation, which resulted in a feature vector of 4,864 dimensions. This multimodal representation was afterward fed as input into a multi-layer perceptron (MLP) with two hidden layer of 100 neurons with a ReLU activation function. 
 Question: Is the dataset multimodal?
Output: Yes
We compare eight different models in our experiments. Four of them have a Projected Layer (see Fig. FIGREF2), while the others do not have, and this is the only difference between these two groups of our models. So, we actually include four models in our experiments (having a projected layer or not). Firstly, LastStateRNN is the classic RNN model, where the last state passes through an MLP and then the LR Layer estimates the corresponding probability. In contrast, in the AvgRNN model we consider the average vector of all states that come out of the cells. The AttentionRNN model is the one that it has been presented in BIBREF9. Moreover, we introduce the MultiAttentionRNN model for the harassment language detection, which instead of one attention, it includes four attentions, one for each category. 
 Question: What are the different variations of the attention-based approach which are examined?
Output: classic RNN model, avgRNN model, attentionRNN model and multiattention RNN model with and without a projected layer
So, we can impose the constraint that our model's representation of the input's syntax cannot contain this context-invariant information. This regularization is strictly preferable to allowing all aspects of word meaning to propagate into the input's syntax representation. Without such a constraint, all inputs could, in principle, be given their own syntactic categories. 
 Question: Does having constrained neural units imply word meanings are fixed across different context?
Output: No
We also run a manual evaluation which shows for the En-It task a slight quality degradation in exchange of a statistically significant reduction in the average length ratio, from 1.05 to 1.01. 
 Question: Do they conduct any human evaluation?
Output: Yes
A quantitative measure is required to reliably evaluate the achieved improvement. One of the methods proposed to measure the interpretability is the word intrusion test BIBREF41 . But, this method is expensive to apply since it requires evaluations from multiple human evaluators for each embedding dimension. In this study, we use a semantic category-based approach based on the method and category dataset (SEMCAT) introduced in BIBREF27 to quantify interpretability. Specifically, we apply a modified version of the approach presented in BIBREF40 in order to consider possible sub-groupings within the categories in SEMCAT.  
 Question: What experiments do they use to quantify the extent of interpretability?
Output: Human evaluation for interpretability using the word intrusion test and automated evaluation for interpretability using a semantic category-based approach based on the method and category dataset (SEMCAT).
Benchmark Evaluation ::: Classifier Models
SVM: A linear support vector machine with bag-of-words sentence representations.

MLP: A multi-layer perceptron with USE embeddings BIBREF4 as input.

FastText: A shallow neural network that averages embeddings of n-grams BIBREF5.

CNN: A convolutional neural network with non-static word embeddings initialized with GloVe BIBREF6.

BERT: A neural network that is trained to predict elided words in text and then fine-tuned on our data BIBREF1.

Platforms: Several platforms exist for the development of task-oriented agents. We consider Google's DialogFlow and Rasa NLU with spacy-sklearn. 
 Question: Which classifiers are evaluated?
Output: SVM MLP FastText CNN BERT Google's DialogFlow Rasa NLU
By providing the Wizard with several dialogue options (aside from free text), we guided the conversation and could introduce actions that change an internal system state. This proposes several advantages:

A guided dialogue allows for set procedures to be learned and reduces the amount of data needed for a machine learning model for dialogue management to converge.

Providing several dialogue options to the Wizard increases the pace of the interaction and allows them to understand and navigate more complex scenarios. Dialogue structure: we introduced structured dialogues through a Finite State Machine (FSM) that controls the current dialogue state and offers multiple suitable and relevant state transitions (actions) to the Wizard depending on the point in the interaction, the state of the world and the history. A graph of dialogue states, transitions and utterances is loaded when the system is initialised, and each chat room has its own dialogue state, which changes through actions. In addition to the transitions that the FSM provides, there are other fixed dialogue options always available such as “Hold on, 2 seconds”, “Okay” or “Sorry, can you repeat that?” as a shortcut for commonly used dialogue acts, as well as the option to type a message freely. The dialogue has several paths to reach the same states with varying levels of Operator control or engagement that enriched the heterogeneity of conversations. 
 Question: What is meant by semiguided dialogue, what part of dialogue is guided?
Output: The Wizard can select one of several predefined messages to send, or type their own message if needed. Free text messages do not change the dialogue state in the FSM, so it is important to minimise their use by providing enough dialogue options to the Wizard.
Then a comparison to a maximum entropy (MaxEnt) proposed in BIBREF23 type of discriminative model and finally a deep neural network (DNN) architecture proposed in BIBREF24 as reported also in BIBREF22 is presented.

 
 Question: What state-of-the-art models are compared against?
Output: a deep neural network (DNN) architecture proposed in BIBREF24  maximum entropy (MaxEnt) proposed in BIBREF23 type of discriminative model
For task 1, we use F1-score as evaluation metric. We use manual evaluation for task 2. There are five evaluation metrics for task 2 as following.

Task completion ratio: The number of completed tasks divided by the number of total tasks.

User satisfaction degree: There are five scores -2, -1, 0, 1, 2, which denote very dissatisfied, dissatisfied, neutral, satisfied and very satisfied, respectively.

Response fluency: There are three scores -1, 0, 1, which indicate nonfluency, neutral, fluency.

Number of dialogue turns: The number of utterances in a task-completed dialogue.

Guidance ability for out of scope input: There are two scores 0, 1, which represent able to guide or unable to guide. 
 Question: What metrics are used in the evaluation?
Output: For task 1, we use F1-score Task completion ratio User satisfaction degree Response fluency Number of dialogue turns Guidance ability for out of scope input
We perform experiments using the following state-of-the-art models: (1) SEQ2SEQ: a sequence-to-sequence model with attention mechanisms BIBREF21, (2) CVAE: a conditional variational auto-encoder model with KL-annealing and a BOW loss BIBREF2, (3) Transformer: an encoder-decoder architecture relying solely on attention mechanisms BIBREF22, (4) HRED: a generalized sequence-to-sequence model with the hierarchical RNN encoder BIBREF23, (5) DialogWAE: a conditional Wasserstein auto-encoder, which models the distribution of data by training a GAN within the latent variable space BIBREF6. 
 Question: What state of the art models were used in experiments?
Output: SEQ2SEQ CVAE Transformer HRED DialogWAE
We introduce here the use of an L-PCFG with two layers of latent states: one layer is intended to capture the usual syntactic information, and the other aims to capture semantic and topical information by using a large set of states with specific feature functions. 
 Question: What latent variables are modeled in the PCFG?
Output: syntactic information semantic and topical information
Below, we explore different ways to back off when the ScRNN predicts UNK (a frequent outcome for rare and unseen words):

Pass-through: word-recognizer passes on the (possibly misspelled) word as is.

Backoff to neutral word: Alternatively, noting that passing $\colorbox {gray!20}{\texttt {UNK}}$ -predicted words through unchanged exposes the downstream model to potentially corrupted text, we consider backing off to a neutral word like `a', which has a similar distribution across classes.

Backoff to background model: We also consider falling back upon a more generic word recognition model trained upon a larger, less-specialized corpus whenever the foreground word recognition model predicts UNK. Figure 1 depicts this scenario pictorially. 
 Question: How do the backoff strategies work?
Output: In pass-through, the recognizer passes on the possibly misspelled word, backoff to neutral word backs off to a word with similar distribution across classes and backoff to background model backs off to a more generic word recognition model trained with larger and less specialized corpus.
Next, for emergent language, we show that the MASC architecture can achieve very high localization accuracy, significantly outperforming the baseline that does not include this mechanism. 
 Question: What evaluation metrics did the authors look at?
Output: localization accuracy
Throughout all our experiments, we used BERT-Base BIBREF2 to provide the state-of-the-art contextualized modeling of the input text.

Semantic Retrieval: We treated the neural semantic retrieval at both the paragraph and sentence level as binary classification problems with models' parameters updated by minimizing binary cross entropy loss. 
 Question: How do they model the neural retrieval modules?
Output: BERT-Base BIBREF2 to provide the state-of-the-art contextualized modeling
The proposed system consists of a Bi-directional Long Short-Term Memory (BiLSTM) BIBREF16, a two-level attention mechanism BIBREF29, BIBREF30 and a shared representation for emotion and sentiment analysis tasks. The BiLSTM encodes the word representation of each word. This representation is shared between the subsystems of sentiment and emotion analysis. Each of the shared representations is then fed to the primary attention mechanism of both the subsystems. The primary attention mechanism finds the best representation for each word for each task. The secondary attention mechanism acts on top of the primary attention to extract the best sentence representation by focusing on the suitable context for each task. Finally, the representations of both the tasks are fed to two different feed-forward neural networks to produce two outputs - one for sentiment analysis and one for emotion analysis. Each component is explained in the subsequent subsections. 
 Question: How is multi-tasking performed?
Output: The proposed system consists of a Bi-directional Long Short-Term Memory (BiLSTM) BIBREF16, a two-level attention mechanism BIBREF29, BIBREF30 and a shared representation for emotion and sentiment analysis tasks. Each of the shared representations is then fed to the primary attention mechanism
IDEA
BIBREF9 Two different BERT models were developed. For Friends, pre-training was done using a sliding window of two utterances to provide dialogue context. Both Next Sentence Prediction (NSP) phase on the complete unlabeled scripts from all 10 seasons of Friends, which are available for download. In addition, the model learned the emotional disposition of each of six main six main characters in Friends (Rachel, Monica, Phoebe, Joey, Chandler and Ross) by adding a special token to represent the speaker. For EmotionPush, pre-training was performed on Twitter data, as it is similar in nature to chat based dialogues. In both cases, special attention was given to the class imbalance issue by applying “weighted balanced warming” on the loss function. 
 Question: What model was used by the top team?
Output: Two different BERT models were developed
We decided to use sentences involving at least one race- or gender-associated word. The sentences were intended to be short and grammatically simple. We also wanted some sentences to include expressions of sentiment and emotion, since the goal is to test sentiment and emotion systems. 
 Question: What criteria are used to select the 8,640 English sentences?
Output: Sentences involving at least one race- or gender-associated word,  sentence  have to be short and grammatically simple,  sentence have to  include expressions of sentiment and emotion.
We present here a step in this direction: a probabilistic semantic parser that uses a large knowledge base (NELL) to form a prior probability distribution on the meanings of sentences it parses, and that "understands" each sentence either by identifying its existing beliefs that correspond to the sentence's meaning, or by creating new beliefs. 
 Question: What knowledge bases do they use?
Output: NELL
The Digital Library in the TextGrid Repository represents an extensive collection of German texts in digital form BIBREF3. 
 Question: What is the corpus used in the study?
Output: TextGrid Repository
We evaluate CBA, LSA, DCA, and GMMv2b using mean opinion score (MOS) naturalness judgments produced by a crowd-sourced pool of raters. Scores range from 1 to 5, with 5 representing “completely natural speech”. 
 Question: How they compare varioius mechanisms in terms of naturalness?
Output: using mean opinion score (MOS) naturalness judgments produced by a crowd-sourced pool of raters
As previously mentioned, there is no German hate speech corpus available for our needs, especially not for the very recent topic of the refugee crisis in Europe. We therefore had to compile our own corpus. 
 Question: What languages are were included in the dataset of hateful content?
Output: German
We evaluate our framework on fastText embeddings trained on Wikipedia (2017), UMBC webbase corpus and statmt.org news dataset (16B tokens) BIBREF11. For simplicity, only the first 22000 words are used in all embeddings, though preliminary results indicate the findings extend to the full corpus. For our novel methods of mitigating bias, a shallow neural network is used to adjust the embedding. The single layer of the model is an embedding layer with weights initialized to those of the original embedding. For the composite method, these weights are initialized to those of the embedding after probabilistic bias mitigation. A batch of word indices is fed into the model, which are then embedded and for which a loss value is calculated, allowing back-propagation to adjust the embeddings. For each of the models, a fixed number of iterations is used to prevent overfitting, which can eventually hurt performance on the embedding benchmarks (See Figure FIGREF12). We evaluated the embedding after 1000 iterations, and stopped training if performance on a benchmark decreased significantly. We construct a list of candidate words to debias, taken from the words used in the WEAT gender bias statistics. Words in this list should be gender neutral, and are related to the topics of career, arts, science, math, family and professions (see appendix). We note that this list can easily be expanded to include a greater proportion of words in the corpus. For example, BIBREF4 suggested a method for identifying inappropriately gendered words using unsupervised learning. We compare this method of bias mitigation with the no bias mitigation ("Orig"), geometric bias mitigation ("Geo"), the two pieces of our method alone ("Prob" and "KNN") and the composite method ("KNN+Prob"). We note that the composite method performs reasonably well according the the RIPA metric, and much better than traditional geometric bias mitigation according to the neighborhood metric, without significant performance loss according to the accepted benchmarks. To our knowledge this is the first bias mitigation method to perform reasonably both on both metrics. 
 Question: How is embedding quality assessed?
Output: We compare this method of bias mitigation with the no bias mitigation ("Orig"), geometric bias mitigation ("Geo"), the two pieces of our method alone ("Prob" and "KNN") and the composite method ("KNN+Prob"). We note that the composite method performs reasonably well according the the RIPA metric, and much better than traditional geometric bias mitigation according to the neighborhood metric, without significant performance loss according to the accepted benchmarks. To our knowledge this is the first bias mitigation method to perform reasonably both on both metrics.
Recently, a number of researchers have endeavored to explore methods for simultaneous translation in the context of NMT BIBREF6, BIBREF7, BIBREF8, BIBREF9. Some of them propose sophisticated training frameworks explicitly designed for simultaneous translation BIBREF5, BIBREF10.  
 Question: Has there been previous work on SNMT?
Output: Yes
To evaluate the proposed BiLSTM model with attention (BiLSTM+att), it is compared with three of its own variants: a BiLSTM without attention (BiLSTM) as well as a single forward-LSTM layer with attention (LSTM+att) and without attention (LSTM). Additional baselines are defined by BIBREF32 who already proposed an LSTM-based architecture that only uses non-temporal features, and the SVM-based estimation model as originally used for reward estimation by BIBREF24. 
 Question: What model do they use a baseline to estimate satisfaction?
Output: a BiLSTM without attention (BiLSTM) as well as a single forward-LSTM layer with attention (LSTM+att) and without attention (LSTM) baselines are defined by BIBREF32 who already proposed an LSTM-based architecture that only uses non-temporal features, and the SVM-based estimation model as originally used for reward estimation by BIBREF24
Specifically, PlEWi supplied 550,755 [error, correction] pairs, from which 298,715 were unique. 
 Question: How is PIEWi annotated?
Output: [error, correction] pairs
Many NLP tasks utilize POS as features, but human annotated POS sequences are difficult and expensive to obtain. Thus, it is important to know if we can learn sentences-level syntactic embeddings for low-sources languages without treebanks.

We performed zero-shot transfer of the syntactic embeddings for French, Portuguese and Indonesian. French and Portuguese are simulated low-resource languages, while Indonesian is a true low-resource language. 
 Question: Do they evaluate on downstream tasks?
Output: Yes
The Hausa data used in this paper is part of the LORELEI language pack. It consists of Broad Operational Language Translation (BOLT) data gathered from news sites, forums, weblogs, Wikipedia articles and twitter messages. We use a split of 10k training and 1k test instances. The Yorùbá NER data used in this work is the annotated corpus of Global Voices news articles recently released by BIBREF22. The dataset consists of 1,101 sentences (26k tokens) divided into 709 training sentences, 113 validation sentences and 279 test sentences based on 65%/10%/25% split ratio. 
 Question: How much labeled data is available for these two languages?
Output: 10k training and 1k test 1,101 sentences (26k tokens)
Which events authors choose to include in their history, which they leave out, and the way the events chosen relate to the march are crucial factors in conveying bias. 
 Question: What factors contribute to interpretive biases according to this research?
Output: Which events authors choose to include in their history, which they leave out, and the way the events chosen relate to the march
In this paper, we manually annotate the predicate–argument structures for the 600 L2-L1 pairs as the basis for the semantic analysis of learner Chinese. 
 Question: Who manually annotated the semantic roles for the set of learner texts?
Output: Authors
Our dataset is annotated based on Chinese pathology reports provided by the Department of Gastrointestinal Surgery, Ruijin Hospital. It contains 17,833 sentences, 826,987 characters and 2,714 question-answer pairs. All question-answer pairs are annotated and reviewed by four clinicians with three types of questions, namely tumor size, proximal resection margin and distal resection margin.  
 Question: Is all text in this dataset a question, or are there unrelated sentences in between questions?
Output: the dataset consists of pathology reports including sentences and questions and answers about tumor size and resection margins so it does include additional sentences
The proposed Multimodal Differential Network (MDN) consists of a representation module and a joint mixture module. We use a triplet network BIBREF41 , BIBREF42 in our representation module. The triplet network consists of three sub-parts: target, supporting, and contrasting networks. All three networks share the same parameters. The Mixture module brings the image and caption embeddings to a joint feature embedding space. 
 Question: How do the authors define a differential network?
Output: The proposed Multimodal Differential Network (MDN) consists of a representation module and a joint mixture module.