niryuu/sni-each-converted huggingface dataset:
Detection: In hypernymy detection, the task is to classify whether pairs of words are in a hypernymy relation. For this task, we evaluate all models on five benchmark datasets: First, we employ the noun-noun subset of bless, which contains hypernymy annotations for 200 concrete, mostly unambiguous nouns. Second, we evaluate on leds BIBREF13 , which consists of 2,770 noun pairs balanced between positive hypernymy examples, and randomly shuffled negative pairs. Direction: In direction prediction, the task is to identify which term is broader in a given pair of words. For this task, we evaluate all models on three datasets described by BIBREF16 : On bless, the task is to predict the direction for all 1337 positive pairs in the dataset. Pairs are only counted correct if the hypernymy direction scores higher than the reverse direction, i.e. INLINEFORM0 . We reserve 10% of the data for validation, and test on the remaining 90%. On wbless, we follow prior work BIBREF17 , BIBREF18 and perform 1000 random iterations in which 2% of the data is used as a validation set to learn a classification threshold, and test on the remainder of the data. We report average accuracy across all iterations. Finally, we evaluate on bibless BIBREF16 , a variant of wbless with hypernymy and hyponymy pairs explicitly annotated for their direction. Graded Entailment: In graded entailment, the task is to quantify the degree to which a hypernymy relation holds. For this task, we follow prior work BIBREF19 , BIBREF18 and use the noun part of hyperlex BIBREF20 , consisting of 2,163 noun pairs which are annotated to what degree INLINEFORM0 is-a INLINEFORM1 holds on a scale of INLINEFORM2 . 
 Question: Which benchmark datasets are used?
Output: noun-noun subset of bless leds BIBREF13 bless wbless bibless hyperlex BIBREF20
We evaluate our model on a publicly available dataset, KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw). The dataset consists of 14 participants, with each prompt presented 11 times to each individual.  
 Question: How many subjects does the EEG data come from?
Output: 14
A community with a very distinctive identity will tend to have distinctive interests, expressed through specialized language. Formally, we define the distinctiveness of a community INLINEFORM0 as the average specificity of all utterances in INLINEFORM1  
 Question: How do the authors measure how distinctive a community is?
Output:  the average specificity of all utterances
Because our conversion rules are interpretable, we identify shortcomings in both resources, using each as validation for the other. We were able to find specific instances of incorrectly applied UniMorph annotation, as well as specific instances of cross-lingual inconsistency in both resources. 
 Question: Do they look for inconsistencies between different languages' annotations in UniMorph?
Output: Yes
We evaluate a character-level variant of our proposed language model over a preprocessed version of the Penn Treebank (PTB) and Text8 datasets. The unsupervised constituency parsing task compares hte tree structure inferred by the model with those annotated by human experts. The experiment is performed on WSJ10 dataset. 
 Question: Which dataset do they experiment with?
Output: Penn Treebank Text8 WSJ10
In this section, we present the details of the analysis performed on the data obtained pertaining to Twitter messages from January 2020 upto now, that is the time since the news of the Coronavirus outbreak in China was spread across nations. Also, the data prominently captures the tweets in English, Spanish, and French languages. 
 Question: Do they collect only English data?
Output: No
Our findings paves the way for research in several new directions. While we probed the effectiveness of virtual edges in a specific text classification task, we could extend this approach for general classification tasks. A systematic comparison of embeddings techniques could also be performed to include other recent techniques BIBREF54, BIBREF55. We could also identify other relevant techniques to create virtual edges, allowing thus the use of the methodology in other networked systems other than texts. For example, a network could be enriched with embeddings obtained from graph embeddings techniques. A simpler approach could also consider link prediction BIBREF56 to create virtual edges. Finally, other interesting family of studies concerns the discrimination between co-occurrence and virtual edges, possibly by creating novel network measurements considering heterogeneous links. 
 Question: What other natural processing tasks authors think could be studied by using word embeddings?
Output: general classification tasks use of the methodology in other networked systems a network could be enriched with embeddings obtained from graph embeddings techniques
In this work, we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way BIBREF8. As the first step, we focus on building a bilingual language model (LM) of English and a target language. Starting from a pre-trained English LM, we learn the target language specific parameters (i.e., word embeddings), while keeping the encoder layers of the pre-trained English LM fixed. We then fine-tune both English and target model to obtain the bilingual LM.  
 Question: How is the model transferred to other languages?
Output: Build a bilingual language model,   learn the target language specific parameters starting from a pretrained English LM , fine-tune both English and target model to obtain the bilingual LM.
We define hateful speech to be the language which explicitly or implicitly threatens or demeans a person or a group based upon a facet of their identity such as gender, ethnicity, or sexual orientation. 
 Question: What is their definition of hate speech?
Output: language which explicitly or implicitly threatens or demeans a person or a group based upon a facet of their identity such as gender, ethnicity, or sexual orientation
According the distribution of the sentiment score, the sentiment on Tesla is slightly skewed towards positive during the testing period. 
 Question: Do the authors give any examples of major events which draw the public's attention and the impact they have on stock price?
Output: Yes
Considering the improvements over the majority baseline achieved by the RNN model for both non-English (on the average 22.76% relative improvement; 15.82% relative improvement on Spanish, 72.71% vs. 84.21%, 30.53% relative improvement on Turkish, 56.97% vs. 74.36%, 37.13% relative improvement on Dutch, 59.63% vs. 81.77%, and 7.55% relative improvement on Russian, 79.60% vs. 85.62%) and English test sets (27.34% relative improvement), we can draw the conclusion that our model is robust to handle multiple languages. 
 Question: which non-english language was the had the worst results?
Output: Turkish
Tweets IDs 6, 8, and 10 are some samples containing offensive words and slurs which arenot hate or offensive in all cases and writers of them used this type of language in their daily communications. Given these pieces of evidence, by considering the content of tweets, we can see in tweets IDs 3, 4, and 9 that our BERT-based classifier can discriminate tweets in which neither and implicit hatred content exist. 
 Question: What evidence do the authors present that the model can capture some biases in data annotation and collection?
Output: The authors showed few tweets where neither and implicit hatred content exist but the model was able to discriminate
WAS: The architecture used in BIBREF3 without the audio input. The decoder output Chinese character at each timestep. Others keep unchanged to the original implementation. LipCH-Net-seq: For a fair comparison, we use sequence-to-sequence with attention framework to replace the Connectionist temporal classification (CTC) loss BIBREF14 used in LipCH-Net BIBREF5 when converting picture to pinyin. CSSMCM-w/o video: To evaluate the necessity of video information when predicting tone, the video stream is removed when predicting tone and Chinese characters. In other word, video is only used when predicting the pinyin sequence. The tone is predicted from the pinyin sequence. Tone information and pinyin information work together to predict Chinese character. 
 Question: What was the previous state of the art model for this task?
Output: WAS LipCH-Net-seq CSSMCM-w/o video
We introduce a new model for difficulty prediction combining learned representations induced via a pre-trained `universal' sentence encoder BIBREF6 , and a sentence encoder learned from scratch for this task. In total, this resulted in 57,505 and 2,428 sentences in the train and test set abstracts, respectively. 
 Question: How much data is needed to train the task-specific encoder?
Output: 57,505 sentences
This metric is used to evaluate the accuracy of dialogue belief tracker BIBREF1.

APRA: Action Per-Response Accuracy (APRA) evaluates the per-turn accuracy of the dialogue actions generated by dialogue policy maker. For baselines, APRA evaluates the classification accuracy of the dialogue policy maker. But our model actually generates each individual token of actions, and we consider a prediction to be correct only if every token of the model output matches the corresponding token in the ground truth.

BLEU BIBREF19: The metric evaluates the quality of the final response generated by natural language generator. The metric is usually used to measure the performance of the task-oriented dialogue system. 
 Question: What metrics are used to measure performance of models?
Output: BPRA APRA BLEU
We are in line with recent work BIBREF16 , proposing to shift evaluation from absolute values to more exploratory evaluations focusing on weaknesses and strengths of the embeddings and not so much in generic scores. For example, one metric could consist in checking whether for any given word, all words that are known to belong to the same class are closer than any words belonging to different classes, independently of the actual cosine. 
 Question: What new metrics are suggested to track progress?
Output:  For example, one metric could consist in checking whether for any given word, all words that are known to belong to the same class are closer than any words belonging to different classes, independently of the actual cosine
However, we also showed difference in produced errors for each method and different impact at word-level depending of the approach or units. Thus, future work will focus on analysing the orthographic output of these systems in two ways: 1) investigate errors produced by the end-to-end methods and explore several approaches to correct common errors done in French and 2) compare the end-to-end methods in a SLU context and evaluate the semantic value of the partially correct produced words. 
 Question: What will be in focus for future work?
Output: 1) investigate errors produced by the end-to-end methods and explore several approaches to correct common errors done in French 2) compare the end-to-end methods in a SLU context and evaluate the semantic value of the partially correct produced words
Adhering to the experimental settings of BIBREF1 , we concatenate sentence representations produced from our model with those obtained from the state-of-the-art unsupervised learning model (Layer Normalized Skip-Thoughts, ST-LN) BIBREF31 . Following the experimental design of BIBREF1 , we conduct experiments on three different learning objectives: Cap2All, Cap2Cap, Cap2Img. 
 Question: What baselines are the proposed method compared against?
Output: (Layer Normalized Skip-Thoughts, ST-LN) BIBREF31 Cap2All, Cap2Cap, Cap2Img
In the first scenario we equip the decoder with an additional morphology table including target-side affixes. We place an attention module on top of the table which is controlled by the decoder. At each step, as the decoder samples a character, it searches the table to find the most relevant information which can enrich its state. Signals sent from the table can be interpreted as additional constraints. In the second scenario we share the decoder between two output channels. The first one samples the target character and the other one predicts the morphological annotation of the character. This multi-tasking approach forces the decoder to send morphology-aware information to the final layer which results in better predictions. In the third scenario we combine these two models.  
 Question: How are the auxiliary signals from the morphology table incorporated in the decoder?
Output: an additional morphology table including target-side affixes. We inject the decoder with morphological properties of the target language.
WikiTableQuestions contains 22,033 questions and is an order of magnitude larger than previous state-of-the-art datasets. Its questions were not designed by predefined templates but were hand crafted by users, demonstrating high linguistic variance. 
 Question: How do they gather data for the query explanation problem?
Output: hand crafted by users
While the neural syntax-agnostic system obtains superior performance on the L1 data, the two syntax-based systems both produce better analyses on the L2 data. Furthermore, as illustrated in the comparison between different parsers, the better the parsing results we get, the better the performance on L2 we achieve. This shows that syntactic parsing is important in semantic construction for learner Chinese. The main reason, according to our analysis, is that the syntax-based system may generate correct syntactic analyses for partial grammatical fragments in L2 texts, which provides crucial information for SRL. 
 Question: Do the authors suggest why syntactic parsing is so important for semantic role labelling for interlanguages?
Output: syntax-based system may generate correct syntactic analyses for partial grammatical fragments
The seed lexicon consists of positive and negative predicates. If the predicate of an extracted event is in the seed lexicon and does not involve complex phenomena like negation, we assign the corresponding polarity score ($+1$ for positive events and $-1$ for negative events) to the event. It is a  
 Question: What is the seed lexicon?
Output: a vocabulary of positive and negative predicates that helps determine the polarity score of an event
Additionally, we also test their performance only on the items from the Zurich and Visp dialect, because most of the samples are from this two dialects. In Table TABREF15 we show the result of the comparison of the two models. 
 Question: Is the model evaluated on the graphemes-to-phonemes task?
Output: Yes
To test the effectiveness of this new architecture in forecasting derailment of online conversations, we develop and distribute two new datasets. The first triples in size the highly curated `Conversations Gone Awry' dataset BIBREF9, where civil-starting Wikipedia Talk Page conversations are crowd-labeled according to whether they eventually lead to personal attacks; the second relies on in-the-wild moderation of the popular subreddit ChangeMyView, where the aim is to forecast whether a discussion will later be subject to moderator action due to “rude or hostile” behavior.  
 Question: What are two datasets model is applied to?
Output:  `Conversations Gone Awry' dataset subreddit ChangeMyView
In this work, we use the datasets released by BIBREF1 and HEOT dataset provided by BIBREF0 .  Both the HEOT and BIBREF1 datasets contain tweets which are annotated in three categories: offensive, abusive and none (or benign) 
 Question: Do they perform some annotation?
Output: No
In this section, we propose new simple disentanglement models that perform better than prior methods, and re-examine prior work. 
 Question: Did they experiment with the corpus?
Output: Yes
This effect of context on human ratings is very similar to the one reported in BIBREF5 . They find that sentences rated as ill formed out of context are improved when they are presented in their document contexts. BIBREF5 suggest that adding context causes speakers to focus on broader semantic and pragmatic issues of discourse coherence, rather than simply judging syntactic well formedness (measured as naturalness) when a sentence is considered in isolation. 
 Question: What provisional explanation do the authors give for the impact of document context?
Output: adding context causes speakers to focus on broader semantic and pragmatic issues of discourse coherence
Furthermore, the gains in WER over the baseline are significantly larger for the Density Ratio method than for Shallow Fusion, with up to 28% relative reduction in WER (17.5% $\rightarrow $ 12.5%) compared to up to 17% relative reduction (17.5% $\rightarrow $ 14.5%) for Shallow Fusion, in the no fine-tuning scenario. 
 Question: What metrics are used for evaluation?
Output: word error rate
The resulting dataset is nearly balanced, with 52.3% of the data (1,857 instances) labeled stressful. 
 Question: Is the dataset balanced across categories?
Output: Yes
This framework has potential applications when comparing different gold standards, considering the design choices for a new gold standard and performing qualitative error analyses for a proposed approach. 
 Question: Have they made any attempt to correct MRC gold standards according to their findings? 
Output: Yes
We create a new manually annotated multimodal hate speech dataset formed by $150,000$ tweets, each one of them containing text and an image. We call the dataset MMHS150K, and made it available online . In this section, we explain the dataset creation steps. 
 Question: How many tweats does MMHS150k contains, 150000?
Output: $150,000$ tweets
Instead of providing an analytic approximation, AEM uses a discriminator network to discriminate between the reconstructed documents from latent events and the original input documents. This essentially helps the generator to construct a more realistic document from a random noise drawn from a Dirichlet distribution. Due to the flexibility of neural networks, the generator is capable of learning complicated nonlinear distributions. And the supervision signal provided by the discriminator will help generator to capture the event-related patterns. 
 Question: How does this model overcome the assumption that all words in a document are generated from a single event?
Output: flexibility of neural networks, the generator is capable of learning complicated nonlinear distributions supervision signal provided by the discriminator will help generator to capture the event-related patterns
Baseline Experiments
In this section, we briefly describe a baseline and evaluation scripts that we release, with a detailed documentation, along with the corpus. 
 Question: What type of evaluation is proposed for this task?
Output: Answer with content missing: (Evaluation Metrics section) Precision, Recall, F1-scores, Strict match, METEOR, ROUGE-2
We end up eliminating 5.86% of total characters, and end up with 45,821 characters and 12,815 unique HLA, resulting in 945,519 total character-HLA pairs.  We end up eliminating 5.86% of total characters, and end up with 45,821 characters and 12,815 unique HLA, resulting in 945,519 total character-HLA pairs. 
 Question: How many different characters were in dataset?
Output: 45,821 characters
Our baseline decoder is a standard beam search decoder BIBREF5 with several straightforward performance optimizations 
 Question: What baseline decoder do they use?
Output: a standard beam search decoder BIBREF5 with several straightforward performance optimizations
We crowdsourced multiple-choice questions in two parts, encouraging workers to be imaginative and varied in their use of language. First, workers were given a seed qualitative relation q+/-( INLINEFORM0 ) in the domain, expressed in English (e.g., “If a surface has more friction, then an object will travel slower”), and asked to enter two objects, people, or situations to compare. 
 Question: Do all questions in the dataset allow the answers to pick from 2 options?
Output: Yes
Impact votes are provided by the users of the platform to evaluate how impactful a particular claim is. Users can pick one of 5 possible impact labels for a particular claim: no impact, low impact, medium impact, high impact and very high impact. While evaluating the impact of a claim, users have access to the full argument context and therefore, they can assess how impactful a claim is in the given context of an argument. 
 Question: What annotations are available in the dataset?
Output: 5 possible impact labels for a particular claim: no impact, low impact, medium impact, high impact and very high impact
We create three additional training datasets by adding sentences involving object RCs to the original Wikipedia corpus (Section lm). To this end, we randomly pick up 30 million sentences from Wikipedia (not overlapped to any sentences in the original corpus), parse by the same parser, and filter sentences containing an object RC, amounting to 680,000 sentences.  
 Question: How do they perform data augmentation?
Output: They randomly sample sentences from Wikipedia that contains an object RC and add them to training data
We consider two types of architectures: a bi-directional language model to augment the sequence to sequence encoder and a uni-directional model to augment the decoder. Both use self-attention BIBREF16 and the uni-directional model contains INLINEFORM0 transformer blocks, followed by a word classifier to predict the next word on the right. bi-directional language model to augment the sequence to sequence encoder 
 Question: What language model architectures are used?
Output: uni-directional model to augment the decoder
Results reveal that, contrary to the common belief, correlations between Rouge and the Pyramid scores are weak, which challenges its effectiveness for scientific summarization. Furthermore, we show a large variance of correlations between different Rouge variants and the manual evaluations which further makes the reliability of Rouge for evaluating scientific summaries less clear. 
 Question: What is the common belief that this paper refutes? (c.f. 'contrary to the common belief, ROUGE is not much [sic] reliable'
Output: correlations between Rouge and the Pyramid scores are weak, which challenges its effectiveness for scientific summarization
We use the Fisher Spanish speech corpus BIBREF11, which consists of 819 phone calls, with an average duration of 12 minutes, amounting to a total of 160 hours of data. 
 Question: What language do they look at?
Output: Spanish
Impact votes are provided by the users of the platform to evaluate how impactful a particular claim is. Users can pick one of 5 possible impact labels for a particular claim: no impact, low impact, medium impact, high impact and very high impact. While evaluating the impact of a claim, users have access to the full argument context and therefore, they can assess how impactful a claim is in the given context of an argument. An interesting observation is that, in this dataset, the same claim can have different impact labels depending on the context in which it is presented. 
 Question: How is pargmative and discourse context added to the dataset?
Output: While evaluating the impact of a claim, users have access to the full argument context and therefore, they can assess how impactful a claim is in the given context of an argument.
For our experiment we decided to use the Europarl dataset, using the data from the WMT11 . 
 Question: Are any experiments performed to try this approach to word embeddings?
Output: Yes
In our experiments, the Memory Neural Networks (MemNNs) proposed in babidataset serve as the baselines. 
 Question: What word level and character level model baselines are used?
Output: None
We evaluate our newly proposed models and related baselines in several low-resource settings across different languages with real, distantly supervised data with non-synthetic noise. The advanced modeling of the noisy labels substantially improves the performance up to 36% over methods without noise-handling and up to 9% over all other noise-handling baselines. 
 Question: Did they evaluate against baseline?
Output: Yes
We evaluate our model on both English and Chinese segmentation. For both languages we used standard datasets for word segmentation and language modeling. English
The Brent corpus is a standard corpus used in statistical modeling of child language acquisition BIBREF15 , BIBREF16  We use the commonly used version of the PTB prepared by BIBREF17 Since Chinese orthography does not mark spaces between words, there have been a number of efforts to annotate word boundaries. We evaluate against two corpora that have been manually segmented according different segmentation standards. The Beijing University Corpus was one of the corpora used for the International Chinese Word Segmentation Bakeoff BIBREF18 . We use the Penn Chinese Treebank Version 5.1 BIBREF19 . 
 Question: What dataset is used?
Output: Brent corpus PTB  Beijing University Corpus Penn Chinese Treebank
However, all previous IE benchmarks BIBREF18 are too small to train neural network models typically used in QA, and thus we need to build a large benchmark. Therefore, we build a large scale benchmark named QA4IE benchmark which consists of 293K Wikipedia articles and 2M golden relation triples with 636 different relation types. We manually find 148 relations which can be projected to a WikiData relation out of 2064 DBpedia relations. 
 Question: Was this benchmark automatically created from an existing dataset?
Output: No
We conducted our experiments on the CMU ARCTIC database BIBREF33, which contains parallel recordings of professional US English speakers sampled at 16 kHz. One female (slt) was chosen as the target speaker and one male (bdl) and one female (clb) were chosen as sources. We selected 100 utterances each for validation and evaluation, and the other 932 utterances were used as training data. For the TTS corpus, we chose a US female English speaker (judy bieber) from the M-AILABS speech dataset BIBREF34 to train a single-speaker Transformer-TTS model. With the sampling rate also at 16 kHz, the training set contained 15,200 utterances, which were roughly 32 hours long. 
 Question: What datasets are experimented with?
Output: the CMU ARCTIC database BIBREF33  the M-AILABS speech dataset BIBREF34
Follow the prior works BIBREF6, BIBREF7, BIBREF9, we adopt the BLEU and the Micro Entity F1 to evaluate our model performance.  We provide human evaluation on our framework and the compared models.  We hire several human experts and ask them to judge the quality of the responses according to correctness, fluency, and humanlikeness on a scale from 1 to 5. 
 Question: What were the evaluation metrics?
Output: BLEU Micro Entity F1 quality of the responses according to correctness, fluency, and humanlikeness on a scale from 1 to 5
We compare classification and regression approaches and show that classification produces better results than regression but the quality of the results depends on the approach followed to annotate the data labels. 
 Question: Did classification models perform better than previous regression one?
Output: Yes
The speech content was obtained from the closed captions (CC) provided by the YouTube ASR system which can be though as subtitles. 
 Question: What ASR system do they use?
Output: YouTube ASR system
Fig. FIGREF4 shows the distributions of seven question types grouped deterministically from the lexicons. 
 Question: How many question types do they find in the datasets analyzed?
Output: seven
Recently, MADAMIRA BIBREF6 system has been evaluated using a blind testset (25K words for Modern Standard Arabic (MSA) selected from Penn Arabic Tree bank (PATB)), and the reported accuracy was 96.2% as the percentage of words where the chosen analysis (provided by SAMA morphological analyzer BIBREF7 ) has the correct lemma. As MSA is usually written without diacritics and IR systems normally remove all diacritics from search queries and indexed data as a basic preprocessing step, so another column for undiacritized lemma is added and it's used for evaluating our lemmatizer and comparing with state-of-the-art system for lemmatization; MADAMIRA. 
 Question: What is the state of the art?
Output:  MADAMIRA BIBREF6 system
Twitter data: We used the Twitter API to scrap tweets with hashtags. All non-English tweets were filtered out by the API. 
 Question: Do they evaluate only on English datasets?
Output: Yes
For every input data point and possible target class, LRP delivers one scalar relevance value per input variable, hereby indicating whether the corresponding part of the input is contributing for or against a specific classifier decision, or if this input variable is rather uninvolved and irrelevant to the classification task at all. 
 Question: Does the LRP method work in settings that contextualize the words with respect to one another?
Output: Yes
Amazon Reviews Dataset BIBREF24 is a large dataset with millions of reviews from different product categories. For our experiments, we consider a subset of 20000 reviews from the domains Cell Phones and Accessories(C), Clothing and Shoes(S), Home and Kitchen(H) and Tools and Home Improvement(T). Out of 20000 reviews, 10000 are positive and 10000 are negative. We use 12800 reviews for training, 3200 reviews for validation and 4000 reviews for testing from each domain. 
 Question: For the purposes of this paper, how is something determined to be domain specific knowledge?
Output: reviews under distinct product categories are considered specific domain knowledge
Chowdhury BIBREF14 and Thomas et al. BIBREF11 proposed methods that use linguistic phenomenons and two-stage SVM to classify DDIs. FBK-irst BIBREF10 is a follow-on work which applies kernel method to the existing model and outperforms it. Neural network based approaches have been proposed by several works. Liu et al. BIBREF9 employ CNN for DDI extraction for the first time which outperforms the traditional machine learning based methods.  Sahu et al. BIBREF12 proposed LSTM based DDI extraction approach and outperforms CNN based approach, since LSTM handles sentence as a sequence instead of slide windows. 
 Question: What are the existing methods mentioned in the paper?
Output: Chowdhury BIBREF14 and Thomas et al. BIBREF11 FBK-irst BIBREF10 Liu et al. BIBREF9 Sahu et al. BIBREF12
Our data collection setup uses a dialogue simulator to generate dialogue outlines first and then paraphrase them to obtain natural utterances. 
 Question: Where is the dataset from?
Output: dialogue simulator
We conduct our annotation study on Amazon Mechanical Turk, presenting Turkers with Human Intelligence Tasks (henceforth, HITs) consisting of a single conversation between a customer and an agent. In each HIT, we present Turkers with a definition of each dialogue act, as well as a sample annotated dialogue for reference. For each turn in the conversation, we allow Turkers to select as many labels from our taxonomy as required to fully characterize the intent of the turn. Additionally, annotators are asked three questions at the end of each conversation HIT, to which they could respond that they agreed, disagreed, or could not tell: 
 Question: How are customer satisfaction, customer frustration and overall problem resolution data collected?
Output: By annotators on Amazon Mechanical Turk.
We started by answering always YES (in batch 2 and 3) to get the baseline performance. For batch 4 we used entailment. 
 Question: What was the baseline model?
Output: by answering always YES (in batch 2 and 3)
terms extracted from Yahoo! Answers tend to be more related, in terms of the number of correlated terms, to attributes related to religion or ethnicity compared to terms from Twitter. However, for two particular attributes (i.e., Price and Buddhist), the number of correlated terms from Twitter is higher than the ones from Yahoo! Answers .  
 Question: On Twitter, do the demographic attributes and answers show more correlations than on Yahoo! Answers?
Output: No
Results in Table TABREF64 show the correctness rates of these scenarios. User correctness score is superior to that of the baseline parser by 7.5% (from 37.1% to 44.6%), while the hybrid approach outscores both with a correctness of 48.7% improving the baseline by 11.6%. 
 Question: Which query explanation method was preffered by the users in terms of correctness?
Output: hybrid approach
When testing across languages, we report accuracy for two setups: average accuracy over each single-language model (Avg), and accuracy obtained when training on the concatenation of all languages but the target one (All). The latter setting is also used for the embeddings model. We report accuracy for all experiments. 
 Question: What are the evaluation metrics used?
Output: average accuracy over each single-language model (Avg), and accuracy obtained when training on the concatenation of all languages but the target one (All)
The two main data sets that we will use to evaluate our architecture are the 2014 i2b2 de-identification challenge data set BIBREF2 and the nursing notes corpus BIBREF3 . 
 Question: Which two datasets is the system tested on?
Output: 2014 i2b2 de-identification challenge data set BIBREF2 nursing notes corpus BIBREF3
First preference is given to the labels that are perfectly matching in all the neural annotators. In Table TABREF11, we can see that both datasets have about 40% of exactly matching labels over all models (AM). Then priority is given to the context-based models to check if the label in all context models is matching perfectly. In case two out of three context models are correct, then it is being checked if that label is also produced by at least one of the non-context models. Then, we allow labels to rely on these at least two context models. As a result, about 47% of the labels are taken based on the context models (CM). When we see that none of the context models is producing the same results, then we rank the labels with their respective confidence values produced as a probability distribution using the $softmax$ function. The labels are sorted in descending order according to confidence values. Then we check if the first three (case when one context model and both non-context models produce the same label) or at least two labels are matching, then we allow to pick that one. There are about 3% in IEMOCAP and 5% in MELD (BM).

Finally, when none the above conditions are fulfilled, we leave out the label with an unknown category. This unknown category of the dialogue act is labeled with `xx' in the final annotations, and they are about 7% in IEMOCAP and 11% in MELD (NM). 
 Question: How does the ensemble annotator extract the final label?
Output: First preference is given to the labels that are perfectly matching in all the neural annotators. In case two out of three context models are correct, then it is being checked if that label is also produced by at least one of the non-context models. When we see that none of the context models is producing the same results, then we rank the labels with their respective confidence values produced as a probability distribution using the $softmax$ function. The labels are sorted in descending order according to confidence values. Then we check if the first three (case when one context model and both non-context models produce the same label) or at least two labels are matching, then we allow to pick that one.  Finally, when none the above conditions are fulfilled, we leave out the label with an unknown category.
For our classification experiments, we use the MH17 Twitter dataset introduced by BIBREF4, a dataset collected in order to study the flow of (dis)information about the MH17 plane crash on Twitter. 
 Question: What dataset is used for this study?
Output: MH17 Twitter dataset
In this task English tweets from conversation threads, each associated to a newsworthy event and the rumours around it, are provided as data. The goal is to determine whether a tweet in the thread is supporting, denying, querying, or commenting the original rumour which started the conversation.  
 Question: Is this an English-language dataset?
Output: Yes
In the sixth international workshop on Vietnamese Language and Speech Processing (VLSP 2019), the Hate Speech Detection (HSD) task is proposed as one of the shared-tasks to handle the problem related to controlling content in SNSs. The fundamental idea of this system is how to make a system that has the diversity of viewing an input. That because of the variety of the meaning in Vietnamese language especially with the acronym, teen code type. 
 Question: Is the data all in Vietnamese?
Output: Yes
We compare our model with the following state-of-the-art joint entity and relation extraction models:

(1) SPTree BIBREF4: This is an end-to-end neural entity and relation extraction model using sequence LSTM and Tree LSTM. (2) Tagging BIBREF5: This is a neural sequence tagging model which jointly extracts the entities and relations using an LSTM encoder and an LSTM decoder. (3) CopyR BIBREF6: This model uses an encoder-decoder approach for joint extraction of entities and relations. (4) HRL BIBREF11: This model uses a reinforcement learning (RL) algorithm with two levels of hierarchy for tuple extraction. (5) GraphR BIBREF14: This model considers each token in a sentence as a node in a graph, and edges connecting the nodes as relations between them. (6) N-gram Attention BIBREF9: This model uses an encoder-decoder approach with N-gram attention mechanism for knowledge-base completion using distantly supervised data. 
 Question: What is previous work authors reffer to?
Output: SPTree Tagging CopyR HRL GraphR N-gram Attention
GPT-2 achieves the highest score and the $n$-gram the lowest. Transformer-XL and the LSTM LM perform in the middle, and at roughly the same level as each other. We report the 12-category accuracy results for all models and human evaluation in Table TABREF14. 
 Question: What is the performance of the models on the tasks?
Output: Overall accuracy per model is: 5-gram (60.5), LSTM (68.9), TXL (68.7), GPT-2 (80.1)
We follow a similar evaluation protocol to those presented in BIBREF6 , BIBREF8 , BIBREF9 which is to use our learned representations as features for a low complexity classifier (typically linear) on a novel supervised task/domain unseen during training without updating the parameters of our sentence representation model. We also consider such a transfer learning evaluation in an artificially constructed low-resource setting. In addition, we also evaluate the quality of our learned individual word representations using standard benchmarks BIBREF36 , BIBREF37 . 
 Question: How do they evaluate their sentence representations?
Output: standard benchmarks BIBREF36 , BIBREF37 to use our learned representations as features for a low complexity classifier (typically linear) on a novel supervised task/domain unseen during training without updating the parameters transfer learning evaluation in an artificially constructed low-resource setting
Our desiderata for the task collection were: sufficient diversity, existence of fairly large datasets for training, and success as standalone training objectives for sentence representations.

Multi-task training setup 
 Question: Which model architecture do they for sentence encoding?
Output: Answer with content missing: (Skip-thought vectors-Natural Language Inference paragraphs) The encoder for the current sentence and the decoders for the previous (STP) and next sentence (STN) are typically parameterized as separate RNNs
- RNN
The resulting sequence of vectors is encoded using an LSTM encoder.  
 Question: What architecture does the encoder have?
Output: LSTM
We compared our model with MLE, RL and GAN baselines. Since COCO and EMNLP2017 WMT don't have input while WeiboDial regards posts as input, we chose the following baselines respectively:

MLE: a RNN model trained with MLE objective BIBREF4 . Its extension, Seq2Seq, can work on the dialogue dataset BIBREF2 .

SeqGAN: The first text GAN model that updates the generator with policy gradient based on the rewards from the discriminator BIBREF7 .

LeakGAN: A variant of SeqGAN that provides rewards based on the leaked information of the discriminator for the generator BIBREF11 .

MaliGAN: A variant of SeqGAN that optimizes the generator with a normalized maximum likelihood objective BIBREF8 .

IRL: This inverse reinforcement learning method replaces the discriminator with a reward approximator to provide dense rewards BIBREF12 .

RAML: A RL approach to incorporate MLE objective into RL training framework, which regards BLEU as rewards BIBREF17 .

DialogGAN: An extension of SeqGAN tuned to dialogue generation task with MLE objective added to the adversarial objective BIBREF16 .

DPGAN: A variant of DialogGAN which uses a language model based discriminator and regards cross-entropy as rewards BIBREF13 . 
 Question: What GAN models were used as baselines to compare against?
Output: MLE SeqGAN LeakGAN MaliGAN IRL RAML DialogGAN DPGAN
We recruited 102 pairs of participants from Amazon Mechanical Turk and randomly assigned speaker and listener roles 
 Question: Was this experiment done in a lab?
Output: No
Following recent works, we compare on two widely used datasets, the Penn Treebank (PTB) BIBREF28 as prepared by BIBREF29 and WikiText2 (WT-2) BIBREF20 . 
 Question: what data did they use?
Output:  Penn Treebank WikiText2
To analyse the results we chose to use the test provided by BIBREF10, which consists of $19\,791$ analogies divided into 19 different categories: 6 related to the “semantic" macro-area (8915 analogies) and 13 to the “syntactic" one (10876 analogies). All the analogies are composed by two pairs of words that share a relation, schematized with the equation: $a:a^{*}=b:b^{*}$ (e.g. “man : woman = king : queen"); where $b^{*}$ is the word to be guessed (“queen"), $b$ is the word coupled to it (“king"), $a$ is the word for the components to be eliminated (“man"), and $a^{*}$ is the word for the components to be added (“woman"). 
 Question: Are the word embeddings tested on a NLP task?
Output: Yes
Topology only embeddings: MMB BIBREF45 , DeepWalk BIBREF1 , LINE BIBREF33 , Node2vec BIBREF46 . ( INLINEFORM1 ) Joint embedding of topology & text: Naive combination, TADW BIBREF5 , CENE BIBREF6 , CANE BIBREF9 , WANE BIBREF10 , DMTE BIBREF34 .  
 Question: Which other embeddings do they compare against?
Output: MMB DeepWalk LINE  Node2vec TADW CENE CANE WANE DMTE
Moreover, due to their noisy nature, they are also processed using some Twitter-specific techniques such as substitution/removal of URLs, of user mentions, of hashtags, and of emoticons, spelling correction, elongation normalization, abbreviation lookup, punctuation removal, detection of amplifiers and diminishers, negation scope detection, etc. That language proved to be quite challenging with its use of creative spelling and punctuation, misspellings, slang, new words, URLs, and genre-specific terminology and abbreviations, e.g., RT for re-tweet and #hashtags. In addition to the genre difference, there is also a difference in length: social media messages are generally short, often length-limited by design as in Twitter, i.e., a sentence or a headline rather than a full document 
 Question: What difficulties does sentiment analysis on Twitter have, compared to sentiment analysis in other domains?
Output: Tweets noisy nature, use of creative spelling and punctuation, misspellings, slang, new words, URLs, and genre-specific terminology and abbreviations, short (length limited) text
The second series of tests consisted of using all the documents of the subcorpus “specialist” $E$, because the documents of the subcorpus of Annodis are not identical.  As far as system evaluations are concerned, we use the 78 $E$ documents as reference.  We calculate the precision $P$, the recall $R$ and the $F$-score on the text corpus used in our tests, as follow: 
 Question: How is segmentation quality evaluated?
Output: Segmentation quality is evaluated by calculating the precision, recall, and F-score of the automatic segmentations in comparison to the segmentations made by expert annotators from the ANNODIS subcorpus.
We draw from research in social psychology to inform our methodology, most prominently Moral Foundations Theory BIBREF26. MFT seeks to explain the structure and variation of human morality across cultures, and proposes five moral foundations: Care / Harm, Fairness / Cheating, Loyalty / Betrayal, Authority / Subversion, and Sanctity / Degradation. Each foundation is summarized by a positive and a negative pole, resulting in ten fine-grained moral categories. 
 Question: Which fine-grained moral dimension examples do they showcase?
Output: Care / Harm, Fairness / Cheating, Loyalty / Betrayal, Authority / Subversion, and Sanctity / Degradation
We started from an initial corpus that did not have revisions, and kept adding new documents and revising existing documents. The revision operations we used were deletion, addition and replacement of words, sentences, paragraphs, section names and document titles. The addition of words, ..., section names, and new documents were pulled from the Wikipedia abstracts. We generated six data sets using different random seeds, and each data set contained six corpora (Corpus 0 - 5). 
 Question: What are simulated datasets collected?
Output: There are 6 simulated datasets collected which is initialised with a corpus of size 550 and simulated by generating new documents from Wikipedia extracts and replacing existing documents
For the language modeling evaluation, we also evaluate a baseline without knowledge distillation (termed NoKD), with a model parameterized identically to the distilled student models but trained directly on the teacher model objective from scratch. For downstream tasks, we compare with NoKD as well as Patient Knowledge Distillation (PKD) from BIBREF34, who distill the 12-layer BERTBASE model into 3 and 6-layer BERT models by using the teacher model's hidden states. 
 Question: What state-of-the-art compression techniques were used in the comparison?
Output: baseline without knowledge distillation (termed NoKD) Patient Knowledge Distillation (PKD)
Once the S-V-O is generated, Text2Visual provides users with visual components that convey the S-V-O text meanings. 
 Question: Does their solution involve connecting images and text?
Output: Yes
Table TABREF21 shows results for Korean INLINEFORM0 English, using the same configurations (1, 2 and 8) as for German–English. Our results confirm that the techniques we apply are successful across datasets, and result in stronger systems than previously reported on this dataset, achieving 10.37 BLEU as compared to 5.97 BLEU reported by gu-EtAl:2018:EMNLP1. 
 Question: what were their experimental results in the low-resource dataset?
Output: 10.37 BLEU
Various multilingual extensions of NMT have already been proposed in the literature. The authors of BIBREF18 , BIBREF19 apply multitask learning to train models for multiple languages. Zoph and Knight BIBREF20 propose a multi-source model and BIBREF21 introduces a character-level encoder that is shared across several source languages. In our setup, we will follow the main idea proposed by Johnson et al. BIBREF22 . The authors of that paper suggest a simple addition by means of a language flag on the source language side (see Figure 2 ) to indicate the target language that needs to be produced by the decoder. This flag will be mapped on a dense vector representation and can be used to trigger the generation of the selected language. The authors of the paper argue that the model enables transfer learning and supports the translation between languages that are not explicitly available in training. 
 Question: What neural machine translation models can learn in terms of transfer learning?
Output: Multilingual Neural Machine Translation Models