niryuu/sni-each-converted huggingface dataset:
Advanced neural architectures based on character input (CNNs, BPE, etc) are supposed to be able to learn how to handle spelling and morphology variations themselves, even for languages with rich morphology: `just add more layers!'. Contextualised embedding models follow this tradition: as a rule, they are trained on raw text collections, with minimal linguistic pre-processing. Below, we show that this is not entirely true. 
 Question: Why is lemmatization not necessary in English?
Output: Abstractive
We asked medical doctors experienced in extracting knowledge related to medical entities from texts to annotate the entities described above. Initially, we asked four annotators to test our guidelines on two texts. Subsequently, identified issues were discussed and resolved. Following this pilot annotation phase, we asked two different annotators to annotate two case reports according to our guidelines. The same annotators annotated an overall collection of 53 case reports. The annotation was performed using WebAnno BIBREF7, a web-based tool for linguistic annotation. The annotators could choose between a pre-annotated version or a blank version of each text. The pre-annotated versions contained suggested entity spans based on string matches from lists of conditions and findings synonym lists. 
 Question: How was annotation performed?
Output: Abstractive
We would expect that explanation performance should correlate with prediction performance. Since Possible-answer knowledge is primarily needed to decide if the net has enough information to answer the challenge question without guessing and relevant-variable knowledge is needed for the net to know what to query, we analyzed the network's performance on querying and answering separately. The memory network has particular difficulty learning to query relevant variables, reaching only about .5 accuracy when querying. At the same time, it learns to answer very well, reaching over .9 accuracy there. Since these two parts of the interaction are what we ask it to explain in the two modes, we find that the quality of the explanations strongly correlates with the quality of the algorithm executed by the network. 
 Question: How do they measure correlation between the prediction and explanation quality?
Output: Abstractive
We used recordings of nurse-initiated telephone conversations for congestive heart failure patients undergoing telemonitoring, post-discharge from the hospital. To analyze the linguistic structure of the inquiry-response pairs in the entire 41-hour dataset, we randomly sampled a seed dataset consisting of 1,200 turns and manually categorized them to different types, which are summarized in Table TABREF14 along with the corresponding occurrence frequency statistics. 
 Question: Which data do they use as a starting point for the dialogue dataset?
Output: Abstractive
Twitter provides well-documented API, which allows to request any information about Tweets, users and their profiles, with respect to rate limits. There is special type of API, called Streaming API, that provides a real-time stream of tweets. The key difference with regular API is that connection is kept alive as long as possible, and Tweets are sent in real-time to the client. There are three endpoints of Streaming API of our interest: “sample”, “filter” and “firehose”. The first one provides a sample (random subset) of the full Tweet stream. The second one allows to receive Tweets matching some search criteria: matching to one or more search keywords, produced by subset of users, or coming from certain geo location. The last one provides the full set of Tweets, although it is not available by default. In order to get Twitter “firehose” one can contact Twitter, or buy this stream from third-parties.

In our case the simplest approach would be to use “sample” endpoint, but it provides Tweets in all possible languages from all over the World, while we are concerned only about one language (Russian). In order to use this endpoint we implemented filtering based on language. The filter is simple: if Tweet does not contain a substring of 3 or more cyrillic symbols, it is considered non-Russian. Although this approach keeps Tweets in Mongolian, Ukrainian and other slavic languages (because they use cyrillic alphabet), the total amount of false-positives in this case is negligible. To demonstrate this we conducted simple experiment: on a random sample of 200 tweets only 5 were in a language different from Russian. In order not to rely on Twitter language detection, we chose to proceed with this method of language-based filtering.

However, the amount of Tweets received through “sample” endpoint was not satisfying. This is probably because “sample” endpoint always streams the same content to all its clients, and small portion of it comes in Russian language. In order to force mining of Tweets in Russian language, we chose "filter" endpoint, which requires some search query. We constructed heuristic query, containing some auxiliary words, specific to Russian language: conjunctions, pronouns, prepositions. The full list is as follows:

russian я, у, к, в, по, на, ты, мы, до, на, она, он, и, да. 
 Question: Which Twitter corpus was used to train the word vectors?
Output: Abstractive
We then proceed to connect these mentions i) if they co-occur within the same document (we will refer to this as DOC-BASED edges), ii) if the pair of named entity mentions is identical (MATCH edges—these may connect nodes across and within documents), or iii) if they are in the same coreference chain, as predicted by the external coreference system (COREF edges). 
 Question: Did they use a relation extraction method to construct the edges in the graph?
Output: Yes-no
Last, we evaluate our approaches in 9 commonly used text classification datasets. We evaluate our methods on several commonly used datasets whose themes range from sentiment, web-page, science to medical and healthcare. 
 Question: What NLP tasks do they consider?
Output: Abstractive
Based on annotated corpora and token-based features, studies used machine learning approaches to build word segmentation systems with accuracy about 94%-97%. 
 Question: How successful are the approaches used to solve word segmentation in Vietnamese?
Output: Abstractive
For a fair comparison with previous work, we train the baseline MT system on the data released by BIBREF11.  
 Question: what was the baseline?
Output: Extractive
In Table TABREF1, we summarize the quantitative results of the above previous studies. It includes the model basis, feature description, languages classified and the used dataset along with accuracy obtained. The table also lists the overall results of our proposed models (at the top). The languages used by various authors along with their acronyms are English (En), Spanish (Es), French (Fr), German (De), Russian (Ru), Italian (It), Bengali (Ben), Hindi (Hi) and Telegu (Tel). 
 Question: Is the performance compared against a baseline model?
Output: Yes-no
In this section, we describe our data collection for these 3 components. We first describe a series of pilot studies that we conducted in order to collect commonsense inference questions (Section SECREF4 ). In Section SECREF5 , we discuss the resulting data collection of questions, texts and answers via crowdsourcing on Amazon Mechanical Turk (henceforth MTurk). Section SECREF17 gives information about some necessary postprocessing steps and the dataset validation.  
 Question: how was the data collected?
Output: Abstractive
Thus we consider the task of answer generation for TweetQA and we use several standard metrics for natural language generation to evaluate QA systems on our dataset, namely we consider BLEU-1 BIBREF16 , Meteor BIBREF17 and Rouge-L BIBREF18 in this paper. 
 Question: What evaluation metrics do they use?
Output: Extractive
Outlier Detection. The Outlier Detection task BIBREF0 is to determine which word in a list INLINEFORM0 of INLINEFORM1 words is unrelated to the other INLINEFORM2 which were chosen to be related. For each INLINEFORM3 , one can compute its compactness score INLINEFORM4 , which is the compactness of INLINEFORM5 . Sentiment analysis. We also consider sentiment analysis as described by BIBREF31 . 
 Question: Do they test their word embeddings on downstream tasks?
Output: Yes-no
Recently bhat-EtAl:2017:EACLshort provided a CS dataset for the evaluation of their parsing models which they trained on the Hindi and English Universal Dependency (UD) treebanks. We extend this dataset by annotating 1,448 more sentences. 
 Question: How big is the provided treebank?
Output: Abstractive
Recommendation diversity. As defined in BIBREF18 , we calculate recommendation diversity as the average dissimilarity of all pairs of tags in the list of recommended tags. 
 Question: how is diversity measured?
Output: Extractive
On another note, we apply our formalization to evaluate multilingual $\textsc {bert} $'s syntax knowledge on a set of six typologically diverse languages. Although it does encode a large amount of information about syntax (more than $81\%$ in all languages), it only encodes at most $5\%$ more information than some trivial baseline knowledge (a type-level representation). This indicates that the task of POS labeling (word-level POS tagging) is not an ideal task for contemplating the syntactic understanding of contextual word embeddings. We see that—in all analysed languages—type level embeddings can already capture most of the uncertainty in POS tagging. We also see that BERT only shares a small amount of extra information with the task, having small (or even negative) gains in all languages. Finally, when put into perspective, multilingual $\textsc {bert} $'s representations do not seem to encode much more information about syntax than a trivial baseline. $\textsc {bert} $ only improves upon fastText in three of the six analysed languages—and even in those, it encodes at most (in English) $5\%$ additional information. 
 Question: Was any variation in results observed based on language typology?
Output: Abstractive
In Figure FIGREF28, we show some examples of the annotation results in CORD-19-NER. We can see that our distantly- or weakly supervised methods achieve high quality recognizing the new entity types, requiring only several seed examples as the input. For example, we recognized “SARS-CoV-2" as the “CORONAVIRUS" type, “bat" and “pangolins" as the “WILDLIFE" type and “Van der Waals forces" as the “PHYSICAL_SCIENCE" type. This NER annotation results help downstream text mining tasks in discovering the origin and the physical nature of the virus. Our NER methods are domain-independent that can be applied to corpus in different domains. In addition, we show another example of NER annotation on New York Times with our system in Figure FIGREF29.

In Figure FIGREF30, we show the comparison of our annotation results with existing NER/BioNER systems. In Figure FIGREF30, we can see that only our method can identify “SARS-CoV-2" as a coronavirus. In Figure FIGREF30, we can see that our method can identify many more entities such as “pylogenetic" as a evolution term and “bat" as a wildlife. In Figure FIGREF30, we can also see that our method can identify many more entities such as “racism" as a social behavior. In summary, our distantly- and weakly-supervised NER methods are reliable for high-quality entity recognition without requiring human effort for training data annotation. 
 Question: Did they experiment with the dataset?
Output: Yes-no
Both the audio and the transcript are de-identified (by removing the identifying information) with digital zeros and [de-identified] tags, respectively. 
 Question: Is the data de-identified?
Output: Yes-no
In this section we present two datasets used in our experiments: The NowThisNews dataset, collected for the purpose of this paper, and The BreakingNews dataset BIBREF4 , publicly available dataset of news articles.

contains 4090 posts with associated videos from NowThisNews Facebook page collected between 07/2015 and 07/2016. For each post we collected its title and the number of views of the corresponding video, which we consider our popularity metric. Due to a fairly lengthy data collection process, we decided to normalize our data by first grouping posts according to their publication month and then labeling the posts for which the popularity metric exceeds the median monthly value as popular, the remaining part as unpopular. 
 Question: Where do they obtain the news videos from?
Output: Extractive
To evaluate the influence of our hypersphere feature for off-the-shelf NER systems, we perform the NE recognition on two standard NER benchmark datasets, CoNLL2003 and ONTONOTES 5.0. 
 Question: Do they evaluate on NER data sets?
Output: Yes-no
The first step is to pre-train the model on one MCQA dataset referred to as the source task, which usually contains abundant training data. The second step is to fine-tune the same model on the other MCQA dataset, which is referred to as the target task, that we actually care about, but that usually contains much less training data.  
 Question: How different is the dataset size of source and target?
Output: Abstractive
The only requirement is that the model accepts as input, an embedding layer (for entities and relations). If a model fulfills this requirement (which a large number of neural models on knowledge graphs do), we can just use Dolores embeddings as a drop-in replacement. We just initialize the corresponding embedding layer with Dolores embeddings. 
 Question: Is fine-tuning required to incorporate these embeddings into existing models?
Output: Yes-no
An emerging body of work in natural language processing and computational social science has investigated how NLP systems can detect moral sentiment in online text. For example, moral rhetoric in social media and political discourse BIBREF19, BIBREF20, BIBREF21, the relation between moralization in social media and violent protests BIBREF22, and bias toward refugees in talk radio shows BIBREF23 have been some of the topics explored in this line of inquiry. In contrast to this line of research, the development of a formal framework for moral sentiment change is still under-explored, with no existing systematic and formal treatment of this topic BIBREF16. 
 Question: Does the paper discuss previous models which have been applied to the same task?
Output: Yes-no
The dataset at HASOC 2019 were given in three languages: Hindi, English, and German. Dataset in Hindi and English had three subtasks each, while German had only two subtasks. We participated in all the tasks provided by the organisers and decided to develop a single model that would be language agnostic. We used the same model architecture for all the three languages. The training dataset of Hindi dataset was more balanced than English or German dataset. Hence, the results were around 0.78. As the dataset in German language was highly imbalanced, the results drops to 0.62. 
 Question: What are the languages used to test the model?
Output: Abstractive
It is interesting to observe that the baseline model amplifies the bias in the training data set as measured by INLINEFORM0 and INLINEFORM1 . From measurements using the described bias metrics, our method effectively mitigates bias in language modelling without a significant increase in perplexity. 
 Question: how is mitigation of gender bias evaluated?
Output: Abstractive
We compare with the following baselines:

(1) Naive: A non-domain-adaptive baseline with bag-of-words representations and SVM classifier trained on the source domain.

(2) mSDA BIBREF7 : This is the state-of-the-art method based on discrete input features. Top 1000 bag-of-words features are kept as pivot features. We set the number of stacked layers to 3 and the corruption probability to 0.5.

(3) NaiveNN: This is a non-domain-adaptive CNN trained on source domain, which is a variant of our model by setting INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 to zeros.

(4) AuxNN BIBREF4 : This is a neural model that exploits auxiliary tasks, which has achieved state-of-the-art results on cross-domain sentiment classification. The sentence encoder used in this model is the same as ours.

(5) ADAN BIBREF16 : This method exploits adversarial training to reduce representation difference between domains. The original paper uses a simple feedforward network as encoder. For fair comparison, we replace it with our CNN-based encoder. We train 5 iterations on the discriminator per iteration on the encoder and sentiment classifier as suggested in their paper.

(6) MMD: MMD has been widely used for minimizing domain discrepancy on images. In those works BIBREF9 , BIBREF13 , variants of deep CNNs are used for encoding images and the MMDs of multiple layers are jointly minimized. In NLP, adding more layers of CNNs may not be very helpful and thus those models from image-related tasks can not be directly applied to our problem. To compare with MMD-based method, we train a model that jointly minimize the classification loss INLINEFORM0 on the source domain and MMD between INLINEFORM1 and INLINEFORM2 . For computing MMD, we use a Gaussian RBF which is a common choice for characteristic kernel. 
 Question: What are the baseline methods?
Output: Extractive
To achieve this, a method is proposed that transforms the goal-labels of the used dataset (DSTC2) into labels whose behaviour can be replicated during deployment. 
 Question: what corpus is used to learn behavior?
Output: Extractive
In the sentiment analysis experiments, we found that a tweet may contain multiple sentiments. The groundtruth labels contain 210 positive sentiments, 521 neutral sentiments, and 305 negative sentiments to the candidates. 
 Question: Is datasets for sentiment analysis balanced?
Output: Yes-no
As previous systems collect relevant data from knowledge bases after observing questions during evaluation BIBREF24 , BIBREF25 , we also explore using this option. Namely, we build a customized text corpus based on questions in commonsense reasoning tasks.  
 Question: Do they fine-tune their model on the end task?
Output: Yes-no
n this paper we provide a novel real-time and adaptive cryptocurrency price prediction platform based on Twitter sentiments. The integrative and modular platform copes with the three aforementioned challenges in several ways. Firstly, it provides a Spark-based architecture which handles the large volume of incoming data in a persistent and fault tolerant way. Secondly, the proposed platform offers an approach that supports sentiment analysis based on VADER which can respond to large amounts of natural language processing queries in real time. Thirdly, the platform supports a predictive approach based on online learning in which a machine learning model adapts its weights to cope with new prices and sentiments. Finally, the platform is modular and integrative in the sense that it combines these different solutions to provide novel real-time tool support for bitcoin price prediction that is more scalable, data-rich, and proactive, and can help accelerate decision-making, uncover new opportunities and provide more timely insights based on the available and ever-larger financial data volume and variety. 
 Question: Which elements of the platform are modular?
Output: Abstractive
We define a metric called the Semantic Text Exchange Score (STES) that evaluates the overall ability of a model to perform STE, and an adjustable parameter masking (replacement) rate threshold (MRT/RRT) that can be used to control the amount of semantic change. 
 Question: Has STES been previously used in the literature to evaluate similar tasks?
Output: Yes-no
We propose extended middle context, a new context representation for CNNs for relation classification. The contexts are split into three disjoint regions based on the two relation arguments: the left context, the middle context and the right context.  Hence, we propose to use two contexts: (1) a combination of the left context, the left entity and the middle context; and (2) a combination of the middle context, the right entity and the right context. The two contexts are processed by two independent convolutional and max-pooling layers. After pooling, the results are concatenated to form the sentence representation. 
 Question: How do they obtain the new context represetation?
Output: Abstractive
To identify legally sound answers, we recruit seven experts with legal training to construct answers to Turker questions.  
 Question: Who were the experts used for annotation?
Output: Abstractive
In this study, we investigate the performance of our proposed models on WSJ BIBREF5 .  
 Question: Which dataset do they use?
Output: Extractive
Thus, for this work, we build upon Hybrid Code Networks (HCN) BIBREF4 since HCNs achieve state-of-the-art performance in a data-efficient way for task-oriented dialogs, and propose AE-HCNs which extend HCNs with an autoencoder (Figure FIGREF8 ).  AE-HCN(-CNN) outperforms HCN on Test-OOD by a large margin about 17(20) points on average while keeping the minimum performance trade-off compared to Test.  
 Question: By how much does their method outperform state-of-the-art OOD detection?
Output: Abstractive
For training and testing we use the train-validation-test split of WikiTableQuestions BIBREF0 , a dataset containing 22,033 pairs of questions and answers based on 2,108 Wikipedia tables. This dataset is also used by our baselines, BIBREF0 , BIBREF3 . 
 Question: Does the dataset they use differ from the one used by Pasupat and Liang, 2015?
Output: Yes-no
The above pseudocode is agnostic with respect to the choice of fragmentation and environment functions; task-specific choices are described in more detail for each experiment below.

Discussion 
 Question: Which languages do they test on?
Output: Abstractive
Contributors record voice clips by reading from a bank of donated sentences. 
 Question: What domains are covered in the corpus?
Output: Abstractive
We compare our approach to two other, the first one uses almost the same tweet data as we use for training, and the second one is the CrowdFlower dataset annotated for emotions. In the first one Wang et al. BIBREF21 downloaded over 5M tweets which included one of 131 emotional hashtags based on Parrott's three-level categorization of emotions in seven categories: joy, sadness, anger, love, fear, thankfulness, surprise.  
 Question: What data is used in experiments?
Output: Extractive
We make copies of the monolingual model for each language and add additional crosslingual latent variables (CLVs) to couple the monolingual models, capturing crosslingual semantic role patterns. Concretely, when training on parallel sentences, whenever the head words of the arguments are aligned, we add a CLV as a parent of the two corresponding role variables. 
 Question: Do they add one latent variable for each language pair in their Bayesian model?
Output: Yes-no
For the 2019 Workshop on Neural Generation of Text (WNGT) Efficiency shared task BIBREF0, the Notre Dame Natural Language Processing (NDNLP) group looked at a method of inducing sparsity in parameters called auto-sizing in order to reduce the number of parameters in the Transformer at the cost of a relatively minimal drop in performance. 
 Question: What is WNGT 2019 shared task?
Output: Abstractive
We manually investigate the errors made by artificial neural networks for morphological inflection in a target language after pretraining on different source languages. For our qualitative analysis, we make use of the validation set. Therefore, we show validation set accuracies in Table TABREF19 for comparison. We manually annotate the outputs for the first 75 development examples for each source-target language combination. All found errors are categorized as belonging to one of the following categories. 
 Question: How is the performance on the task evaluated?
Output: Abstractive
We are using the dataset of the competition, which includes text from tweets having the aforementioned categories.  
 Question: What dataset is used for this work?
Output: Abstractive
Methods ::: Combining the two methods
We further propose to use the two methods together to combine their strengths. In fact, while the length token acts as a soft constraint to bias NMT to produce short or long translation with respect to the source, actually no length information is given to the network. On the other side, length encoding leverages information about the target length, but it is agnostic of the source length. 
 Question: Do they experiment with combining both methods?
Output: Yes-no
Macaw also supports Wizard of Oz studies or intermediary-based information seeking studies. The architecture of Macaw for such setup is presented in FIGREF16. As shown in the figure, the seeker interacts with a real conversational interface that supports multi-modal and mixed-initiative interactions in multiple devices. The intermediary (or the wizard) receives the seeker's message and performs different information seeking actions with Macaw. All seeker-intermediary and intermediary-system interactions will be logged for further analysis. 
 Question: What is a wizard of oz setup?
Output: Extractive
This results in three additional vectors corresponding to INLINEFORM3 , INLINEFORM4 and INLINEFORM5 difference vectors.

Results 
 Question: How are the EAU text spans annotated?
Output: Abstractive
In order to evaluate the models trained on generated data, we manually annotated a named entities dataset comprising 53453 tokens and 2566 sentences selected from over 250 news texts from ilur.am. 
 Question: what is the source of the news sentences?
Output: Extractive
We performed the annotation with freely available tools for the Portuguese language. 
 Question: Are the annotations automatic or manually created?
Output: Abstractive
They can answer each question with either `yes', `rather yes', `rather no', or `no'. They can supplement each answer with a comment of at most 500 characters. 
 Question: What annotations are present in dataset?
Output: Extractive
In this experiment set, our BERT implementation is compared to several systems that participated in the MEDDOCAN challenge: a CRF classifier BIBREF18, a spaCy entity recogniser BIBREF18, and NLNDE BIBREF12, the winner of the shared task and current state of the art for sensitive information detection and classification in Spanish clinical text.  However, attending to the obtained results, BERT remains only 0.3 F1-score points behind, and would have achieved the second position among all the MEDDOCAN shared task competitors. Taking into account that only 3% of the gold labels remain incorrectly annotated, the task can be considered almost solved, and it is not clear if the differences among the systems are actually significant, or whether they stem from minor variations in initialisation or a long-tail of minor labelling inconsistencies. 
 Question: Does BERT reach the best performance among all the algorithms compared?
Output: Yes-no
We approached the first and second challenges by using a Bayesian approach to learn which terms were associated with events, regardless of whether they are standard language, acronyms, or even a made-up word, so long as they match the events of interest. The third and fourth challenges are approached by using word-pairs, where we extract all the pairs of co-occurring words within each tweet. To find the words most associated with events, we search for the words that achieve the highest number of spikes matching the days of events.  
 Question: How are the keywords associated with events such as protests selected?
Output: Abstractive
We used the following affective resources relying on different emotion models.

Emolex: it contains 14,182 words associated with eight primary emotion based on the Plutchik model BIBREF10 , BIBREF11 .

EmoSenticNet(EmoSN): it is an enriched version of SenticNet BIBREF12 including 13,189 words labeled by six Ekman's basic emotion BIBREF13 , BIBREF14 .

Dictionary of Affect in Language (DAL): includes 8,742 English words labeled by three scores representing three dimensions: Pleasantness, Activation and Imagery BIBREF15 .

Affective Norms for English Words (ANEW): consists of 1,034 English words BIBREF16 rated with ratings based on the Valence-Arousal-Dominance (VAD) model BIBREF17 .

Linguistic Inquiry and Word Count (LIWC): this psycholinguistic resource BIBREF18 includes 4,500 words distributed into 64 emotional categories including positive (PosEMO) and negative (NegEMO). 
 Question: What affective-based features are used?
Output: Abstractive
Span detector. We adopt a multi-turn answer module for the span detector BIBREF1 . Formally, at time step INLINEFORM0 in the range of INLINEFORM1 , the state is defined by INLINEFORM2 . The initial state INLINEFORM3 is the summary of the INLINEFORM4 : INLINEFORM5 , where INLINEFORM6 . Here, INLINEFORM7 is computed from the previous state INLINEFORM8 and memory INLINEFORM9 : INLINEFORM10 and INLINEFORM11 . Finally, a bilinear function is used to find the begin and end point of answer spans at each reasoning step INLINEFORM12 : DISPLAYFORM0 DISPLAYFORM1

The final prediction is the average of each time step: INLINEFORM0 . We randomly apply dropout on the step level in each time step during training, as done in BIBREF1 . 
 Question: What is the architecture of the span detector?
Output: Extractive
The corpus consists of 53 documents, which contain an average number of 156.1 sentences per document, each with 19.55 tokens on average. The corpus comprises 8,275 sentences and 167,739 words in total. 
 Question: How large is the corpus?
Output: Extractive
jiant provides support for cutting-edge sentence encoder models, including support for Huggingface's Transformers. Supported models include: BERT BIBREF17, RoBERTa BIBREF27, XLNet BIBREF28, XLM BIBREF29, GPT BIBREF30, GPT-2 BIBREF31, ALBERT BIBREF32 and ELMo BIBREF33. jiant also supports the from-scratch training of (bidirectional) LSTMs BIBREF34 and deep bag of words models BIBREF35, as well as syntax-aware models such as PRPN BIBREF36 and ON-LSTM BIBREF37. jiant also supports word embeddings such as GloVe BIBREF38. 
 Question: Is jiant compatible with models in any programming language?
Output: Yes-no
Table TABREF34 shows the inference speed of our method when implementing the sequnece modeling layer with the LSTM-based, CNN-based, and Transformer-based architecture, respectively.  From the table, we can see that our method has a much faster inference speed than Lattice-LSTM when using the LSTM-based sequence modeling layer, and it was also much faster than LR-CNN, which used an CNN architecture to implement the sequence modeling layer. And as expected, our method with the CNN-based sequence modeling layer showed some advantage in inference speed than those with the LSTM-based and Transformer-based sequence model layer. 
 Question:  What percentage of improvement in inference speed is obtained by the proposed method over the newest state-of-the-art methods?
Output: Abstractive
In recent years, there has been a rapid growth in the usage of social media. People post their day-to-day happenings on regular basis. BIBREF0 propose four tasks for detecting drug names, classifying medication intake, classifying adverse drug reaction and detecting vaccination behavior from tweets. We participated in the Task2 and Task4. 
 Question: Was the system only evaluated over the second shared task?
Output: Yes-no
Data was collected from a 10% uniform sample of Twitter posts made during 2013, specifically the Gardenhose API. Control documents were also selected. These documents did not contain any of `caused', `causing', or `causes', nor any bidirectional words, and are further matched temporally to obtain the same number of control documents as causal documents in each fifteen-minute period during 2013. Control documents were otherwise selected randomly; causal synonyms may be present. 
 Question: how do they collect the comparable corpus?
Output: Abstractive
It also needs to be noted that due to the issues in actual practice, the objective of this work is to generate candidate prescriptions to facilitate the prescribing procedure instead of substituting the human practitioners completely. 
 Question: Why did they think this was a good idea?
Output: Abstractive
A quantitative measure is required to reliably evaluate the achieved improvement. One of the methods proposed to measure the interpretability is the word intrusion test BIBREF41 . But, this method is expensive to apply since it requires evaluations from multiple human evaluators for each embedding dimension. In this study, we use a semantic category-based approach based on the method and category dataset (SEMCAT) introduced in BIBREF27 to quantify interpretability. Specifically, we apply a modified version of the approach presented in BIBREF40 in order to consider possible sub-groupings within the categories in SEMCAT.  
 Question: What experiments do they use to quantify the extent of interpretability?
Output: Abstractive
For MSA, we acquired the diacritized corpus that was used to train the RDI BIBREF7 diacritizer and the Farasa diacritizer BIBREF31. The corpus contains 9.7M tokens with approximately 194K unique surface forms (excluding numbers and punctuation marks). For testing, we used the freely available WikiNews test set BIBREF31, which is composed of 70 MSA WikiNews articles (18,300 tokens) and evenly covers a variety of genres including politics, economics, health, science and technology, sports, arts and culture. For CA, we obtained a large collection of fully diacritized classical texts (2.7M tokens) from a book publisher, and we held-out a small subset of 5,000 sentences (approximately 400k words) for testing. 
 Question: what datasets were used?
Output: Extractive
Consequently, our architecture is based on Embeddings from Language Model or ELMo BIBREF10 Concretely, we use a pre-trained ELMo model, obtained using the 1 Billion Word Benchmark which contains about 800M tokens of news crawl data from WMT 2011 BIBREF24 . Subsequently, the contextualized embeddings are passed on to a BiLSTM with 2,048 hidden units. We aggregate the LSTM hidden states using max-pooling, which in our preliminary experiments offered us better results, and feed the resulting vector to a 2-layer feed-forward network, where each layer has 512 units. The output of this is then fed to the final layer of the model, which performs the binary classification. 
 Question: What type of model are the ELMo representations used in?
Output: Abstractive
To study the lexical and semantic diversities of responses, we performed three analyses. First, we aggregated all worker responses to a particular question into a single list corresponding to that question. Second, we compared the diversity of individual responses between Control and AUI for each question. To measure diversity for a question, we computed the number of responses divided by the number of unique responses to that question. We call this the response density.  
 Question: How was lexical diversity measured?
Output: Abstractive
The resulting dataset consists of 22,880 users, 41,094 blogs, and 561,003 posts. Table TABREF2 presents additional statistics of our dataset. 
 Question: How many users do they look at?
Output: Extractive
We use a siamese neural network, shown to perform state-of-the-art few-shot learning BIBREF11, as our baseline model. We modify the original to account for sequential data, with each twin composed of an embedding layer, a Long-Short Term Memory (LSTM) BIBREF12 layer, and a feed-forward layer with Rectified Linear Unit (ReLU) activations. 
 Question: What were the baselines?
Output: Abstractive
The total F-1 score on the OntoNotes dataset is 88%, and the total F-1 cross-validation score on the 112 class Wiki(gold) dataset is 53%.  
 Question: What results do they achieve using their proposed approach?
Output: Abstractive
As a document classifier we employ a word-based CNN similar to Kim consisting of the following sequence of layers: $ \texttt {Conv} \xrightarrow{} \texttt {ReLU} \xrightarrow{} \texttt {1-Max-Pool} \xrightarrow{} \texttt {FC} \\ $ Future work would include applying LRP to other neural network architectures (e.g. character-based or recurrent models) on further NLP tasks, as well as exploring how relevance information could be taken into account to improve the classifier's training procedure or prediction performance. 
 Question: Do the experiments explore how various architectures and layers contribute towards certain decisions?
Output: Yes-no
In test batch 4, our system (called FACTOIDS) achieved highest recall score of ‘0.7033’ but low precision of 0.1119, leaving open the question of how could we have better balanced the two measures. 
 Question: What was their highest recall score?
Output: Extractive
Baseline Results ::: Speech Synthesis
In our previous work on building speech systems on found data in 700 languages, BIBREF7, we addressed alignment issues (when audio is not segmented into turn/sentence sized chunks) and correctness issues (when the audio does not match the transcription). We used the same techniques here, as described above.

For the best quality speech synthesis we need a few hours of phonetically-balanced, single-speaker, read speech. Our first step was to use the start and end points for each turn in the dialogues, and select those of the most frequent speaker, nmlch. This gave us around 18250 segments. We further automatically removed excessive silence from the start, middle and end of these turns (based on occurrence of F0). This gave us 13 hours and 48 minutes of speech.

We phonetically aligned this data and built a speech clustergen statistical speech synthesizer BIBREF9 from all of this data. For speech recognition (ASR) we used Kaldi BIBREF11. We built neural end-to-end machine translation systems between Mapudungun and Spanish in both directions, using state-of-the-art Transformer architecture BIBREF14 with the toolkit of BIBREF15. 
 Question: What are the models used for the baseline of the three NLP tasks?
Output: Extractive
SVM: We define 3 sets of features to characterize each question. The first is a simple bag-of-words set of features over the question (SVM-BOW), the second is bag-of-words features of the question as well as length of the question in words (SVM-BOW + LEN), and lastly we extract bag-of-words features, length of the question in words as well as part-of-speech tags for the question (SVM-BOW + LEN + POS). This results in vectors of 200, 201 and 228 dimensions respectively, which are provided to an SVM with a linear kernel. No-Answer Baseline (NA) : Most of the questions we receive are difficult to answer in a legally-sound way on the basis of information present in the privacy policy. We establish a simple baseline to quantify the effect of identifying every question as unanswerable. Word Count Baseline : To quantify the effect of using simple lexical matching to answer the questions, we retrieve the top candidate policy sentences for each question using a word count baseline BIBREF53, which counts the number of question words that also appear in a sentence. We include the top 2, 3 and 5 candidates as baselines. Human Performance: We pick each reference answer provided by an annotator, and compute the F1 with respect to the remaining references, as described in section 4.2.1. Each reference answer is treated as the prediction, and the remaining n-1 answers are treated as the gold reference. The average of the maximum F1 across all reference answers is computed as the human baseline. 
 Question: Were other baselines tested to compare with the neural baseline?
Output: Extractive
We evaluate our newly proposed models and related baselines in several low-resource settings across different languages with real, distantly supervised data with non-synthetic noise. The advanced modeling of the noisy labels substantially improves the performance up to 36% over methods without noise-handling and up to 9% over all other noise-handling baselines. 
 Question: Did they evaluate against baseline?
Output: Yes-no
Some authors use pretrained embeddings (especially when their data set is too small to train their own embeddings) or try to modify these embeddings and adjust to their set. But the biggest drawback of these approaches is that the corpus for training embeddings can be not related to the specific task where embeddings are utilized. A lot of medical concepts are not contained in well-known embeddings bases. Furthermore, the similarity of words may vary in different contexts. Then, we compute embeddings of concepts (by GloVe) for interview descriptions and for examination descriptions separately. 
 Question: Do they fine-tune the used word embeddings on their medical texts?
Output: Yes-no
Our approach consists of structuring input to the transformer network to use and guide the self-attention of the transformers, conditioning it on the entity. Our main mode of encoding the input, the entity-first method, is shown in Figure FIGREF4.  We also have an entity-last variant where the entity is primarily observed just before the classification token to condition the [CLS] token's self-attention accordingly.  As an additional variation, we can either run the transformer once per document with multiple [CLS] tokens (a document-level model as shown in Figure FIGREF4) or specialize the prediction to a single timestep (a sentence-level model). 
 Question: In what way is the input restructured?
Output: Abstractive
Level B categorizes the type of offense and two labels are used: targeted (TIN) and untargeted (INT) insults and threats.

Targeted Insult (TIN): Posts which contain an insult/threat to an individual, group, or others (see next layer);

Untargeted (UNT): Posts containing non-targeted profanity and swearing. Posts with general profanity are not targeted, but they contain non-acceptable language. Level C categorizes the targets of insults and threats as individual (IND), group (GRP), and other (OTH).

Individual (IND): Posts targeting an individual. It can be a a famous person, a named individual or an unnamed participant in the conversation. Insults and threats targeted at individuals are often defined as cyberbulling.

Group (GRP): The target of these offensive posts is a group of people considered as a unity due to the same ethnicity, gender or sexual orientation, political affiliation, religious belief, or other common characteristic. Many of the insults and threats targeted at a group correspond to what is commonly understood as hate speech.

Other (OTH): The target of these offensive posts does not belong to any of the previous two categories (e.g. an organization, a situation, an event, or an issue). 
 Question: What kinds of offensive content are explored?
Output: Abstractive
In HotpotQA, on average we can find 6 candidate chains (2-hop) in a instance, and the human labeled true reasoning chain is unique. A predicted chain is correct if the chain only contains all supporting passages (exact match of passages).

In MedHop, on average we can find 30 candidate chains (3-hop). For each candidate chain our human annotators labeled whether it is correct or not, and the correct reasoning chain is not unique. A predicted chain is correct if it is one of the chains that human labeled as correct.

The accuracy is defined as the ratio: The accuracy is defined as the ratio: 
 Question: What benchmarks are created?
Output: Abstractive
We use the pre-trained uncased BERT$_\mathrm {BASE}$ model for fine-tuning, because we find that BERT$_\mathrm {LARGE}$ model performs slightly worse than BERT$_\mathrm {BASE}$ in this task.  
 Question: Do they use large or small BERT?
Output: Abstractive
The SG model achieved a high average similarity score of 0.650 followed by CBoW with a 0.632 average similarity score. The GloVe also achieved a considerable average score of 0.591 respectively. However, the average similarity score of SdfastText is 0.388 and the word pair Microsoft-Bill Gates is not available in the vocabulary of SdfastText. Moreover, the average semantic relatedness similarity score between countries and their capitals is shown in Table TABREF78 with English translation, where SG also yields the best average score of 0.663 followed by CBoW with 0.611 similarity score. The GloVe also yields better semantic relatedness of 0.576 and the SdfastText yield an average score of 0.391. 
 Question: How does proposed word embeddings compare to Sindhi fastText word representations?
Output: Abstractive
In cooperation with the ruling regime, Weibo sets strict control over the content published under its service BIBREF0. According to Zhu et al. zhu-etal:2013, Weibo uses a variety of strategies to target censorable posts, ranging from keyword list filtering to individual user monitoring. Among all posts that are eventually censored, nearly 30% of them are censored within 5-30 minutes, and nearly 90% within 24 hours BIBREF1. We hypothesize that the former are done automatically, while the latter are removed by human censors. 
 Question: Is is known whether Sina Weibo posts are censored by humans or some automatic classifier?
Output: Yes-no
Three datasets annotated with emotions are commonly used for the development and evaluation of emotion detection systems, namely the Affective Text dataset, the Fairy Tales dataset, and the ISEAR dataset. A summary is provided in Table TABREF8 , which also shows, in the bottom row, what role each dataset has in our experiments: apart from the development portion of the Affective Text, which we used to develop our models (Section SECREF4 ), all three have been used as benchmarks for our evaluation. 
 Question: Which existing benchmarks did they compare to?
Output: Extractive
We use the existing large-scale antonym and synonym pairs previously used by Nguyen:16. Originally, the data pairs were collected from WordNet BIBREF9 and Wordnik. 
 Question: What dataset do they use to evaluate their method?
Output: Extractive
MIMIC-III is a freely available, deidentified database containing electronic health records of patients admitted to an Intensive Care Unit (ICU) at Beth Israel Deaconess Medical Center between 2001 and 2012 
 Question: what datasets were used?
Output: Extractive
Intuitively, under-translated input words should contribute little to the NMT outputs, yielding much smaller word importance.  By exploiting the word importance calculated by Attribution method, we can identify the under-translation errors automatically without the involvement of human interpreters.  
 Question: How do they measure which words are under-translated by NMT models?
Output: Abstractive
In this work, we analyze a set of tweets related to a specific classical music radio channel, BBC Radio 3, interested in detecting two types of musical named entities, Contributor and Musical Work. 
 Question: What language is the Twitter content in?
Output: Abstractive
Understanding a user's intent and sentiment is of utmost importance for current intelligent chatbots to respond appropriately to human requests. However, current systems are not able to perform to their best capacity when presented with incomplete data, meaning sentences with missing or incorrect words. This scenario is likely to happen when one considers human error done in writing. In fact, it is rather naive to assume that users will always type fully grammatically correct sentences.  
 Question: How do the authors define or exemplify 'incorrect words'?
Output: Abstractive
As we show below, STransE performs better than the SE and TransE models and other state-of-the-art link prediction models on two standard link prediction datasets WN18 and FB15k, so it can serve as a new baseline for KB completion. 
 Question: What datasets are used to evaluate the model?
Output: Abstractive
During each evaluation, target terms are masked, predicted, and then compared to the masked (known) value. 
 Question: Are the Transformers masked?
Output: Yes-no
We focus primarily on the smaller data set of 645 hand-labeled articles provided to task participants, both for training and for validation.  We pre-trained BERT-base on the 600,000 articles without labels by using the same Cloze task BIBREF5 that BERT had originally used for pre-training.  
 Question: How long is the dataset?
Output: Abstractive
We adopt a DNN-based acoustic model BIBREF0 with 11 hidden layers and the alignment used to train the model is derived from a HMM-GMM model trained with SAT criterion. 
 Question: What are the deep learning architectures used in the task?
Output: Extractive