niryuu/sni-each-converted huggingface dataset:
We have implemented the following interfaces for Macaw:

[leftmargin=*]

File IO: This interface is designed for experimental purposes, such as evaluating the performance of a conversational search technique on a dataset with multiple queries. This is not an interactive interface.

Standard IO: This interactive command line interface is designed for development purposes to interact with the system, see the logs, and debug or improve the system.

Telegram: This interactive interface is designed for interaction with real users (see FIGREF4). Telegram is a popular instant messaging service whose client-side code is open-source. We have implemented a Telegram bot that can be used with different devices (personal computers, tablets, and mobile phones) and different operating systems (Android, iOS, Linux, Mac OS, and Windows). This interface allows multi-modal interactions (text, speech, click, image). It can be also used for speech-only interactions. For speech recognition and generation, Macaw relies on online APIs, e.g., the services provided by Google Cloud and Microsoft Azure. In addition, there exist multiple popular groups and channels in Telegram, which allows further integration of social networks with conversational systems. For example, see the Naseri and Zamani's study on news popularity in Telegram BIBREF12.
Output: What interface does Macaw currently have?
holistic: The baseline model which maps the holistic image feature and LSTM-encoded question feature to a common space and perform element-wise multiplication between them.

TraAtt: The traditional attention model, implementation of WTL model BIBREF9 using the same $3\times 3$ regions in SalAtt model.

RegAtt: The region attention model which employs our novel attention method, same as the SalAtt model but without region pre-selection.

ConAtt: The convolutional region pre-selection attention model which replaces the BiLSTM in SalAtt model with a weight-sharing linear mapping, implemented by a convolutional layer.

Besides, we also compare our SalAtt model with the popular baseline models i.e. iBOWIMG BIBREF4 , VQA BIBREF1 , and the state-of-the-art attention-based models i.e. WTL BIBREF9 , NMN BIBREF21 , SAN BIBREF14 , AMA BIBREF33 , FDA BIBREF34 , D-NMN BIBREF35 , DMN+ BIBREF8 on two tasks of COCO-VQA.
Output: To which previous papers does this work compare its results?
Numerically, Our model improves the DAR accuracy over Bi-LSTM-CRF by 2.1% and 0.8% on SwDA and MRDA respectively.
Output: By how much do they outperform state-of-the-art solutions on SWDA and MRDA?
As one of our aims is to compare coreference chain properties in automatic translation with those of the source texts and human reference, we derive data from ParCorFull, an English-German corpus annotated with full coreference chains BIBREF46.
Output: What languages are seen in the news and TED datasets?
1) Typical Methods. Three typical knowledge graph embedding methods includes TransE, TransR and TransH are selected as baselines. 2) Path-based Methods. We compare our method with two typical path-based model include PTransE, and ALL-PATHS BIBREF18. 3) Attribute-incorporated Methods. Several state-of-art attribute-incorporated methods including R-GCN BIBREF24 and KR-EAR BIBREF26 are used to compare with our methods on three real datasets.
Output: What seven state-of-the-art methods are used for comparison?
In addition, we propose a new standard experimental protocol for the IBM-UB-1 dataset BIBREF25 (Sec. SECREF50 ) to enable easier comparison between approaches in the future. The IAM-OnDB dataset BIBREF42 is probably the most used evaluation dataset for online handwriting recognition. It consists of 298 523 characters in 86 272 word instances from a dictionary of 11 059 words written by 221 writers. We use the standard IAM-OnDB dataset separation: one training set, two validations sets and a test set containing 5 363, 1 438, 1 518 and 3 859 written lines, respectively. We tune the decoder weights using the validation set with 1 438 items and report error rates on the test set. We provide an evaluation of our production system trained on our in-house datasets applied to a number of publicly available benchmark datasets from the literature. Note that for all experiments presented in this section we evaluate our current live system without any tuning specifec to the tasks at hand.

The ICDAR-2013 Competition for Online Handwriting Chinese Character Recognition BIBREF45 introduced a dataset for classifying the most common Chinese characters. We report the error rates in comparison to published results from the competition and more recent work done by others in Table TABREF56 . In the ICFHR2018 Competition on Vietnamese Online Handwritten Text Recognition using VNOnDB BIBREF50 , our production system was evaluated against other systems. The system used in the competition is the one reported and described in this paper. Due to licensing restrictions we were unable to do any experiments on the competition training data, or specific tuning for the competition, which was not the case for the other systems mentioned here.
Output: What datasets did they use?
Various multilingual extensions of NMT have already been proposed in the literature. The authors of BIBREF18 , BIBREF19 apply multitask learning to train models for multiple languages. Zoph and Knight BIBREF20 propose a multi-source model and BIBREF21 introduces a character-level encoder that is shared across several source languages. In our setup, we will follow the main idea proposed by Johnson et al. BIBREF22 . The authors of that paper suggest a simple addition by means of a language flag on the source language side (see Figure 2 ) to indicate the target language that needs to be produced by the decoder. This flag will be mapped on a dense vector representation and can be used to trigger the generation of the selected language. The authors of the paper argue that the model enables transfer learning and supports the translation between languages that are not explicitly available in training.
Output: What neural machine translation models can learn in terms of transfer learning?
For the first task, the human raters are asked to evaluate these outputs on the first three aspects, relevance, attractiveness, and language fluency on a Likert scale from 1 to 10 (integer values). For attractiveness, annotators are asked how attractive the headlines are.
Output: How is attraction score measured?
From the results which we get, it is evident that the transformer model achieves higher BLEU score than both Attention encoder-decoder and sequence-sequence model.
Output: How were their results compared to state-of-the-art?
As is common in the literature BIBREF4 , BIBREF8 , we use 300-dimensional vectors for our embeddings and all word vectors are normalized to unit length prior to evaluation.
Output: What dimensions of word embeddings do they produce using factorization?
The lexicon is built over all the data and therefore includes the vocabulary from both the training and testing sets.
Output: Is the lexicon the same for all languages?
As the gold standard sentiment lexica, we chose manually created lexicon in Czech BIBREF11 , German BIBREF12 , French BIBREF13 , Macedonian BIBREF14 , and Spanish BIBREF15 .
Output: what sentiment sources do they compare with?
Inspired by the supervised reordering in conventional SMT, in this paper, we propose a Supervised Attention based NMT (SA-NMT) model. Specifically, similar to conventional SMT, we first run off-the-shelf aligners (GIZA++ BIBREF3 or fast_align BIBREF4 etc.) to obtain the alignment of the bilingual training corpus in advance. Then, treating this alignment result as the supervision of attention, we jointly learn attention and translation, both in supervised manners. Since the conventional aligners delivers higher quality alignment, it is expected that the alignment in the supervised attention NMT will be improved leading to better end-to-end translation performance.
Output: Which conventional alignment models do they use as guidance?
For the evaluation of summaries we use the standard ROGUE metric. For comparison with previous AMR based summarization methods, we report the Recall, Precision and INLINEFORM0 scores for ROGUE-1.
Output: Which evaluation methods are used?
Recently, Wen et al. wensclstm15 proposed an RNN-based approach, which outperformed previous methods on several metrics. However, the generated sentences often did not include all desired attributes.
Output: How is some information lost in the RNN-based generation models?
Encoders with induced latent structures have been shown to benefit several tasks including document classification, natural language inference BIBREF12, BIBREF13, and machine translation BIBREF11. 
Output: Is there any evidence that encoders with latent structures work well on other tasks?
Our VG training set consists of 85% of the data: 16k images and 740k corresponding region descriptions.
Output: How big is data provided by this research?
We leveraged an existing, annotated Twitter dataset that was constructed based on a hierarchical model of depression-related symptoms BIBREF12 , BIBREF13 . The dataset contains 9,473 annotations for 9,300 tweets. Each tweet is annotated as no evidence of depression (e.g., “Citizens fear an economic depression") or evidence of depression (e.g., “depressed over disappointment"). 
Output: Do they evaluate only on English datasets?
We use two datasets, NELL-One and Wiki-One which are constructed by BIBREF11 .
Output: What datasets are used to evaluate the approach?
Our Recurrent +ELMo model uses the language model from BIBREF9 to provide contextualized embeddings to the baseline model outlined above, as recommended by the authors.

Our OpenAI GPT model fine-tunes the 12 layer 768 dimensional uni-directional transformer from BIBREF27 , which has been pre-trained as a language model on the Books corpus BIBREF36 .
Output: did they use other pretrained language models besides bert?
To report the result for answer selection, MAP and MRR are used; however, the answer triggering task is evaluated by F1-score. The result for MULT Method is reported in Table. 1.
Output: Is accuracy the only metric they used to compare systems?
Given a multiple-choice question $qa$ with question text $q$ and answer choices A= $\lbrace a_i\rbrace $ , we select the most relevant tuples from $T$ and $S$ as follows. Selecting from Tuple KB: We use an inverted index to find the 1,000 tuples that have the most overlapping tokens with question tokens $tok(qa).$ .
Output: Is an entity linking process used?
Our data has been developed by crawling and pre-processing an OSG web forum. The forum has a great variety of different groups such as depression, anxiety, stress, relationship, cancer, sexually transmitted diseases, etc.
Output: How did they obtain the OSG dataset?
We assessed the proposed models on four different NLG domains: finding a restaurant, finding a hotel, buying a laptop, and buying a television. The Restaurant and Hotel were collected in BIBREF4 , while the Laptop and TV datasets have been released by BIBREF22 with a much larger input space but only one training example for each DA so that the system must learn partial realization of concepts and be able to recombine and apply them to unseen DAs. 
Output: Does the model evaluated on NLG datasets or dialog datasets?
Database Construction: we crawled travel information in Beijing from the Web, including Hotel, Attraction, and Restaurant domains (hereafter we name the three domains as HAR domains). Then, we used the metro information of entities in HAR domains to build the metro database. Goal Generation: a multi-domain goal generator was designed based on the database. The relation across domains is captured in two ways. One is to constrain two targets that locate near each other. The other is to use a taxi or metro to commute between two targets in HAR domains mentioned in the context.  Dialogue Collection: before the formal data collection starts, we required the workers to make a small number of dialogues and gave them feedback about the dialogue quality. Then, well-trained workers were paired to converse according to the given goals.  Dialogue Annotation: we used some rules to automatically annotate dialogue acts according to user states, system states, and dialogue histories. 
Output: How was the dataset collected?
The performances of the NER experiments are reported separately for three different parts of the system proposed. Results shown that, in the first case, in the training set the F1 score is always greater than 97%, with a maximum of 99.65%. With both test sets performances decrease, varying between 94-97%. In the case of UGC, comparing the F1 score we can observe how performances significantly decrease.
Output: What are their results on the entity recognition task?
We compare our approach to two other, the first one uses almost the same tweet data as we use for training, and the second one is the CrowdFlower dataset annotated for emotions. In the first one Wang et al. BIBREF21 downloaded over 5M tweets which included one of 131 emotional hashtags based on Parrott's three-level categorization of emotions in seven categories: joy, sadness, anger, love, fear, thankfulness, surprise. 
Output: What data is used in experiments?
Specifically, we group the models based on the objective function it optimizes. We believe this work can aid the understanding of the existing literature.  We believe that the performance of these models is highly dependent on the objective function it optimizes – predicting adjacent word (within-tweet relationships), adjacent tweet (inter-tweet relationships), the tweet itself (autoencoder), modeling from structured resources like paraphrase databases and weak supervision. 
Output: How do they encourage understanding of literature as part of their objective function?
The baseline model for our experiments is explained in the paper by Alec Go [1]. The model uses the Naive Bayes, SVM, and the Maximum Entropy classifiers for their experiment.
Output: What previously proposed methods is this method compared against?
N-GrAM ranked first in all cases except for the language variety task. In this case, the baseline was the top-ranked system, and ours was second by a small margin. Our system significantly out-performed the baseline on the joint task, as the baseline scored significantly lower for the gender task than for the variety task.


Output: On which task does do model do worst?
The basic concept is to use a phone-discriminative model to produce frame-level phonetic features, and then use these features to enhance RNN LID systems that were originally built with raw acoustic features. The initial step is therefore feature combination, with the phonetic feature used as auxiliary information to assist acoustic RNN LID. This is improved further, as additional research identified that a simpler model using only the phonetic feature as the RNN LID input provides even better performance. We call this RNN model based on phonetic features the phonetic temporal neural LID approach, or PTN LID. As well as having a simplified model structure, the PTN offers deeper insight into the LID task by rediscovering the value of the phonetic temporal property in language discrimination. 
Output: What is the main contribution of the paper?
We evaluate the sequence-to-sequence QRNN architecture described in SECREF5 on a challenging neural machine translation task, IWSLT German–English spoken-domain translation, applying fully character-level segmentation. 
Output: What languages pairs are used in machine translation?
we take the CORD-19 dataset BIBREF2, which contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. We develop sentence classification methods to identify all sentences narrating radiological findings from COVID-19.  We conduct experiments to verify the effectiveness of our method. From the CORD-19 dataset, our method successfully discovers a set of clinical findings that are closely related with COVID-19.
Output: How large is the collection of COVID-19 literature?
The dataset consists of 5000 reviews in bahasa Indonesia.
Output: Does the dataset contain non-English reviews?
To model an answer which is a collection of spans, the multi-span head uses the $\mathtt {BIO}$ tagging format BIBREF8: $\mathtt {B}$ is used to mark the beginning of a span, $\mathtt {I}$ is used to mark the inside of a span and $\mathtt {O}$ is used to mark tokens not included in a span. In this way, we get a sequence of chunks that can be decoded to a final answer - a collection of spans.
Output: How they use sequence tagging to answer multi-span questions?
Wikilinks can be seen as a large-scale, naturally-occurring, crowd-sourced dataset where thousands of human annotators provide ground truths for mentions of interest. This means that the dataset contains various kinds of noise, especially due to incoherent contexts. We prepare our dataset from the local-context version of Wikilinks, and resolve ground-truth links using a Wikipedia dump from April 2016. We use the page and redirect tables for resolution, and keep the database pageid column as a unique identifier for Wikipedia entities. We discard mentions where the ground-truth could not be resolved (only 3% of mentions).
Output: How was a quality control performed so that the text is noisy but the annotations are accurate?
This action space is of the order of $A=\mathcal {O}(|V| \times |O|^2)$ where $V$ is the number of action verbs, and $O$ is the number of distinct objects in the world that the agent can interact with. As this is too large a space for a RL agent to effectively explore, the knowledge graph is used to prune this space by ranking actions based on their presence in the current knowledge graph and the relations between the objects in the graph as in BIBREF7
Output: How is the domain knowledge transfer represented as knowledge graph?
Following prior work BIBREF56 , BIBREF57 , BIBREF17 , we employ the MNCut spectral clustering algorithm BIBREF58 , which has wide applicability in similar NLP tasks which involve high-dimensional feature spaces BIBREF59 , BIBREF60 , BIBREF18 .
Output: What clustering algorithm is used on top of the VerbNet-specialized representations?
Hence, in this paper, we propose a cascaded multimodal STT with two components: (i) an English ASR system trained on the How2 dataset and (ii) a transformer-based BIBREF0 visually grounded MMT system.
Output: What dataset was used in this work?
In order to derive the discourse features, an entity grid is constructed by feeding the document through an NLP pipeline to identify salient entities. Two flavors of discourse features are created by populating the entity grid with either (i) grammatical relations (GR) or (ii) RST discourse relations (RST).  Denoting a feature as such with INLINEFORM4 , we construct the pooling vector INLINEFORM5 for the char-bigrams, and concatenate INLINEFORM6 to INLINEFORM7 before feeding the resulting vector to the softmax layer.
Output: How are discourse features incorporated into the model?
Compared to single-task baselines, performance improved on the low-resource En-De task and was comparable on high-resource En-Fr task.
Output: How big are negative effects of proposed techniques on high-resource tasks?
In terms of speed, our system was able to lemmatize 7.4 million words on a personal laptop in almost 2 minutes compared to 2.5 hours for MADAMIRA, i.e. 75 times faster. 
Output: How was speed measured?
We compared our multi-turn emotionally engaging dialog model (denoted as MEED) with two baselines—the vanilla sequence-to-sequence model (denoted as S2S) and HRAN.
Output: What two baseline models are used?
We experiment with small feed-forward networks for four diverse NLP tasks: language identification, part-of-speech tagging, word segmentation, and preordering for statistical machine translation.
Output: What NLP tasks do the authors evaluate feed-forward networks on?
We carried out the experimentation with a range of classifiers of different types: Support Vector Machines (SVM), Gaussian Naive Bayes, Multinomial Naive Bayes, Decision Trees, Random Forests and a Maximum Entropy classifier.
Output: What model do they train?
However, as shown in Figure 1 , to assess the similarity between two research papers, a more effective strategy would compare and align (via local-weighting) individual important words (keywords) within a pair of abstracts, while information from other words (e.g., stop words) that tend to be less relevant can be effectively ignored (down-weighted).  We propose to learn a semantic-aware Network Embedding (NE) that incorporates word-level alignment features abstracted from text sequences associated with vertex pairs. Given a pair of sentences, our model first aligns each word within one sentence with keywords from the other sentence (adaptively up-weighted via an attention mechanism), producing a set of fine-grained matching vectors
Output: What text sequences are associated with each vertex?
Prior to the SAOKE data set, an annotated data set for OIE tasks with 3,200 sentences in 2 domains was released in BIBREF20 to evaluate OIE algorithms, in which the data set was said BIBREF20 “13 times larger than the previous largest annotated Open IE corpus”.
Output: What's the size of the previous largest OpenIE dataset?
By analyzing large numbers of dialogue act sequences correlated with specific outcomes, various rules can be derived, i.e. "Continuing to request information late in a conversation often leads to customer dissatisfaction." This can then be codified into a best practice pattern rules for automated systems, such as "A request for information act should be issued early in a conversation, followed by an answer, informative statement, or apology towards the end of the conversation." Our analysis helps zone in on how the use of certain dialogue acts may be likely to result in different outcomes. The weights we observe vary in the amount of insight provided: for example, offering extra help at the end of a conversation, or thanking the customer yields more satisfied customers, and more resolved problems (with ratios of above 6:1). However, some outcomes are much more subtle: for example, asking yes-no questions early-on in a conversation is highly associated with problem resolution (ratio 3:1), but asking them at the end of a conversation has as similarly strong association with unsatisfied customers. Giving elaborate answers that are not a simple affirmative, negative, or response acknowledgement (i.e. Answer (Other)) towards the middle of a conversation leads to satisfied customers that are not frustrated. Likewise, requesting information towards the end of a conversation (implying that more information is still necessary at the termination of the dialogue) leads to unsatisfied and unresolved customers, with ratios of at least 4:1.
Output: Which patterns and rules are derived?
At the same time, some information about inflected word forms in the context can be useful, but it is lost during lemmatization, and this leads to the decreased score. Arguably, this means that lemmatization brings along both advantages and disadvantages for WSD with ELMo. 
Output: Do the authors mention any downside of lemmatizing input before training ELMo?
For evaluation of clustering performance, we use Calinski-Harabasz score BIBREF42 , also known as the variance ratio criterion. CH score is defined as the ratio between the within-cluster dispersion and the between-cluster dispersion. CH score has a range of $[0, +\infty ]$ and a higher CH score corresponds to a better clustering.  For visual validation, we plot and inspect the t-Distributed Stochastic Neighbor Embedding (t-SNE) BIBREF52 and Uniform Manifold Approximation and Projection (UMAP) BIBREF53 mappings of the learned representations as well.
Output: How do they evaluate their method?
The results are shown in Fig. FIGREF87 for different numbers of topics, where we can see that the proposed model outperforms all the baselines, being the svi version the one that performs best. In order to assess the computational advantages of the stochastic variational inference (svi) over the batch algorithm, the log marginal likelihood (or log evidence) was plotted against the number of iterations. Fig. FIGREF88 shows this comparison. Not surprisingly, the svi version converges much faster to higher values of the log marginal likelihood when compared to the batch version, which reflects the efficiency of the svi algorithm.
Output: what are the advantages of the proposed model?
We build a dataset of Twitter accounts based on two lists annotated in previous works. For the non-factual accounts, we rely on a list of 180 Twitter accounts from BIBREF1. On the other hand, for the factual accounts, we use a list with another 32 Twitter accounts from BIBREF19 that are considered trustworthy by independent third parties.
Output: How did they obtain the dataset?
For evaluating the proposed model on behavior related data, we employ the Couples Therapy Corpus (CoupTher) BIBREF21 and Cancer Couples Interaction Dataset (Cancer) BIBREF22. These are the targeted conditions under which a behavior-gated language model can offer improved performance. We utilize the Couple's Therapy Corpus as an in-domain experimental corpus since our behavior classification model is also trained on the same. 
Output: On which dataset is model trained?
In this section, we first describe a baseline method inspired by the “align to segment” of BIBREF12, BIBREF13. We then propose two extensions providing the model with a signal relevant to the segmentation process, so as to move towards a joint learning of segmentation and alignment.
Output: Does the paper report any alignment-only baseline?
Just as introduced in sec:introduction, it is intractable to compute similarity exactly, as involving INLINEFORM0 computation. Hence, we consider the monte-carlo approximation: DISPLAYFORM0

where INLINEFORM0 is a list of entity pairs sampled from INLINEFORM1 . We use sequential sampling to gain INLINEFORM6 , which means we first sample INLINEFORM7 given INLINEFORM8 from INLINEFORM9 , and then sample INLINEFORM10 given INLINEFORM11 and INLINEFORM12 from INLINEFORM13 .
Output: Which sampling method do they use to approximate similarity between the conditional probability distributions over entity pairs?
We choose the following three models as the baselines:

K-means is a well known data clustering algorithm, we implement the algorithm using sklearn toolbox, and represent documents using bag-of-words weighted by TF-IDF.

LEM BIBREF13 is a Bayesian modeling approach for open-domain event extraction. It treats an event as a latent variable and models the generation of an event as a joint distribution of its individual event elements. We implement the algorithm with the default configuration.

DPEMM BIBREF14 is a non-parametric mixture model for event extraction. It addresses the limitation of LEM that the number of events should be known beforehand. We implement the model with the default configuration.
Output: What baseline approaches does this approach out-perform?
Word2vec representation is far better, advanced and a recent technique which functions by mapping words to a 300 dimensional vector representations.
Output: What is the dimension of the embeddings?
We use three datasets for our experiments: IWSLT14 German-English, Turkish-English, and WMT14 English-German.
Output: what datasets were used?
We use three datasets for our experiments: IWSLT14 German-English, Turkish-English, and WMT14 English-German.
Output: what language pairs are explored?
In this paper, we propose an improved RNN-T model with language bias to alleviate the problem. The model is trained to predict language IDs as well as the subwords. To ensure the model can learn CS information, we add language IDs in the CS point of transcription, as illustrated in Fig. 1.
Output: How do they obtain language identities?
Results form Table TABREF31 may give an idea that INLINEFORM0 is just an scaled INLINEFORM1 . While it is true that they show a linear correlation, INLINEFORM2 may produce a different system ranking than INLINEFORM3 given the integral multi-reference principle it follows. However, what we consider the most profitable about INLINEFORM4 is the twofold inclusion of all available references it performs. First, the construction of INLINEFORM5 to provide a more inclusive reference against to whom be evaluated and then, the computation of INLINEFORM6 , which scales the result depending of the agreement between references. We showed how INLINEFORM0 is an inclusive metric which not only evaluates the performance of a system against all references, but also takes into account the agreement between them. 
Output: What makes it a more reliable metric?
We construct the embeddings for each document using BERT models finetuned on MLDoc. We mean-pool each document embedding to create a single vector per document. We then calculate the cosine similarity between the embeddings for the English document and its translation. In Table TABREF21 , we observe that the median cosine similarity increases dramatically with adversarial training, which suggests that the embeddings became more language-independent.
Output: How do they quantify alignment between the embeddings of a document and its translation?
We then show the tweet text and image and we ask them to classify it in one of 6 categories: No attacks to any community, racist, sexist, homophobic, religion based attacks or attacks to other communities.
Output: What annotations are available in the dataset - tweat used hate speach or not?
In target data, the code is written in Python programming language.
Output: What programming language is target language?
We compare the performance with the Recurrent Entity Networks model (EntNet) BIBREF17 .  The RelNet model achieves a mean error of 0.285% across tasks which is better than the results of the EntNet model BIBREF17 . The RelNet model is able to achieve 0% test error on 11 of the tasks, whereas the EntNet model achieves 0% error on 7 of the tasks.
Output: What methods is RelNet compared to?
We evaluated our models on 3 different datasets:

CSAT dataset for CSAT prediction, consisting of spoken transcripts (automatic via ASR).

20 newsgroups for topic identification task, consisting of written text;

Fisher Phase 1 corpus for topic identification task, consisting of spoken transcripts (manual);
Output: What datasets did they use for evaluation?
However, we also showed difference in produced errors for each method and different impact at word-level depending of the approach or units. Thus, future work will focus on analysing the orthographic output of these systems in two ways: 1) investigate errors produced by the end-to-end methods and explore several approaches to correct common errors done in French and 2) compare the end-to-end methods in a SLU context and evaluate the semantic value of the partially correct produced words.
Output: What will be in focus for future work?
We use the existing large-scale antonym and synonym pairs previously used by Nguyen:16. Originally, the data pairs were collected from WordNet BIBREF9 and Wordnik.
Output: What dataset do they use to evaluate their method?
Dataset: We perform experiments on two widely-used tasks for the English-to-Japanese language pair: KFTT BIBREF12 and BTEC BIBREF13 . KFTT is a collection of Wikipedia article about city of Kyoto and BTEC is a travel conversation corpus. BTEC is an easier translation task than KFTT, because KFTT covers a broader domain, has a larger vocabulary of rare words, and has relatively long sentences. 
Output: What datasets were used?
For a purpose of learning from limited annotated linguistic resources, our preliminary discovery shows that it is possible to build a geometric space projection between embedding spaces to help cross-lingual NE recognition.
Output: What is their model?
The data stems from a joint ADAPT-Microsoft project. 
Output: what dataset was used?
The proposed system consists of a Bi-directional Long Short-Term Memory (BiLSTM) BIBREF16, a two-level attention mechanism BIBREF29, BIBREF30 and a shared representation for emotion and sentiment analysis tasks. The BiLSTM encodes the word representation of each word. This representation is shared between the subsystems of sentiment and emotion analysis. Each of the shared representations is then fed to the primary attention mechanism of both the subsystems. The primary attention mechanism finds the best representation for each word for each task. The secondary attention mechanism acts on top of the primary attention to extract the best sentence representation by focusing on the suitable context for each task. Finally, the representations of both the tasks are fed to two different feed-forward neural networks to produce two outputs - one for sentiment analysis and one for emotion analysis. Each component is explained in the subsequent subsections.
Output: How is multi-tasking performed?
In the first scenario we equip the decoder with an additional morphology table including target-side affixes.
Output: What type of morphological information is contained in the "morphology table"?
We conducted experiments on the Amazon reviews dataset BIBREF9, which is a benchmark dataset in the cross-domain sentiment analysis field. This dataset contains Amazon product reviews of four different product domains: Books (B), DVD (D), Electronics (E), and Kitchen (K) appliances. Each review is originally associated with a rating of 1-5 stars and is encoded in 5,000 dimensional feature vectors of bag-of-words unigrams and bigrams.

Experiment ::: Dataset and Task Design ::: Binary-Class.
From this dataset, we constructed 12 binary-class cross-domain sentiment analysis tasks: B$\rightarrow $D, B$\rightarrow $E, B$\rightarrow $K, D$\rightarrow $B, D$\rightarrow $E, D$\rightarrow $K, E$\rightarrow $B, E$\rightarrow $D, E$\rightarrow $K, K$\rightarrow $B, K$\rightarrow $D, K$\rightarrow $E. Following the setting of previous works, we treated a reviews as class `1' if it was ranked up to 3 stars, and as class `2' if it was ranked 4 or 5 stars. For each task, $\mathcal {D}_S$ consisted of 1,000 examples of each class, and $\mathcal {D}_T$ consists of 1500 examples of class `1' and 500 examples of class `2'. In addition, since it is reasonable to assume that $\mathcal {D}_T$ can reveal the distribution of target domain data, we controlled the target domain testing dataset to have the same class ratio as $\mathcal {D}_T$. Using the same label assigning mechanism, we also studied model performance over different degrees of $\rm {P}(\rm {Y})$ shift, which was evaluated by the max value of $\rm {P}_S(\rm {Y}=i)/\rm {P}_T(\rm {Y}=i), \forall i=1, \cdots , L$. Please refer to Appendix C for more detail about the task design for this study.

Experiment ::: Dataset and Task Design ::: Multi-Class.
We additionally constructed 12 multi-class cross-domain sentiment classification tasks. Tasks were designed to distinguish reviews of 1 or 2 stars (class 1) from those of 4 stars (class 2) and those of 5 stars (class 3). For each task, $\mathcal {D}_S$ contained 1000 examples of each class, and $\mathcal {D}_T$ consisted of 500 examples of class 1, 1500 examples of class 2, and 1000 examples of class 3. Similarly, we also controlled the target domain testing dataset to have the same class ratio as $\mathcal {D}_T$.
Output: Which sentiment analysis tasks are addressed?
The accuracy of our model is 7.8% higher than the best result achieved by LSVM. The results show that this model can perform better than state-of-the-art baselines including hybrid CNN BIBREF15 and LSTM with attention BIBREF16 by 3.1% on the validation set and 1% on the test set.
Output: What are state of the art methods authors compare their work with?
Our evaluation comprises of a rich set of 19 different algorithms to recommend tags for e-books, which we group into (i) popularity-based, (ii) similarity-based (i.e., using content information), and (iii) hybrid approaches.
Output: what algorithms did they use?
the new DMN+ model does not require that supporting facts In addition, we introduce a new input module to represent images.
Output: What improvements they did for DMN?
LUWAK is implemented in pure JavaScript code, and it uses the LocalStorage of a web browser.
Output: What programming language is the tool written in?
We have evaluated our framework using a synthetic literal set (S-Lite) and a real literal set (R-Lite) from DBpedia BIBREF0 . 
Output: What KB is used in this work?
CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese.
Output: Is Arabic one of the 11 languages in CoVost?
Using the fine-grained gating mechanism conditioned on the lexical features, we can accurately control the information flows between word-level and char-level. Intuitively, the formulation is as follows: INLINEFORM0

where INLINEFORM0 is the element-wise multiplication operator. when the gate has high value, more information flows from the word-level representation; otherwise, char-level will take the dominating place. It is practical in real scenarios. For example, for unfamiliar noun entities, the gates tend to bias towards char-level representation in order to care richer morphological structure.
Output: How does the gatint mechanism combine word and character information?
Next, we analyze why they do not perform well in this task and with this data:

[noitemsep,leftmargin=*]

Noisy data. A major challenge of this task is the discrepancy between annotations due to subjective judgement. Although this affects also detection using only text, its repercussion is bigger in more complex tasks, such as detection using images or multimodal detection.

Complexity and diversity of multimodal relations. Hate speech multimodal publications employ a lot of background knowledge which makes the relations between visual and textual elements they use very complex and diverse, and therefore difficult to learn by a neural network.

Small set of multimodal examples. Fig. FIGREF5 shows some of the challenging multimodal hate examples that we aimed to detect. But although we have collected a big dataset of $150K$ tweets, the subset of multimodal hate there is still too small to learn the complex multimodal relations needed to identify multimodal hate.
Output: What is author's opinion on why current multimodal models cannot outperform models analyzing only text?
Probing into the assorted empirical results may help us discover some interesting phenomenons:

The advantage of pre-training gradually diminishes with the increase of labeled data BIBREF14, BIBREF17, BIBREF18.

Fixed representations yield better results than fine-tuning in some cases BIBREF24.

Overall, pre-training the Seq2Seq encoder outperforms pre-training the decoder BIBREF24, BIBREF17, BIBREF15, BIBREF16.
Output: What experimental phenomena are presented?
We first investigated overall detection of any NMT-Fake reviews (1,006 fake reviews and 994 real reviews).
Output: How many reviews in total (both generated and true) do they evaluate on Amazon Mechanical Turk?
PERPLEXITY
Output: Does the paper consider the use of perplexity in order to identify text anomalies?
For automatically evaluating our methods, we propose to use widely used metric for image/video captioning. This is because the proposed CommonGen task can be regarded as also a caption task where the context are incomplete scenes with given concept-sets. Therefore, we choose BLEU-3/4 BIBREF28, ROUGE-2/L BIBREF29, CIDEr BIBREF30, and SPICE BIBREF31 as the main metrics. Apart from these classic metrics, we also include a novel embedding-based metric named BERTScore BIBREF32. To make the comparisons more clear, we show the delta of BERTScore results by subtracting the score of merely using input concept-sets as target sentences, named $\triangle $BERTS.
Output: What automatic metrics are used for this task?
Deep convolutional neural networks (CNNs) with 2D convolutions and small kernels BIBREF1, have achieved state-of-the-art results for several speech recognition tasks BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6. r Models: Our baseline CNN model BIBREF21 consists of 15 convolutional and one fully-connected layer.
Output: Is model compared against state of the art models on these datasets?
We perform a direct comparison of expert and crowd contributors, for 1000 term groups based on the number of total annotations(200 term groups with 2 total annotations, 200 term groups with 3 total annotations, and so on up to term groups with 6 total annotations).  Evaluators were given a summary of the annotations received for the term group in the form of:The term group "inequality inequity" received annotations as 50.0% sadness, 33.33% disgust, 16.67% anger. Then, they were asked to evaluate on a scale from 1 to 5, how valid these annotations were considered.
Output: How do they compare lexicons?
We compare eight different models in our experiments. Four of them have a Projected Layer (see Fig. FIGREF2), while the others do not have, and this is the only difference between these two groups of our models. So, we actually include four models in our experiments (having a projected layer or not). Firstly, LastStateRNN is the classic RNN model, where the last state passes through an MLP and then the LR Layer estimates the corresponding probability. In contrast, in the AvgRNN model we consider the average vector of all states that come out of the cells. The AttentionRNN model is the one that it has been presented in BIBREF9. Moreover, we introduce the MultiAttentionRNN model for the harassment language detection, which instead of one attention, it includes four attentions, one for each category.
Output: What are the different variations of the attention-based approach which are examined?
We evaluate our model on both English and Chinese segmentation. For both languages we used standard datasets for word segmentation and language modeling. English
The Brent corpus is a standard corpus used in statistical modeling of child language acquisition BIBREF15 , BIBREF16  We use the commonly used version of the PTB prepared by BIBREF17 Since Chinese orthography does not mark spaces between words, there have been a number of efforts to annotate word boundaries. We evaluate against two corpora that have been manually segmented according different segmentation standards. The Beijing University Corpus was one of the corpora used for the International Chinese Word Segmentation Bakeoff BIBREF18 . We use the Penn Chinese Treebank Version 5.1 BIBREF19 .
Output: What dataset is used?
To better exploit such existing data sources, we propose an end-to-end (E2E) model based on pointer networks with attention, which can be trained end-to-end on the input/output pairs of human IE tasks, without requiring token-level annotations. Since our model does not need token-level labels, we create an E2E version of each data set without token-level labels by chunking the BIO-labeled words and using the labels as fields to extract.
Output: Do they assume sentence-level supervision?
Before designing this NLP toolkit, we conducted a survey among engineers and identified a spectrum of three typical personas.
Output: How do the authors evidence the claim that many engineers find it a big overhead to choose from multiple frameworks, models and optimization techniques?
Each of the methods we explore improve in % gendered words, % male bias, and F1 over the baseline Transformer generation model, but we find combining all methods in one – the ALL model is the most advantageous.
Output: What baseline is used to compare the experimental results against?
We discuss two core models for addressing sequence labeling problems and describe, for each, training them in a single-model multilingual setting: (1) the Meta-LSTM BIBREF0 , an extremely strong baseline for our tasks, and (2) a multilingual BERT-based model BIBREF1 .
Output: What is the multilingual baseline?
We collected a total of 657 ratings by 14 volunteers, 5 Italian and 9 non-Italian listeners, spread over the 24 clips and three testing conditions.
Output: How many people are employed for the subjective evaluation?
We evaluate our approach by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews BIBREF6 (see Appendix for details). We quantify the efficiency of a communication scheme $(q_{\alpha },p_{\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords. The accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence.
Output: How are models evaluated in this human-machine communication game?
We evaluate our models on two typical tasks: text classification and text semantic matching.
Output: What tasks do they experiment with?
The focus of this publication is on representing information that is valuable for these tasks but that hitherto has largely been ignored in machine learning approaches centering around simplified language, specifically, text structure (e.g., paragraphs, lines), typography (e.g., font type, font style), and image (content, position, and dimensions) information. Information on the font type and font style (e.g., italics, bold print) of a token and its position on the physical page (for PDFs only) was specified as attributes to the token elements of the tokens layer (cf. Figure FIGREF34 for an example)
Output: Which information about typography is included in the corpus?
In this work, we introduce a methodology that provides VQA algorithms with the ability to generate human interpretable attention maps which effectively ground the answer to the relevant image regions. We accomplish this by leveraging region descriptions and object annotations available in the Visual Genome dataset, and using these to automatically construct attention maps that can be used for attention supervision, instead of requiring human annotators to manually provide grounding labels.
Output: How do they obtain region descriptions and object annotations?
The embedding-wise convolution is to apply a convolution filter w $\in \mathbb {R}^{\mathcal {\\}k*d}$ to a window of $k$ word embeddings to generate a new feature, i.e., summarizing a local context of $k$ words. By applying the convolutional filter to all possible windows in the sentence, a feature map $c$ will be generated.
Output: What supports the claim that injected CNN into recurent units will enhance ability of the model to catch local context and reduce ambiguities?
As shown in Table TABREF12, the T-T model significantly outperforms the LSTM-based RNN-T baseline.
Output: What was previous state of the art model?
Let us take a closer look at what we mean by agreement between the two parse trees. Essentially, if we have two words in the English sentence denoted by i and i', aligned to words j and j' in the parallel Hindi sentence respectively, we can expect a dependency edge between i and i' in the English parse tree to correspond to an edge between j and j' in the Hindi parse tree, and vice versa. For an edge in the English parse tree, we term the corresponding edge or path in the Hindi parse tree as the projection or projected path of the English edge on the Hindi parse tree, and similarly there are projected paths from Hindi to English. The dual decomposition inference algorithm tries to bring the parse trees in the two languages through its constraints.
Output: How does enforcing agreement between parse trees work across different languages?
We start by crawling the content of a few notable debating websites: idebate.com, debatewise.org, procon.org.
Output: What debate websites did they look at?
Therefore, we used the crowdsourcing platform CrowdFlower (CF) for our data collection.
Output: How was this data collected?
In general, the baseline model CAEVO is outperformed by both NN models, and NN model with BERT embedding achieves the greatest performance.
Output: Do the BERT-based embeddings improve results?