prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is datasets for sentiment analysis balanced?
Context 1: Introduction
As social media, specially Twitter, takes on an influential role in presidential elections in the U S , natural language processing of political tweets BIBREF0 has the potential to help with nowcasting and forecasting of election results as well as identifying the main issues with a candidate – tasks of much interest to journalists, political scientists, and campaign organizers BIBREF1  As a methodology to obtain training data for a machine learning system that analyzes political tweets, BIBREF2 devised a crowdsourcing scheme with variable crowdworker numbers based on the difficulty of the annotation task  They provided a dataset of tweets where the sentiments towards political candidates were labeled both by experts in political communication and by crowdworkers who were likely not domain experts  BIBREF2 revealed that crowdworkers can match expert performance relatively accurately and in a budget-efficient manner  Given this result, the authors envisioned future work in which groundtruth labels would be crowdsourced for a large number of tweets and then used to design an automated NLP tool for political tweet analysis 
The question we address here is: How accurate are existing NLP tools for political tweet analysis? These tools would provide a baseline performance that any new machine learning system for political tweet analysis would compete against  We here explore whether existing NLP systems can answer the questions "What sentiment?" and "Towards whom?" accurately for the dataset of political tweets provided by BIBREF2  In our analysis, we include NLP tools with publicly-available APIs, even if the tools were not specifically designed for short texts like tweets, and, in particular, political tweets 
Our experiments reveal that the task of entity-level sentiment analysis is difficult for existing tools to answer accurately while the recognition of the entity, here, which politician, was easier 
NLP Toolkits
NLP toolkits typically have the following capabilities: tokenization, part-of-speech (PoS) tagging, chunking, named entity recognition and sentiment analysis  In a study by BIBREF3, it is shown that the well-known NLP toolkits NLTK BIBREF4, Stanford CoreNLP BIBREF5, and TwitterNLP BIBREF6 have tokenization, PoS tagging and NER modules in their pipelines  There are two main approaches for NER: (1) rule-based and (2) statistical or machine learning based  The most ubiquitous algorithms for sequence tagging use Hidden Markov Models BIBREF7, Maximum Entropy Markov Models BIBREF7, BIBREF8, or Conditional Random Fields BIBREF9  Recent works BIBREF10, BIBREF11 have used recurrent neural networks with attention modules for NER 
Sentiment detection tools like SentiStrength BIBREF12 and TensiStrength BIBREF13 are rule-based tools, relying on various dictionaries of emoticons, slangs, idioms, and ironic phrases, and set of rules that can detect the sentiment of a sentence overall or a targeted sentiment  Given a list of keywords, TensiStrength (similar to SentiStrength) reports the sentiment towards selected entities in a sentence, based on five levels of relaxation and five levels of stress 
Among commercial NLP toolkits (e g , BIBREF14, BIBREF15, BIBREF16), we selected BIBREF17 and BIBREF18 for our experiments, which, to the best of our knowledge, are the only publicly accessible commercial APIs for the task of entity-level sentiment analysis that is agnostic to the text domain  We also report results of TensiStrength BIBREF13, TwitterNLP BIBREF6, BIBREF19, CogComp-NLP BIBREF20, and Stanford NLP NER BIBREF21 
Dataset and Analysis Methodology
We used the 1,000-tweet dataset by BIBREF2 that contains the named-entities labels and entity-level sentiments for each of the four 2016 presidential primary candidates Bernie Sanders, Donald Trump, Hillary Clinton, and Ted Cruz, provided by crowdworkers, and by experts in political communication, whose labels are considered groundtruth  The crowdworkers were located in the US and hired on the BIBREF22 platform  For the task of entity-level sentiment analysis, a 3-scale rating of "negative," "neutral," and "positive" was used by the annotators 
BIBREF2 proposed a decision tree approach for computing the number of crowdworkers who should analyze a tweet based on the difficulty of the task  Tweets are labeled by 2, 3, 5, or 7 workers based on the difficulty of the task and the level of disagreement between the crowdworkers  The model computes the number of workers based on how long a tweet is, the presence of a link in a tweet, and the number of present sarcasm signals  Sarcasm is often used in political tweets and causes disagreement between the crowdworkers  The tweets that are deemed to be sarcastic by the decision tree model, are expected to be more difficult to annotate, and hence are allocated more crowdworkers to work on 
We conducted two sets of experiments  In the first set, we used BIBREF23, BIBREF17, and BIBREF18, for entity-level sentiment analysis; in the second set, BIBREF17, BIBREF19, BIBREF24, BIBREF25, and BIBREF26, BIBREF18 for named-entity recognition 
In the experiments that we conducted with TwitterNLP for named-entity recognition, we worked with the default values of the model  Furthermore, we selected the 3-class Stanford NER model, which uses the classes “person,” “organization,” and “location” because it resulted in higher accuracy compared to the 7-class model  For CogComp-NLP NER we used Ontonotes 5 0 NER model BIBREF27  For spaCy NER we used the `en_core_web_lg' model 
We report the experimental results for our two tasks in terms of the correct classification rate (CCR)  For sentiment analysis, we have a three-class problem (positive, negative, and neutral), where the classes are mutually exclusive  The CCR, averaged for a set of tweets, is defined to be the number of correctly-predicted sentiments over the number of groundtruth sentiments in these tweets  For NER, we consider that each tweet may reference up to four candidates, i e , targeted entities  The CCR, averaged for a set of tweets, is the number of correctly predicted entities (candidates) over the number of groundtruth entities (candidates) in this set 
Results and Discussion
The dataset of 1,000 randomly selected tweets contains more than twice as many tweets about Trump than about the other candidates  In the named-entity recognition experiment, the average CCR of crowdworkers was 98 6%, while the CCR of the automated systems ranged from 77 2% to 96 7%  For four of the automated systems, detecting the entity Trump was more difficult than the other entities (e g , spaCy 72 7% for the entity Trump vs  above 91% for the other entities)  An example of incorrect NER is shown in Figure FIGREF1 top  The difficulties the automated tools had in NER may be explained by the fact that the tools were not trained on tweets, except for TwitterNLP, which was not in active development when the data was created BIBREF1 
In the sentiment analysis experiments, we found that a tweet may contain multiple sentiments  The groundtruth labels contain 210 positive sentiments, 521 neutral sentiments, and 305 negative sentiments to the candidates  We measured the CCR, across all tweets, to be 31 7% for Rosette Text Analytics, 43 2% for Google Cloud, 44 2% for TensiStrength, and 74 7% for the crowdworkers  This means the difference between the performance of the tools and the crowdworkers is significant – more than 30 percent points 
Crowdworkers correctly identified 62% of the neutral, 85% of the positive, and 92% of the negative sentiments  Google Cloud correctly identified 88% of the neutral sentiments, but only 3% of the positive, and 19% of the negative sentiments  TensiStrength correctly identified 87 2% of the neutral sentiments, but 10 5% of the positive, and 8 1% of the negative sentiments  Rosette Text Analytics correctly identified 22 7% of neutral sentiments, 38 1% of negative sentiments and 40 9% of positive sentiments  The lowest and highest CCR pertains to tweets about Trump and Sanders for both Google Cloud and TensiStrength, Trump and Clinton for Rosette Text Analytics, and Clinton and Cruz for crowdworkers  An example of incorrect ELS analysis is shown in Figure FIGREF1 bottom 
Conclusions and Future Work
Our results show that existing NLP systems cannot accurately perform sentiment analysis of political tweets in the dataset we experimented with  Labeling by humans, even non-expert crowdworkers, yields accuracy results that are well above the results of existing automated NLP systems  In future work we will therefore use a crowdworker-labeled dataset to train a new machine-learning based NLP system for tweet analysis  We will ensure that the training data is balanced among classes  Our plan is to use state-of-the-art deep neural networks and compare their performance for entity-level sentiment analysis of political tweets 
Acknowledgments
Partial support of this work by the Hariri Institute for Computing and Computational Science & Engineering at Boston University (to L G ) and a Google Faculty Research Award (to M B  and L G ) is gratefully acknowledged  Additionally, we would like to thank Daniel Khashabi for his help in running the CogComp-NLP Python API and Mike Thelwal for his help with TensiStrength  We are also grateful to the Stanford NLP group for clarifying some of the questions we had with regards to the Stanford NER tool 
Context 2: Introduction
Machine translation has made remarkable progress, and studies claiming it to reach a human parity are starting to appear BIBREF0  However, when evaluating translations of the whole documents rather than isolated sentences, human raters show a stronger preference for human over machine translation BIBREF1  These findings emphasize the need to shift towards context-aware machine translation both from modeling and evaluation perspective 
Most previous work on context-aware NMT assumed that either all the bilingual data is available at the document level BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10 or at least its fraction BIBREF11  But in practical scenarios, document-level parallel data is often scarce, which is one of the challenges when building a context-aware system 
We introduce an approach to context-aware machine translation using only monolingual document-level data  In our setting, a separate monolingual sequence-to-sequence model (DocRepair) is used to correct sentence-level translations of adjacent sentences  The key idea is to use monolingual data to imitate typical inconsistencies between context-agnostic translations of isolated sentences  The DocRepair model is trained to map inconsistent groups of sentences into consistent ones  The consistent groups come from the original training data; the inconsistent groups are obtained by sampling round-trip translations for each isolated sentence 
To validate the performance of our model, we use three kinds of evaluation: the BLEU score, contrastive evaluation of translation of several discourse phenomena BIBREF11, and human evaluation  We show strong improvements for all metrics 
We analyze which discourse phenomena are hard to capture using monolingual data only  Using contrastive test sets for targeted evaluation of several contextual phenomena, we compare the performance of the models trained on round-trip translations and genuine document-level parallel data  Among the four phenomena in the test sets we use (deixis, lexical cohesion, VP ellipsis and ellipsis which affects NP inflection) we find VP ellipsis to be the hardest phenomenon to be captured using round-trip translations 
Our key contributions are as follows:
we introduce the first approach to context-aware machine translation using only monolingual document-level data;
our approach shows substantial improvements in translation quality as measured by BLEU, targeted contrastive evaluation of several discourse phenomena and human evaluation;
we show which discourse phenomena are hard to capture using monolingual data only 
Our Approach: Document-level Repair
We propose a monolingual DocRepair model to correct inconsistencies between sentence-level translations of a context-agnostic MT system  It does not use any states of a trained MT model whose outputs it corrects and therefore can in principle be trained to correct translations from any black-box MT system 
The DocRepair model requires only monolingual document-level data in the target language  It is a monolingual sequence-to-sequence model that maps inconsistent groups of sentences into consistent ones  Consistent groups come from monolingual document-level data  To obtain inconsistent groups, each sentence in a group is replaced with its round-trip translation produced in isolation from context  More formally, forming a training minibatch for the DocRepair model involves the following steps (see also Figure FIGREF9):
sample several groups of sentences from the monolingual data;
for each sentence in a group, (i) translate it using a target-to-source MT model, (ii) sample a translation of this back-translated sentence in the source language using a source-to-target MT model;
using these round-trip translations of isolated sentences, form an inconsistent version of the initial groups;
use inconsistent groups as input for the DocRepair model, consistent ones as output 
At test time, the process of getting document-level translations is two-step (Figure FIGREF10):
produce translations of isolated sentences using a context-agnostic MT model;
apply the DocRepair model to a sequence of context-agnostic translations to correct inconsistencies between translations 
In the scope of the current work, the DocRepair model is the standard sequence-to-sequence Transformer  Sentences in a group are concatenated using a reserved token-separator between sentences  The Transformer is trained to correct these long inconsistent pseudo-sentences into consistent ones  The token-separator is then removed from corrected translations 
Evaluation of Contextual Phenomena
We use contrastive test sets for evaluation of discourse phenomena for English-Russian by BIBREF11  These test sets allow for testing different kinds of phenomena which, as we show, can be captured from monolingual data with varying success  In this section, we provide test sets statistics and briefly describe the tested phenomena  For more details, the reader is referred to BIBREF11 
Evaluation of Contextual Phenomena ::: Test sets
There are four test sets in the suite  Each test set contains contrastive examples  It is specifically designed to test the ability of a system to adapt to contextual information and handle the phenomenon under consideration  Each test instance consists of a true example (a sequence of sentences and their reference translation from the data) and several contrastive translations which differ from the true one only in one specific aspect  All contrastive translations are correct and plausible translations at the sentence level, and only context reveals the inconsistencies between them  The system is asked to score each candidate translation, and we compute the system accuracy as the proportion of times the true translation is preferred to the contrastive ones  Test set statistics are shown in Table TABREF15  The suites for deixis and lexical cohesion are split into development and test sets, with 500 examples from each used for validation purposes and the rest for testing  Convergence of both consistency scores on these development sets and BLEU score on a general development set are used as early stopping criteria in models training  For ellipsis, there is no dedicated development set, so we evaluate on all the ellipsis data and do not use it for development 
Evaluation of Contextual Phenomena ::: Phenomena overview
Deixis Deictic words or phrases, are referential expressions whose denotation depends on context  This includes personal deixis (“I”, “you”), place deixis (“here”, “there”), and discourse deixis, where parts of the discourse are referenced (“that's a good question”)  The test set examples are all related to person deixis, specifically the T-V distinction between informal and formal you (Latin “tu” and “vos”) in the Russian translations, and test for consistency in this respect 
Ellipsis Ellipsis is the omission from a clause of one or more words that are nevertheless understood in the context of the remaining elements  In machine translation, elliptical constructions in the source language pose a problem in two situations  First, if the target language does not allow the same types of ellipsis, requiring the elided material to be predicted from context  Second, if the elided material affects the syntax of the sentence  For example, in Russian the grammatical function of a noun phrase, and thus its inflection, may depend on the elided verb, or, conversely, the verb inflection may depend on the elided subject 
There are two different test sets for ellipsis  One contains examples where a morphological form of a noun group in the last sentence can not be understood without context beyond the sentence level (“ellipsis (infl )” in Table TABREF15)  Another includes cases of verb phrase ellipsis in English, which does not exist in Russian, thus requires predicting the verb when translating into Russian (“ellipsis (VP)” in Table TABREF15) 
Lexical cohesion The test set focuses on reiteration of named entities  Where several translations of a named entity are possible, a model has to prefer consistent translations over inconsistent ones 
Experimental Setup ::: Data preprocessing
We use the publicly available OpenSubtitles2018 corpus BIBREF12 for English and Russian  For a fair comparison with previous work, we train the baseline MT system on the data released by BIBREF11  Namely, our MT system is trained on 6m instances  These are sentence pairs with a relative time overlap of subtitle frames between source and target language subtitles of at least $0 9$ 
We gathered 30m groups of 4 consecutive sentences as our monolingual data  We used only documents not containing groups of sentences from general development and test sets as well as from contrastive test sets  The main results we report are for the model trained on all 30m fragments 
We use the tokenization provided by the corpus and use multi-bleu perl on lowercased data to compute BLEU score  We use beam search with a beam of 4 
Sentences were encoded using byte-pair encoding BIBREF13, with source and target vocabularies of about 32000 tokens  Translation pairs were batched together by approximate sequence length  Each training batch contained a set of translation pairs containing approximately 15000 source tokens  It has been shown that Transformer's performance depends heavily on batch size BIBREF14, and we chose a large batch size to ensure the best performance  In training context-aware models, for early stopping we use both convergence in BLEU score on the general development set and scores on the consistency development sets  After training, we average the 5 latest checkpoints 
Experimental Setup ::: Models
The baseline model, the model used for back-translation, and the DocRepair model are all Transformer base models BIBREF15  More precisely, the number of layers is $N=6$ with $h = 8$ parallel attention layers, or heads  The dimensionality of input and output is $d_{model} = 512$, and the inner-layer of a feed-forward networks has dimensionality $d_{ff}=2048$  We use regularization as described in BIBREF15 
As a second baseline, we use the two-pass CADec model BIBREF11  The first pass produces sentence-level translations  The second pass takes both the first-pass translation and representations of the context sentences as input and returns contextualized translations  CADec requires document-level parallel training data, while DocRepair only needs monolingual training data 
Experimental Setup ::: Generating round-trip translations
On the selected 6m instances we train sentence-level translation models in both directions  To create training data for DocRepair, we proceed as follows  The Russian monolingual data is first translated into English, using the Russian$\rightarrow $English model and beam search with beam size of 4  Then, we use the English$\rightarrow $Russian model to sample translations with temperature of $0{ }5$  For each sentence, we precompute 20 sampled translations and randomly choose one of them when forming a training minibatch for DocRepair  Also, in training, we replace each token in the input with a random one with the probability of $10\%$ 
Experimental Setup ::: Optimizer
As in BIBREF15, we use the Adam optimizer BIBREF16, the parameters are $\beta _1 = 0{ }9$, $\beta _2 = 0{ }98$ and $\varepsilon = 10^{-9}$  We vary the learning rate over the course of training using the formula:
where $warmup\_steps = 16000$ and $scale=4$ 
Results ::: General results
The BLEU scores are provided in Table TABREF24 (we evaluate translations of 4-sentence fragments)  To see which part of the improvement is due to fixing agreement between sentences rather than simply sentence-level post-editing, we train the same repair model at the sentence level  Each sentence in a group is now corrected separately, then they are put back together in a group  One can see that most of the improvement comes from accounting for extra-sentential dependencies  DocRepair outperforms the baseline and CADec by 0 7 BLEU, and its sentence-level repair version by 0 5 BLEU 
Results ::: Consistency results
Scores on the phenomena test sets are provided in Table TABREF26  For deixis, lexical cohesion and ellipsis (infl ) we see substantial improvements over both the baseline and CADec  The largest improvement over CADec (22 5 percentage points) is for lexical cohesion  However, there is a drop of almost 5 percentage points for VP ellipsis  We hypothesize that this is because it is hard to learn to correct inconsistencies in translations caused by VP ellipsis relying on monolingual data alone  Figure FIGREF27(a) shows an example of inconsistency caused by VP ellipsis in English  There is no VP ellipsis in Russian, and when translating auxiliary “did” the model has to guess the main verb  Figure FIGREF27(b) shows steps of generating round-trip translations for the target side of the previous example  When translating from Russian, main verbs are unlikely to be translated as the auxiliary “do” in English, and hence the VP ellipsis is rarely present on the English side  This implies the model trained using the round-trip translations will not be exposed to many VP ellipsis examples in training  We discuss this further in Section SECREF34 
Table TABREF28 provides scores for deixis and lexical cohesion separately for different distances between sentences requiring consistency  It can be seen, that the performance of DocRepair degrades less than that of CADec when the distance between sentences requiring consistency gets larger 
Results ::: Human evaluation
We conduct a human evaluation on random 700 examples from our general test set  We picked only examples where a DocRepair translation is not a full copy of the baseline one 
The annotators were provided an original group of sentences in English and two translations: baseline context-agnostic one and the one corrected by the DocRepair model  Translations were presented in random order with no indication which model they came from  The task is to pick one of the three options: (1) the first translation is better, (2) the second translation is better, (3) the translations are of equal quality  The annotators were asked to avoid the third answer if they are able to give preference to one of the translations  No other guidelines were given 
The results are provided in Table TABREF30  In about $52\%$ of the cases annotators marked translations as having equal quality  Among the cases where one of the translations was marked better than the other, the DocRepair translation was marked better in $73\%$ of the cases  This shows a strong preference of the annotators for corrected translations over the baseline ones 
Varying Training Data
In this section, we discuss the influence of the training data chosen for document-level models  In all experiments, we used the DocRepair model 
Varying Training Data ::: The amount of training data
Table TABREF33 provides BLEU and consistency scores for the DocRepair model trained on different amount of data  We see that even when using a dataset of moderate size (e g , 5m fragments) we can achieve performance comparable to the model trained on a large amount of data (30m fragments)  Moreover, we notice that deixis scores are less sensitive to the amount of training data than lexical cohesion and ellipsis scores  The reason might be that, as we observed in our previous work BIBREF11, inconsistencies in translations due to the presence of deictic words and phrases are more frequent in this dataset than other types of inconsistencies  Also, as we show in Section SECREF7, this is the phenomenon the model learns faster in training 
Varying Training Data ::: One-way vs round-trip translations
In this section, we discuss the limitations of using only monolingual data to model inconsistencies between sentence-level translations  In Section SECREF25 we observed a drop in performance on VP ellipsis for DocRepair compared to CADec, which was trained on parallel data  We hypothesized that this is due to the differences between one-way and round-trip translations, and now we test this hypothesis  To do so, we fix the dataset and vary the way in which the input for DocRepair is generated: round-trip or one-way translations  The latter assumes that document-level data is parallel, and translations are sampled from the source side of the sentences in a group rather than from their back-translations  For parallel data, we take 1 5m parallel instances which were used for CADec training and add 1m instances from our monolingual data  For segments in the parallel part, we either sample translations from the source side or use round-trip translations  The results are provided in Table TABREF35 
The model trained on one-way translations is slightly better than the one trained on round-trip translations  As expected, VP ellipsis is the hardest phenomena to be captured using round-trip translations, and the DocRepair model trained on one-way translated data gains 6% accuracy on this test set  This shows that the DocRepair model benefits from having access to non-synthetic English data  This results in exposing DocRepair at training time to Russian translations which suffer from the same inconsistencies as the ones it will have to correct at test time 
Varying Training Data ::: Filtering: monolingual (no filtering) or parallel
Note that the scores of the DocRepair model trained on 2 5m instances randomly chosen from monolingual data (Table TABREF33) are different from the ones for the model trained on 2 5m instances combined from parallel and monolingual data (Table TABREF35)  For convenience, we show these two in Table TABREF36 
The domain, the dataset these two data samples were gathered from, and the way we generated training data for DocRepair (round-trip translations) are all the same  The only difference lies in how the data was filtered  For parallel data, as in the previous work BIBREF6, we picked only sentence pairs with large relative time overlap of subtitle frames between source-language and target-language subtitles  This is necessary to ensure the quality of translation data: one needs groups of consecutive sentences in the target language where every sentence has a reliable translation 
Table TABREF36 shows that the quality of the model trained on data which came from the parallel part is worse than the one trained on monolingual data  This indicates that requiring each sentence in a group to have a reliable translation changes the distribution of the data, which might be not beneficial for translation quality and provides extra motivation for using monolingual data 
Learning Dynamics
Let us now look into how the process of DocRepair training progresses  Figure FIGREF38 shows how the BLEU scores with the reference translation and with the baseline context-agnostic translation (i e  the input for the DocRepair model) are changing during training  First, the model quickly learns to copy baseline translations: the BLEU score with the baseline is very high  Then it gradually learns to change them, which leads to an improvement in BLEU with the reference translation and a drop in BLEU with the baseline  Importantly, the model is reluctant to make changes: the BLEU score between translations of the converged model and the baseline is 82 5  We count the number of changed sentences in every 4-sentence fragment in the test set and plot the histogram in Figure FIGREF38  In over than 20$\%$ of the cases the model has not changed base translations at all  In almost $40\%$, it modified only one sentence and left the remaining 3 sentences unchanged  The model changed more than half sentences in a group in only $14\%$ of the cases  Several examples of the DocRepair translations are shown in Figure FIGREF43 
Figure FIGREF42 shows how consistency scores are changing in training  For deixis, the model achieves the final quality quite quickly; for the rest, it needs a large number of training steps to converge 
Related Work
Our work is most closely related to two lines of research: automatic post-editing (APE) and document-level machine translation 
Related Work ::: Automatic post-editing
Our model can be regarded as an automatic post-editing system – a system designed to fix systematic MT errors that is decoupled from the main MT system  Automatic post-editing has a long history, including rule-based BIBREF17, statistical BIBREF18 and neural approaches BIBREF19, BIBREF20, BIBREF21 
In terms of architectures, modern approaches use neural sequence-to-sequence models, either multi-source architectures that consider both the original source and the baseline translation BIBREF19, BIBREF20, or monolingual repair systems, as in BIBREF21, which is concurrent work to ours  True post-editing datasets are typically small and expensive to create BIBREF22, hence synthetic training data has been created that uses original monolingual data as output for the sequence-to-sequence model, paired with an automatic back-translation BIBREF23 and/or round-trip translation as its input(s) BIBREF19, BIBREF21 
While previous work on automatic post-editing operated on the sentence level, the main novelty of this work is that our DocRepair model operates on groups of sentences and is thus able to fix consistency errors caused by the context-agnostic baseline MT system  We consider this strategy of sentence-level baseline translation and context-aware monolingual repair attractive when parallel document-level data is scarce 
For training, the DocRepair model only requires monolingual document-level data  While we create synthetic training data via round-trip translation similarly to earlier work BIBREF19, BIBREF21, note that we purposefully use sentence-level MT systems for this to create the types of consistency errors that we aim to fix with the context-aware DocRepair model  Not all types of consistency errors that we want to fix emerge from a round-trip translation, so access to parallel document-level data can be useful (Section SECREF34) 
Related Work ::: Document-level NMT
Neural models of MT that go beyond the sentence-level are an active research area BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF10, BIBREF9, BIBREF11  Typically, the main MT system is modified to take additional context as its input  One limitation of these approaches is that they assume that parallel document-level training data is available 
Closest to our work are two-pass models for document-level NMT BIBREF24, BIBREF11, where a second, context-aware model takes the translation and hidden representations of the sentence-level first-pass model as its input  The second-pass model can in principle be trained on a subset of the parallel training data BIBREF11, somewhat relaxing the assumption that all training data is at the document level 
Our work is different from this previous work in two main respects  Firstly, we show that consistency can be improved with only monolingual document-level training data  Secondly, the DocRepair model is decoupled from the first-pass MT system, which improves its portability 
Conclusions
We introduce the first approach to context-aware machine translation using only monolingual document-level data  We propose a monolingual DocRepair model to correct inconsistencies between sentence-level translations  The model performs automatic post-editing on a sequence of sentence-level translations, refining translations of sentences in context of each other  Our approach results in substantial improvements in translation quality as measured by BLEU, targeted contrastive evaluation of several discourse phenomena and human evaluation  Moreover, we perform error analysis and detect which discourse phenomena are hard to capture using only monolingual document-level data  While in the current work we used text fragments of 4 sentences, in future work we would like to consider longer contexts 
Acknowledgments
We would like to thank the anonymous reviewers for their comments  The authors also thank David Talbot and Yandex Machine Translation team for helpful discussions and inspiration  Ivan Titov acknowledges support of the European Research Council (ERC StG BroadSem 678254) and the Dutch National Science Foundation (NWO VIDI 639 022 518)  Rico Sennrich acknowledges support from the Swiss National Science Foundation (105212_169888), the European Union’s Horizon 2020 research and innovation programme (grant agreement no 825460), and the Royal Society (NAF\R1\180122) 

ground_truth: No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what was the baseline?
Context 4: Introduction
Cancer is one of the leading causes of death in the world, with over 80,000 deaths registered in Canada in 2017 (Canadian Cancer Statistics 2017)  A computer-aided system for cancer diagnosis usually involves a pathologist rendering a descriptive report after examining the tissue glass slides obtained from the biopsy of a patient  A pathology report contains specific analysis of cells and tissues, and other histopathological indicators that are crucial for diagnosing malignancies  An average sized laboratory may produces a large quantity of pathology reports annually (e g , in excess of 50,000), but these reports are written in mostly unstructured text and with no direct link to the tissue sample  Furthermore, the report for each patient is a personalized document and offers very high variability in terminology due to lack of standards and may even include misspellings and missing punctuation, clinical diagnoses interspersed with complex explanations, different terminology to label the same malignancy, and information about multiple carcinoma appearances included in a single report BIBREF0  
In Canada, each Provincial and Territorial Cancer Registry (PTCR) is responsible for collecting the data about cancer diseases and reporting them to Statistics Canada (StatCan)  Every year, Canadian Cancer Registry (CCR) uses the information sources of StatCan to compile an annual report on cancer and tumor diseases  Many countries have their own cancer registry programs  These programs rely on the acquisition of diagnostic, treatment, and outcome information through manual processing and interpretation from various unstructured sources (e g , pathology reports, autopsy/laboratory reports, medical billing summaries)  The manual classification of cancer pathology reports is a challenging, time-consuming task and requires extensive training BIBREF0  
With the continued growth in the number of cancer patients, and the increase in treatment complexity, cancer registries face a significant challenge in manually reviewing the large quantity of reports BIBREF1 , BIBREF0   In this situation, Natural Language Processing (NLP) systems can offer a unique opportunity to automatically encode the unstructured reports into structured data  Since, the registries already have access to the large quantity of historically labeled and encoded reports, a supervised machine learning approach of feature extraction and classification is a compelling direction for making their workflow more effective and streamlined  If successful, such a solution would enable processing reports in much lesser time allowing trained personnel to focus on their research and analysis  However, developing an automated solution with high accuracy and consistency across wide variety of reports is a challenging problem 
For cancer registries, an important piece of information in a pathology report is the associated ICD-O code which describes the patient's histological diagnosis, as described by the World Health Organization's (WHO) International Classification of Diseases for Oncology BIBREF2   Prediction of the primary diagnosis from a pathology report provides a valuable starting point for exploration of machine learning techniques for automated cancer surveillance  A major application for this purpose would be “auto-reporting” based on analysis of whole slide images, the digitization of the biopsy glass slides  Structured, summarized and categorized reports can be associated with the image content when searching in large archives  Such as system would be able to drastically increase the efficiency of diagnostic processes for the majority of cases where in spite of obvious primary diagnosis, still time and effort is required from the pathologists to write a descriptive report 
The primary objective of our study is to analyze the efficacy of existing machine learning approaches for the automated classification of pathology reports into different diagnosis categories  We demonstrate that TF-IDF feature vectors combined with linear SVM or XGBoost classifier can be an effective method for classification of the reports, achieving up to 83% accuracy  We also show that TF-IDF features are capable of identifying important keywords within a pathology report  Furthermore, we have created a new dataset consisting of 1,949 pathology reports across 37 primary diagnoses  Taken together, our exploratory experiments with a newly introduced dataset on pathology reports opens many new opportunities for researchers to develop a scalable and automatic information extraction from unstructured pathology reports 
Background
NLP approaches for information extraction within the biomedical research areas range from rule-based systems BIBREF3 , to domain-specific systems using feature-based classification BIBREF1 , to the recent deep networks for end-to-end feature extraction and classification BIBREF0   NLP has had varied degree of success with free-text pathology reports BIBREF4   Various studies have acknowledge the success of NLP in interpreting pathology reports, especially for classification tasks or extracting a single attribute from a report BIBREF4 , BIBREF5  
The Cancer Text Information Extraction System (caTIES) BIBREF6 is a framework developed in a caBIG project focuses on information extraction from pathology reports  Specifically, caTIES extracts information from surgical pathology reports (SPR) with good precision as well as recall 
Another system known as Open Registry BIBREF7 is capable of filtering the reports with disease codes containing cancer  In BIBREF8 , an approach called Automated Retrieval Console (ARC) is proposed which uses machine learning models to predict the degree of association of a given pathology or radiology with the cancer  The performance ranges from an F-measure of 0 75 for lung cancer to 0 94 for colorectal cancer  However, ARC uses domain-specific rules which hiders with the generalization of the approach to variety of pathology reports 
This research work is inspired by themes emerging in many of the above studies  Specifically, we are evaluating the task of predicting the primary diagnosis from the pathology report  Unlike previous approaches, the system does not rely on custom rule-based knowledge, domain specific features, balanced dataset with fewer number of classes 
Materials and Methods
We assembled a dataset of 1,949 cleaned pathology reports  Each report is associated with one of the 37 different primary diagnoses based on IDC-O codes  The reports are collected from four different body parts or primary sites from multiple patients  The distribution of reports across different primary diagnoses and primary sites is reported in tab:report-distribution  The dataset was developed in three steps as follows 
Collecting pathology reports: The total of 11,112 pathology reports were downloaded from NCI's Genomic Data Commons (GDC) dataset in PDF format BIBREF9   Out of all PDF files, 1,949 reports were selected across multiple patients from four specific primary sites—thymus, testis, lung, and kidney  The selection was primarily made based on the quality of PDF files 
Cleaning reports: The next step was to extract the text content from these reports  Due to the significant time expense of manually re-typing all the pathology reports, we developed a new strategy to prepare our dataset  We applied an Optical Character Recognition (OCR) software to convert the PDF reports to text files

ground_truth:  MT system on the data released by BIBREF11

Transformer base, two-pass CADec model

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the GhostVLAD approach?
Context 3: INTRODUCTION
The idea of language identification is to classify a given audio signal into a particular class using a classification algorithm  Commonly language identification task was done using i-vector systems [1]  A very well known approach for language identification proposed by N  Dahek et al  [1] uses the GMM-UBM model to obtain utterance level features called i-vectors  Recent advances in deep learning [15,16] have helped to improve the language identification task using many different neural network architectures which can be trained efficiently using GPUs for large scale datasets  These neural networks can be configured in various ways to obtain better accuracy for language identification task  Early work on using Deep learning for language Identification was published by Pavel Matejka et al  [2], where they used stacked bottleneck features extracted from deep neural networks for language identification task and showed that the bottleneck features learned by Deep neural networks are better than simple MFCC or PLP features  Later the work by I  Lopez-Moreno et al  [3] from Google showed how to use Deep neural networks to directly map the sequence of MFCC frames into its language class so that we can apply language identification at the frame level  Speech signals will have both spatial and temporal information, but simple DNNs are not able to capture temporal information  Work done by J  Gonzalez-Dominguez et al  [4] by Google developed an LSTM based language identification model which improves the accuracy over the DNN based models  Work done by Alicia et al  [5] used CNNs to improve upon i-vector [1] and other previously developed systems  The work done by Daniel Garcia-Romero et al  [6] has used a combination of Acoustic model trained for speech recognition with Time-delay neural networks where they train the TDNN model by feeding the stacked bottleneck features from acoustic model to predict the language labels at the frame level  Recently X-vectors [7] is proposed for speaker identification task and are shown to outperform all the previous state of the art speaker identification algorithms and are also used for language identification by David Snyder et al  [8] 
In this paper, we explore multiple pooling strategies for language identification task  Mainly we propose Ghost-VLAD based pooling method for language identification  Inspired by the recent work by W  Xie et al  [9] and Y  Zhong et al  [10], we use Ghost-VLAD to improve the accuracy of language identification task for Indian languages  We explore multiple pooling strategies including NetVLAD pooling [11], Average pooling and Statistics pooling( as proposed in X-vectors [7]) and show that Ghost-VLAD pooling is the best pooling strategy for language identification  Our model obtains the best accuracy of 98 24%, and it outperforms all the other previously proposed pooling methods  We conduct all our experiments on 635hrs of audio data for 7 Indian languages collected from $\textbf {All India Radio}$ news channel  The paper is organized as follows  In section 2, we explain the proposed pooling method for language identification  In section 3, we explain our dataset  In section 4, we describe the experiments, and in section 5, we describe the results 
POOLING STRATEGIES
In any language identification model, we want to obtain utterance level representation which has very good language discriminative features  These representations should be compact and should be easily separable by a linear classifier  The idea of any pooling strategy is to pool the frame-level representations into a single utterance level representation  Previous works by [7] have used simple mean and standard deviation aggregation to pool the frame-level features from the top layer of the neural network to obtain the utterance level features  Recently [9] used VLAD based pooling strategy for speaker identification which is inspired from [10] proposed for face recognition  The NetVLAD [11] and Ghost-VLAD [10] methods are proposed for Place recognition and face recognition, respectively, and in both cases, they try to aggregate the local descriptors into global features  In our case, the local descriptors are features extracted from ResNet [15], and the global utterance level feature is obtained by using GhostVLAD pooling  In this section, we explain different pooling methods, including NetVLAD, Ghost-VLAD, Statistic pooling, and Average pooling 
POOLING STRATEGIES ::: NetVLAD pooling
The NetVLAD pooling strategy was initially developed for place recognition by R  Arandjelovic et al  [11]  The NetVLAD is an extension to VLAD [18] approach where they were able to replace the hard assignment based clustering with soft assignment based clustering so that it can be trained with neural network in an end to end fashion  In our case, we use the NetVLAD layer to map N local features of dimension D into a fixed dimensional vector, as shown in Figure 1 (Left side) 
The model takes spectrogram as an input and feeds into CNN based ResNet architecture  The ResNet is used to map the spectrogram into 3D feature map of dimension HxWxD  We convert this 3D feature map into 2D by unfolding H and W dimensions, creating a NxD dimensional feature map, where N=HxW  The NetVLAD layer is kept on top of the feature extraction layer of ResNet, as shown in Figure 1  The NetVLAD now takes N features vectors of dimension D and computes a matrix V of dimension KxD, where K is the number clusters in the NetVLAD layer, and D is the dimension of the feature vector  The matrix V is computed as follows 
Where $w_k$,$b_k$ and $c_k$ are trainable parameters for the cluster $k$ and V(j,k) represents a point in the V matrix for (j,k)th location  The matrix is constructed using the equation (1) where the first term corresponds to the soft assignment of the input $x_i$ to the cluster $c_k$, whereas the second term corresponds to the residual term which tells how far the input descriptor $x_i$ is from the cluster center $c_k$ 
POOLING STRATEGIES ::: GhostVLAD pooling
GhostVLAD is an extension of the NetVLAD approach, which we discussed in the previous section  The GhostVLAD model was proposed for face recognition by Y  Zhong [10]  GhostVLAD works exactly similar to NetVLAD except it adds Ghost clusters along with the NetVLAD clusters  So, now we will have a K+G number of clusters instead of K clusters  Where G is the number of ghost clusters, we want to add (typically 2-4)  The Ghost clusters are added to map any noisy or irrelevant content into ghost clusters and are not included during the feature aggregation stage, as shown in Figure 1 (Right side)  Which means that we compute the matrix V for both normal cluster K and ghost clusters G, but we will not include the vectors belongs to ghost cluster from V during concatenation of the features  Due to which, during feature aggregation stage the contribution of the noisy and unwanted features to normal VLAD clusters are assigned less weights while Ghost clusters absorb most of the weight  We illustrate this in Figure 1(Right Side), where the ghost clusters are shown in red color  We use Ghost clusters when we are computing the V matrix, but they are excluded during the concatenation stage  These concatenated features are fed into the projection layer, followed by softmax to predict the language label 
POOLING STRATEGIES ::: Statistic and average pooling
In statistic pooling, we compute the first and second order statistics of the local features from the top layer of the ResNet model  The 3-D feature map is unfolded to create N features of D dimensions, and then we compute the mean and standard deviation of all these N vectors and get two D dimensional vectors, one for mean and the other for standard deviation  We then concatenate these 2 features and feed it to the projection layer for predicting the language label 
In the Average pooling layer, we compute only the first-order statistics (mean) of the local features from the top layer of the CNN model  The feature map from the top layer of CNN is unfolded to create N features of D dimensions, and then we compute the mean of all these N vectors and get D dimensional representation  We then feed this feature to the projection layer followed by softmax for predicting the language label 
DATASET
In this section, we describe our dataset collection process  We collected and curated around 635Hrs of audio data for 7 Indian languages, namely Kannada, Hindi, Telugu, Malayalam, Bengali, and English  We collected the data from the All India Radio news channel where an actor will be reading news for about 5-10 mins  To cover many speakers for the dataset, we crawled data from 2010 to 2019  Since the audio is very long to train any deep neural network directly, we segment the audio clips into smaller chunks using Voice activity detector  Since the audio clips will have music embedded during the news, we use Inhouse music detection model to remove the music segments from the dataset to make the dataset clean and our dataset contains 635Hrs of clean audio which is divided into 520Hrs of training data containing 165K utterances and 115Hrs of testing data containing 35K utterances  The amount of audio data for training and testing for each of the language is shown in the table bellow 
EXPERIMENTS
In this section, we describe the feature extraction process and network architecture in detail  We use spectral features of 256 dimensions computed using 512 point FFT for every frame, and we add an energy feature for every frame giving us total 257 features for every frame  We use a window size of 25ms and frame shift of 10ms during feature computation  We crop random 5sec audio data from each utterance during training which results in a spectrogram of size 257x500 (features x number of features)  We use these spectrograms as input to our CNN model during training  During testing, we compute the prediction score irrespective of the audio length 
For the network architecture, we use ResNet-34 architecture, as described in [9]  The model uses convolution layers with Relu activations to map the spectrogram of size 257x500 input into 3D feature map of size 1x32x512  This feature cube is converted into 2D feature map of dimension 32x512 and fed into Ghost-VLAD/NetVLAD layer to generate a representation that has more language discrimination capacity  We use Adam optimizer with an initial learning rate of 0 01 and a final learning rate of 0 00001 for training  Each model is trained for 15 epochs with early stopping criteria 
For the baseline, we train an i-vector model using GMM-UBM  We fit a small classifier on top of the generated i-vectors to measure the accuracy  This model is referred as i-vector+svm   To compare our model with the previous state of the art system, we set up the x-vector language identification system [8]  The x-vector model used time-delay neural networks (TDNN) along with statistic-pooling  We use 7 layer TDNN architecture similar to [8] for training  We refer to this model as tdnn+stat-pool   Finally, we set up a Deep LSTM based language identification system similar to [4] but with little modification where we add statistics pooling for the last layers hidden activities before classification  We use 3 layer Bi-LSTM with 256 hidden units at each layer  We refer to this model as LSTM+stat-pool  We train our i-vector+svm and TDNN+stat-pool using Kaldi toolkit  We train our NetVLAD and GhostVLAD experiments using Keras by modifying the code given by [9] for language identification  We train the LSTM+stat-pool and the remaining experiments using Pytorch [14] toolkit, and we will opensource all the codes and data soon 
RESULTS
In this section, we compare the performance of our system with the recent state of the art language identification approaches  We also compare different pooling strategies and finally, compare the robustness of our system to the length of the input spectrogram during training  We visualize the embeddings generated by the GhostVLAD method and conclude that the GhostVLAD embeddings shows very good feature discrimination capabilities 
RESULTS ::: Comparison with different approaches
We compare our system performance with the previous state of the art language identification approaches, as shown in Table 2  The i-vector+svm system is trained using GMM-UBM models to generate i-vectors as proposed in [1]  Once the i-vectors are extracted, we fit SVM classifier to classify the audio  The TDNN+stat-pool system is trained with a statistics pooling layer and is called the x-vector system as proposed by David Snyder et al  [11] and is currently the state of the art language identification approach as far as our knowledge  Our methods outperform the state of the art x-vector system by absolute 1 88% improvement in F1-score, as shown in Table 2 
RESULTS ::: Comparison with different pooling techniques
We compare our approach with different pooling strategies in Table 3  We use ResNet as our base feature extraction network  We keep the base network the same and change only the pooling layers to see which pooling approach performs better for language identification task  Our experiments show that GhostVLAD pooling outperforms all the other pooling methods by achieving 98 43% F1-Score 
RESULTS ::: Duration analysis
To observe the performance of our method with different input durations, we conducted an experiment where we train our model on different input durations  Since our model uses ResNet as the base feature extractor, we need to feed fixed-length spectrogram  We conducted 4 different experiments where we trained the model using 2sec, 3sec, 4sec and 5sec spectrograms containing 200,300,400 and 500 frames respectively  We observed that the model trained with a 5sec spectrogram is the best model, as shown in Table 4 
RESULTS ::: Visualization of embeddings
We visualize the embeddings generated by our approach to see the effectiveness  We extracted 512-dimensional embeddings for our testing data and reduced the dimensionality using t-sne projection  The t-sne plot of the embeddings space is shown in Figure 3  The plot shows that the embeddings learned by our approach has very good discriminative properties
Conclusion
In this work, we use Ghost-VLAD pooling approach that was originally proposed for face recognition to improve language identification performance for Indian languages  We collected and curated 630 hrs audio data from news All India Radio news channel for 7 Indian languages  Our experimental results shows that our approach outperforms the previous state of the art methods by an absolute 1 88% F1-score  We have also conducted experiments with different pooling strategies proposed in the past, and the GhostVLAD pooling approach turns out to be the best approach for aggregating frame-level features into a single utterance level feature  Our experiments also prove that our approach works much better even if the input during training contains smaller durations  Finally, we see that the embeddings generated by our method has very good language discriminative features and helps to improve the performance of language identification 

ground_truth: extension of the NetVLAD, adds Ghost clusters along with the NetVLAD clusters

An extension of NetVLAD which replaces hard assignment-based clustering with soft assignment-based clustering with the additon o fusing Ghost clusters to deal with noisy content.