prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the GhostVLAD approach?
Context 1: INTRODUCTION
The idea of language identification is to classify a given audio signal into a particular class using a classification algorithm  Commonly language identification task was done using i-vector systems [1]  A very well known approach for language identification proposed by N  Dahek et al  [1] uses the GMM-UBM model to obtain utterance level features called i-vectors  Recent advances in deep learning [15,16] have helped to improve the language identification task using many different neural network architectures which can be trained efficiently using GPUs for large scale datasets  These neural networks can be configured in various ways to obtain better accuracy for language identification task  Early work on using Deep learning for language Identification was published by Pavel Matejka et al  [2], where they used stacked bottleneck features extracted from deep neural networks for language identification task and showed that the bottleneck features learned by Deep neural networks are better than simple MFCC or PLP features  Later the work by I  Lopez-Moreno et al  [3] from Google showed how to use Deep neural networks to directly map the sequence of MFCC frames into its language class so that we can apply language identification at the frame level  Speech signals will have both spatial and temporal information, but simple DNNs are not able to capture temporal information  Work done by J  Gonzalez-Dominguez et al  [4] by Google developed an LSTM based language identification model which improves the accuracy over the DNN based models  Work done by Alicia et al  [5] used CNNs to improve upon i-vector [1] and other previously developed systems  The work done by Daniel Garcia-Romero et al  [6] has used a combination of Acoustic model trained for speech recognition with Time-delay neural networks where they train the TDNN model by feeding the stacked bottleneck features from acoustic model to predict the language labels at the frame level  Recently X-vectors [7] is proposed for speaker identification task and are shown to outperform all the previous state of the art speaker identification algorithms and are also used for language identification by David Snyder et al  [8] 
In this paper, we explore multiple pooling strategies for language identification task  Mainly we propose Ghost-VLAD based pooling method for language identification  Inspired by the recent work by W  Xie et al  [9] and Y  Zhong et al  [10], we use Ghost-VLAD to improve the accuracy of language identification task for Indian languages  We explore multiple pooling strategies including NetVLAD pooling [11], Average pooling and Statistics pooling( as proposed in X-vectors [7]) and show that Ghost-VLAD pooling is the best pooling strategy for language identification  Our model obtains the best accuracy of 98 24%, and it outperforms all the other previously proposed pooling methods  We conduct all our experiments on 635hrs of audio data for 7 Indian languages collected from $\textbf {All India Radio}$ news channel  The paper is organized as follows  In section 2, we explain the proposed pooling method for language identification  In section 3, we explain our dataset  In section 4, we describe the experiments, and in section 5, we describe the results 
POOLING STRATEGIES
In any language identification model, we want to obtain utterance level representation which has very good language discriminative features  These representations should be compact and should be easily separable by a linear classifier  The idea of any pooling strategy is to pool the frame-level representations into a single utterance level representation  Previous works by [7] have used simple mean and standard deviation aggregation to pool the frame-level features from the top layer of the neural network to obtain the utterance level features  Recently [9] used VLAD based pooling strategy for speaker identification which is inspired from [10] proposed for face recognition  The NetVLAD [11] and Ghost-VLAD [10] methods are proposed for Place recognition and face recognition, respectively, and in both cases, they try to aggregate the local descriptors into global features  In our case, the local descriptors are features extracted from ResNet [15], and the global utterance level feature is obtained by using GhostVLAD pooling  In this section, we explain different pooling methods, including NetVLAD, Ghost-VLAD, Statistic pooling, and Average pooling 
POOLING STRATEGIES ::: NetVLAD pooling
The NetVLAD pooling strategy was initially developed for place recognition by R  Arandjelovic et al  [11]  The NetVLAD is an extension to VLAD [18] approach where they were able to replace the hard assignment based clustering with soft assignment based clustering so that it can be trained with neural network in an end to end fashion  In our case, we use the NetVLAD layer to map N local features of dimension D into a fixed dimensional vector, as shown in Figure 1 (Left side) 
The model takes spectrogram as an input and feeds into CNN based ResNet architecture  The ResNet is used to map the spectrogram into 3D feature map of dimension HxWxD  We convert this 3D feature map into 2D by unfolding H and W dimensions, creating a NxD dimensional feature map, where N=HxW  The NetVLAD layer is kept on top of the feature extraction layer of ResNet, as shown in Figure 1  The NetVLAD now takes N features vectors of dimension D and computes a matrix V of dimension KxD, where K is the number clusters in the NetVLAD layer, and D is the dimension of the feature vector  The matrix V is computed as follows 
Where $w_k$,$b_k$ and $c_k$ are trainable parameters for the cluster $k$ and V(j,k) represents a point in the V matrix for (j,k)th location  The matrix is constructed using the equation (1) where the first term corresponds to the soft assignment of the input $x_i$ to the cluster $c_k$, whereas the second term corresponds to the residual term which tells how far the input descriptor $x_i$ is from the cluster center $c_k$ 
POOLING STRATEGIES ::: GhostVLAD pooling
GhostVLAD is an extension of the NetVLAD approach, which we discussed in the previous section  The GhostVLAD model was proposed for face recognition by Y  Zhong [10]  GhostVLAD works exactly similar to NetVLAD except it adds Ghost clusters along with the NetVLAD clusters  So, now we will have a K+G number of clusters instead of K clusters  Where G is the number of ghost clusters, we want to add (typically 2-4)  The Ghost clusters are added to map any noisy or irrelevant content into ghost clusters and are not included during the feature aggregation stage, as shown in Figure 1 (Right side)  Which means that we compute the matrix V for both normal cluster K and ghost clusters G, but we will not include the vectors belongs to ghost cluster from V during concatenation of the features  Due to which, during feature aggregation stage the contribution of the noisy and unwanted features to normal VLAD clusters are assigned less weights while Ghost clusters absorb most of the weight  We illustrate this in Figure 1(Right Side), where the ghost clusters are shown in red color  We use Ghost clusters when we are computing the V matrix, but they are excluded during the concatenation stage  These concatenated features are fed into the projection layer, followed by softmax to predict the language label 
POOLING STRATEGIES ::: Statistic and average pooling
In statistic pooling, we compute the first and second order statistics of the local features from the top layer of the ResNet model  The 3-D feature map is unfolded to create N features of D dimensions, and then we compute the mean and standard deviation of all these N vectors and get two D dimensional vectors, one for mean and the other for standard deviation  We then concatenate these 2 features and feed it to the projection layer for predicting the language label 
In the Average pooling layer, we compute only the first-order statistics (mean) of the local features from the top layer of the CNN model  The feature map from the top layer of CNN is unfolded to create N features of D dimensions, and then we compute the mean of all these N vectors and get D dimensional representation  We then feed this feature to the projection layer followed by softmax for predicting the language label 
DATASET
In this section, we describe our dataset collection process  We collected and curated around 635Hrs of audio data for 7 Indian languages, namely Kannada, Hindi, Telugu, Malayalam, Bengali, and English  We collected the data from the All India Radio news channel where an actor will be reading news for about 5-10 mins  To cover many speakers for the dataset, we crawled data from 2010 to 2019  Since the audio is very long to train any deep neural network directly, we segment the audio clips into smaller chunks using Voice activity detector  Since the audio clips will have music embedded during the news, we use Inhouse music detection model to remove the music segments from the dataset to make the dataset clean and our dataset contains 635Hrs of clean audio which is divided into 520Hrs of training data containing 165K utterances and 115Hrs of testing data containing 35K utterances  The amount of audio data for training and testing for each of the language is shown in the table bellow 
EXPERIMENTS
In this section, we describe the feature extraction process and network architecture in detail  We use spectral features of 256 dimensions computed using 512 point FFT for every frame, and we add an energy feature for every frame giving us total 257 features for every frame  We use a window size of 25ms and frame shift of 10ms during feature computation  We crop random 5sec audio data from each utterance during training which results in a spectrogram of size 257x500 (features x number of features)  We use these spectrograms as input to our CNN model during training  During testing, we compute the prediction score irrespective of the audio length 
For the network architecture, we use ResNet-34 architecture, as described in [9]  The model uses convolution layers with Relu activations to map the spectrogram of size 257x500 input into 3D feature map of size 1x32x512  This feature cube is converted into 2D feature map of dimension 32x512 and fed into Ghost-VLAD/NetVLAD layer to generate a representation that has more language discrimination capacity  We use Adam optimizer with an initial learning rate of 0 01 and a final learning rate of 0 00001 for training  Each model is trained for 15 epochs with early stopping criteria 
For the baseline, we train an i-vector model using GMM-UBM  We fit a small classifier on top of the generated i-vectors to measure the accuracy  This model is referred as i-vector+svm   To compare our model with the previous state of the art system, we set up the x-vector language identification system [8]  The x-vector model used time-delay neural networks (TDNN) along with statistic-pooling  We use 7 layer TDNN architecture similar to [8] for training  We refer to this model as tdnn+stat-pool   Finally, we set up a Deep LSTM based language identification system similar to [4] but with little modification where we add statistics pooling for the last layers hidden activities before classification  We use 3 layer Bi-LSTM with 256 hidden units at each layer  We refer to this model as LSTM+stat-pool  We train our i-vector+svm and TDNN+stat-pool using Kaldi toolkit  We train our NetVLAD and GhostVLAD experiments using Keras by modifying the code given by [9] for language identification  We train the LSTM+stat-pool and the remaining experiments using Pytorch [14] toolkit, and we will opensource all the codes and data soon 
RESULTS
In this section, we compare the performance of our system with the recent state of the art language identification approaches  We also compare different pooling strategies and finally, compare the robustness of our system to the length of the input spectrogram during training  We visualize the embeddings generated by the GhostVLAD method and conclude that the GhostVLAD embeddings shows very good feature discrimination capabilities 
RESULTS ::: Comparison with different approaches
We compare our system performance with the previous state of the art language identification approaches, as shown in Table 2  The i-vector+svm system is trained using GMM-UBM models to generate i-vectors as proposed in [1]  Once the i-vectors are extracted, we fit SVM classifier to classify the audio  The TDNN+stat-pool system is trained with a statistics pooling layer and is called the x-vector system as proposed by David Snyder et al  [11] and is currently the state of the art language identification approach as far as our knowledge  Our methods outperform the state of the art x-vector system by absolute 1 88% improvement in F1-score, as shown in Table 2 
RESULTS ::: Comparison with different pooling techniques
We compare our approach with different pooling strategies in Table 3  We use ResNet as our base feature extraction network  We keep the base network the same and change only the pooling layers to see which pooling approach performs better for language identification task  Our experiments show that GhostVLAD pooling outperforms all the other pooling methods by achieving 98 43% F1-Score 
RESULTS ::: Duration analysis
To observe the performance of our method with different input durations, we conducted an experiment where we train our model on different input durations  Since our model uses ResNet as the base feature extractor, we need to feed fixed-length spectrogram  We conducted 4 different experiments where we trained the model using 2sec, 3sec, 4sec and 5sec spectrograms containing 200,300,400 and 500 frames respectively  We observed that the model trained with a 5sec spectrogram is the best model, as shown in Table 4 
RESULTS ::: Visualization of embeddings
We visualize the embeddings generated by our approach to see the effectiveness  We extracted 512-dimensional embeddings for our testing data and reduced the dimensionality using t-sne projection  The t-sne plot of the embeddings space is shown in Figure 3  The plot shows that the embeddings learned by our approach has very good discriminative properties
Conclusion
In this work, we use Ghost-VLAD pooling approach that was originally proposed for face recognition to improve language identification performance for Indian languages  We collected and curated 630 hrs audio data from news All India Radio news channel for 7 Indian languages  Our experimental results shows that our approach outperforms the previous state of the art methods by an absolute 1 88% F1-score  We have also conducted experiments with different pooling strategies proposed in the past, and the GhostVLAD pooling approach turns out to be the best approach for aggregating frame-level features into a single utterance level feature  Our experiments also prove that our approach works much better even if the input during training contains smaller durations  Finally, we see that the embeddings generated by our method has very good language discriminative features and helps to improve the performance of language identification 

ground_truth: extension of the NetVLAD, adds Ghost clusters along with the NetVLAD clusters

An extension of NetVLAD which replaces hard assignment-based clustering with soft assignment-based clustering with the additon o fusing Ghost clusters to deal with noisy content.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what previous RNN models do they compare with?
Context 2: Introduction
Long short term memory (LSTM) units BIBREF1 are popular for many sequence modeling tasks and are used extensively in language modeling  A key to their success is their articulated gating structure, which allows for more control over the information passed along the recurrence  However, despite the sophistication of the gating mechanisms employed in LSTMs and similar recurrent units, the input and context vectors are treated with simple linear transformations prior to gating  Non-linear transformations such as convolutions BIBREF2 have been used, but these have not achieved the performance of well regularized LSTMs for language modeling BIBREF3  
A natural way to improve the expressiveness of linear transformations is to increase the number of dimensions of the input and context vectors, but this comes with a significant increase in the number of parameters which may limit generalizability  An example is shown in Figure FIGREF1 , where LSTMs performance decreases with the increase in dimensions of the input and context vectors  Moreover, the semantics of the input and context vectors are different, suggesting that each may benefit from specialized treatment 
Guided by these insights, we introduce a new recurrent unit, the Pyramidal Recurrent Unit (PRU), which is based on the LSTM gating structure  Figure FIGREF2 provides an overview of the PRU  At the heart of the PRU is the pyramidal transformation (PT), which uses subsampling to effect multiple views of the input vector  The subsampled representations are combined in a pyramidal fusion structure, resulting in richer interactions between the individual dimensions of the input vector than is possible with a linear transformation  Context vectors, which have already undergone this transformation in the previous cell, are modified with a grouped linear transformation (GLT) which allows the network to learn latent representations in high dimensional space with fewer parameters and better generalizability (see Figure FIGREF1 ) 
We show that PRUs can better model contextual information and demonstrate performance gains on the task of language modeling  The PRU improves the perplexity of the current state-of-the-art language model BIBREF0 by up to 1 3 points, reaching perplexities of 56 56 and 64 53 on the Penn Treebank and WikiText2 datasets while learning 15-20% fewer parameters  Replacing an LSTM with a PRU results in improvements in perplexity across a variety of experimental settings  We provide detailed ablations which motivate the design of the PRU architecture, as well as detailed analysis of the effect of the PRU on other components of the language model 
Related work
Multiple methods, including a variety of gating structures and transformations, have been proposed to improve the performance of recurrent neural networks (RNNs)  We first describe these approaches and then provide an overview of recent work in language modeling 
Pyramidal Recurrent Units
We introduce Pyramidal Recurrent Units (PRUs), a new RNN architecture which improves modeling of context by allowing for higher dimensional vector representations while learning fewer parameters  Figure FIGREF2 provides an overview of PRU  We first elaborate on the details of the pyramidal transformation and the grouped linear transformation  We then describe our recurrent unit, PRU 
Pyramidal transformation for input
The basic transformation in many recurrent units is a linear transformation INLINEFORM0 defined as: DISPLAYFORM0
where INLINEFORM0 are learned weights that linearly map INLINEFORM1 to INLINEFORM2   To simplify notation, we omit the biases 
Motivated by successful applications of sub-sampling in computer vision (e g , BIBREF22 , BIBREF23 , BIBREF9 , BIBREF24 ), we subsample input vector INLINEFORM0 into INLINEFORM1 pyramidal levels to achieve representation of the input vector at multiple scales  This sub-sampling operation produces INLINEFORM2 vectors, represented as INLINEFORM3 , where INLINEFORM4 is the sampling rate and INLINEFORM5   We learn scale-specific transformations INLINEFORM6 for each INLINEFORM7   The transformed subsamples are concatenated to produce the pyramidal analog to INLINEFORM8 , here denoted as INLINEFORM9 : DISPLAYFORM0
where INLINEFORM0 indicates concatenation  We note that pyramidal transformation with INLINEFORM1 is the same as the linear transformation 
To improve gradient flow inside the recurrent unit, we combine the input and output using an element-wise sum (when dimension matches) to produce residual analog of pyramidal transformation, as shown in Figure FIGREF2 BIBREF25  
We sub-sample the input vector INLINEFORM0 into INLINEFORM1 pyramidal levels using the kernel-based approach BIBREF8 , BIBREF9   Let us assume that we have a kernel INLINEFORM2 with INLINEFORM3 elements  Then, the input vector INLINEFORM4 can be sub-sampled as: DISPLAYFORM0
where INLINEFORM0 represents the stride and INLINEFORM1  
The number of parameters learned by the linear transformation and the pyramidal transformation with INLINEFORM0 pyramidal levels to map INLINEFORM1 to INLINEFORM2 are INLINEFORM3 and INLINEFORM4 respectively  Thus, pyramidal transformation reduces the parameters of a linear transformation by a factor of INLINEFORM5   For example, the pyramidal transformation (with INLINEFORM6 and INLINEFORM7 ) learns INLINEFORM8 fewer parameters than the linear transformation 
Grouped linear transformation for context
Many RNN architectures apply linear transformations to both the input and context vector  However, this may not be ideal due to the differing semantics of each vector  In many NLP applications including language modeling, the input vector is a dense word embedding which is shared across all contexts for a given word in a dataset  In contrast, the context vector is highly contextualized by the current sequence  The differences between the input and context vector motivate their separate treatment in the PRU architecture 
The weights learned using the linear transformation (Eq  EQREF9 ) are reused over multiple time steps, which makes them prone to over-fitting BIBREF26   To combat over-fitting, various methods, such as variational dropout BIBREF26 and weight dropout BIBREF0 , have been proposed to regularize these recurrent connections  To further improve generalization abilities while simultaneously enabling the recurrent unit to learn representations at very high dimensional space, we propose to use grouped linear transformation (GLT) instead of standard linear transformation for recurrent connections BIBREF27   While pyramidal and linear transformations can be applied to transform context vectors, our experimental results in Section SECREF39 suggests that GLTs are more effective 
The linear transformation INLINEFORM0 maps INLINEFORM1 linearly to INLINEFORM2   Grouped linear transformations break the linear interactions by factoring the linear transformation into two steps  First, a GLT splits the input vector INLINEFORM3 into INLINEFORM4 smaller groups such that INLINEFORM5   Second, a linear transformation INLINEFORM6 is applied to map INLINEFORM7 linearly to INLINEFORM8 , for each INLINEFORM9   The INLINEFORM10 resultant output vectors INLINEFORM11 are concatenated to produce the final output vector INLINEFORM12   DISPLAYFORM0
GLTs learn representations at low dimensionality  Therefore, a GLT requires INLINEFORM0 fewer parameters than the linear transformation  We note that GLTs are subset of linear transformations  In a linear transformation, each neuron receives an input from each element in the input vector while in a GLT, each neuron receives an input from a subset of the input vector  Therefore, GLT is the same as a linear transformation when INLINEFORM1  
Pyramidal Recurrent Unit
We extend the basic gating architecture of LSTM with the pyramidal and grouped linear transformations outlined above to produce the Pyramidal Recurrent Unit (PRU), whose improved sequence modeling capacity is evidenced in Section SECREF4  
At time INLINEFORM0 , the PRU combines the input vector INLINEFORM1 and the previous context vector (or previous hidden state vector) INLINEFORM2 using the following transformation function as: DISPLAYFORM0
where INLINEFORM0 indexes the various gates in the LSTM model, and INLINEFORM1 and INLINEFORM2 represents the pyramidal and grouped linear transformations defined in Eqns  EQREF10 and EQREF15 , respectively 
We will now incorporate INLINEFORM0 into LSTM gating architecture to produce PRU  At time INLINEFORM1 , a PRU cell takes INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 as inputs to produce forget INLINEFORM5 , input INLINEFORM6 , output INLINEFORM7 , and content INLINEFORM8 gate signals  The inputs are combined with these gate signals to produce context vector INLINEFORM9 and cell state INLINEFORM10   Mathematically, the PRU with the LSTM gating architecture can be defined as: DISPLAYFORM0
where INLINEFORM0 represents the element-wise multiplication operation, and INLINEFORM1 and INLINEFORM2 are the sigmoid and hyperbolic tangent activation functions  We note that LSTM is a special case of PRU when INLINEFORM3 = INLINEFORM4 =1 
Experiments
To showcase the effectiveness of the PRU, we evaluate the performance on two standard datasets for word-level language modeling and compare with state-of-the-art methods  Additionally, we provide a detailed examination of the PRU and its behavior on the language modeling tasks 
Set-up
Following recent works, we compare on two widely used datasets, the Penn Treebank (PTB) BIBREF28 as prepared by BIBREF29 and WikiText2 (WT-2) BIBREF20   For both datasets, we follow the same training, validation, and test splits as in BIBREF0  
We extend the language model, AWD-LSTM BIBREF0 , by replacing LSTM layers with PRU  Our model uses 3-layers of PRU with an embedding size of 400  The number of parameters learned by state-of-the-art methods vary from 18M to 66M with majority of the methods learning about 22M to 24M parameters on the PTB dataset  For a fair comparison with state-of-the-art methods, we fix the model size to 19M and vary the value of INLINEFORM0 and hidden layer sizes so that total number of learned parameters is similar across different configurations  We use 1000, 1200, and 1400 as hidden layer sizes for values of INLINEFORM1 =1,2, and 4, respectively  We use the same settings for the WT-2 dataset  We set the number of pyramidal levels INLINEFORM2 to two in our experiments and use average pooling for sub-sampling  These values are selected based on our ablation experiments on the validation set (Section SECREF39 )  We measure the performance of our models in terms of word-level perplexity  We follow the same training strategy as in BIBREF0  
To understand the effect of regularization methods on the performance of PRUs, we perform experiments under two different settings: (1) Standard dropout: We use a standard dropout BIBREF12 with probability of 0 5 after embedding layer, the output between LSTM layers, and the output of final LSTM layer  (2) Advanced dropout: We use the same dropout techniques with the same dropout values as in BIBREF0   We call this model as AWD-PRU 
Results
Table TABREF23 compares the performance of the PRU with state-of-the-art methods  We can see that the PRU achieves the best performance with fewer parameters 
PRUs achieve either the same or better performance than LSTMs  In particular, the performance of PRUs improves with the increasing value of INLINEFORM0   At INLINEFORM1 , PRUs outperform LSTMs by about 4 points on the PTB dataset and by about 3 points on the WT-2 dataset  This is explained in part by the regularization effect of the grouped linear transformation (Figure FIGREF1 )  With grouped linear and pyramidal transformations, PRUs learn rich representations at very high dimensional space while learning fewer parameters  On the other hand, LSTMs overfit to the training data at such high dimensions and learn INLINEFORM2 to INLINEFORM3 more parameters than PRUs 
With the advanced dropouts, the performance of PRUs improves by about 4 points on the PTB dataset and 7 points on the WT-2 dataset  This further improves with finetuning on the PTB (about 2 points) and WT-2 (about 1 point) datasets 
For similar number of parameters, the PRU with standard dropout outperforms most of the state-of-the-art methods by large margin on the PTB dataset (e g  RAN BIBREF7 by 16 points with 4M less parameters, QRNN BIBREF33 by 16 points with 1M more parameters, and NAS BIBREF31 by 1 58 points with 6M less parameters)  With advanced dropouts, the PRU delivers the best performance  On both datasets, the PRU improves the perplexity by about 1 point while learning 15-20% fewer parameters 
PRU is a drop-in replacement for LSTM, therefore, it can improve language models with modern inference techniques such as dynamic evaluation BIBREF21   When we evaluate PRU-based language models (only with standard dropout) with dynamic evaluation on the PTB test set, the perplexity of PRU ( INLINEFORM0 ) improves from 62 42 to 55 23 while the perplexity of an LSTM ( INLINEFORM1 ) with similar settings improves from 66 29 to 58 79; suggesting that modern inference techniques are equally applicable to PRU-based language models 
Analysis
It is shown above that the PRU can learn representations at higher dimensionality with more generalization power, resulting in performance gains for language modeling  A closer analysis of the impact of the PRU in a language modeling system reveals several factors that help explain how the PRU achieves these gains 
As exemplified in Table TABREF34 , the PRU tends toward more confident decisions, placing more of the probability mass on the top next-word prediction than the LSTM  To quantify this effect, we calculate the entropy of the next-token distribution for both the PRU and the LSTM using 3687 contexts from the PTB validation set  Figure FIGREF32 shows a histogram of the entropies of the distribution, where bins of size 0 23 are used to effect categories  We see that the PRU more often produces lower entropy distributions corresponding to higher confidences for next-token choices  This is evidenced by the mass of the red PRU curve lying in the lower entropy ranges compared to the blue LSTM's curve  The PRU can produce confident decisions in part because more information is encoded in the higher dimensional context vectors 
The PRU has the ability to model individual words at different resolutions through the pyramidal transform; which provides multiple paths for the gradient to the embedding layer (similar to multi-task learning) and improves the flow of information  When considering the embeddings by part of speech, we find that the pyramid level 1 embeddings exhibit higher variance than the LSTM across all POS categories (Figure FIGREF33 ), and that pyramid level 2 embeddings show extremely low variance  We hypothesize that the LSTM must encode both coarse group similarities and individual word differences into the same vector space, reducing the space between individual words of the same category  The PRU can rely on the subsampled embeddings to account for coarse-grained group similarities, allowing for finer individual word distinctions in the embedding layer  This hypothesis is strengthened by the entropy results described above: a model which can make finer distinctions between individual words can more confidently assign probability mass  A model that cannot make these distinctions, such as the LSTM, must spread its probability mass across a larger class of similar words 
Saliency analysis using gradients help identify relevant words in a test sequence that contribute to the prediction BIBREF34 , BIBREF35 , BIBREF36   These approaches compute the relevance as the squared norm of the gradients obtained through back-propagation  Table TABREF34 visualizes the heatmaps for different sequences  PRUs, in general, give more relevance to contextual words than LSTMs, such as southeast (sample 1), cost (sample 2), face (sample 4), and introduced (sample 5), which help in making more confident decisions  Furthermore, when gradients during back-propagation are visualized BIBREF37 (Table TABREF34 ), we find that PRUs have better gradient coverage than LSTMs, suggesting PRUs use more features than LSTMs that contributes to the decision  This also suggests that PRUs update more parameters at each iteration which results in faster training  Language model in BIBREF0 takes 500 and 750 epochs to converge with PRU and LSTM as a recurrent unit, respectively 
Ablation studies
In this section, we provide a systematic analysis of our design choices  Our training methodology is the same as described in Section SECREF19 with the standard dropouts  For a thorough understanding of our design choices, we use a language model with a single layer of PRU and fix the size of embedding and hidden layers to 600  The word-level perplexities are reported on the validation sets of the PTB and the WT-2 datasets 
The two hyper-parameters that control the trade-off between performance and number of parameters in PRUs are the number of pyramidal levels INLINEFORM0 and groups INLINEFORM1   Figure FIGREF35 provides a trade-off between perplexity and recurrent unit (RU) parameters 
Variable INLINEFORM0 and fixed INLINEFORM1 : When we increase the number of pyramidal levels INLINEFORM2 at a fixed value of INLINEFORM3 , the performance of the PRU drops by about 1 to 4 points while reducing the total number of recurrent unit parameters by up to 15%  We note that the PRU with INLINEFORM4 at INLINEFORM5 delivers similar performance as the LSTM while learning about 15% fewer recurrent unit parameters 
Fixed INLINEFORM0 and variable INLINEFORM1 : When we vary the value of INLINEFORM2 at fixed number of pyramidal levels INLINEFORM3 , the total number of recurrent unit parameters decreases significantly with a minimal impact on the perplexity  For example, PRUs with INLINEFORM4 and INLINEFORM5 learns 77% fewer recurrent unit parameters while its perplexity (lower is better) increases by about 12% in comparison to LSTMs  Moreover, the decrease in number of parameters at higher value of INLINEFORM6 enables PRUs to learn the representations in high dimensional space with better generalizability (Table TABREF23 ) 
Table TABREF43 shows the impact of different transformations of the input vector INLINEFORM0 and the context vector INLINEFORM1   We make following observations: (1) Using the pyramidal transformation for the input vectors improves the perplexity by about 1 point on both the PTB and WT-2 datasets while reducing the number of recurrent unit parameters by about 14% (see R1 and R4)  We note that the performance of the PRU drops by up to 1 point when residual connections are not used (R4 and R6)  (2) Using the grouped linear transformation for context vectors reduces the total number of recurrent unit parameters by about 75% while the performance drops by about 11% (see R3 and R4)  When we use the pyramidal transformation instead of the linear transformation, the performance drops by up to 2% while there is no significant drop in the number of parameters (R4 and R5) 
We set sub-sampling kernel INLINEFORM0 (Eq  EQREF12 ) with stride INLINEFORM1 and size of 3 ( INLINEFORM2 ) in four different ways: (1) Skip: We skip every other element in the input vector  (2) Convolution: We initialize the elements of INLINEFORM3 randomly from normal distribution and learn them during training the model  We limit the output values between -1 and 1 using INLINEFORM4 activation function to make training stable  (3) Avg  pool: We initialize the elements of INLINEFORM5 to INLINEFORM6   (4) Max pool: We select the maximum value in the kernel window INLINEFORM7  
Table TABREF45 compares the performance of the PRU with different sampling methods  Average pooling performs the best while skipping give comparable performance  Both of these methods enable the network to learn richer word representations while representing the input vector in different forms, thus delivering higher performance  Surprisingly, a convolution-based sub-sampling method does not perform as well as the averaging method  The INLINEFORM0 function used after convolution limits the range of output values which are further limited by the LSTM gating structure, thereby impeding in the flow of information inside the cell  Max pooling forces the network to learn representations from high magnitude elements, thus distinguishing features between elements vanishes, resulting in poor performance 
Conclusion
We introduce the Pyramidal Recurrent Unit, which better model contextual information by admitting higher dimensional representations with good generalizability  When applied to the task of language modeling, PRUs improve perplexity across several settings, including recent state-of-the-art systems  Our analysis shows that the PRU improves the flow of gradient and expand the word embedding subspace, resulting in more confident decisions  Here we have shown improvements for language modeling  In future, we plan to study the performance of PRUs on different tasks, including machine translation and question answering  In addition, we will study the performance of the PRU on language modeling with more recent inference techniques, such as dynamic evaluation and mixture of softmax 
Acknowledgments
This research was supported by NSF (IIS 1616112, III 1703166), Allen Distinguished Investigator Award, and gifts from Allen Institute for AI, Google, Amazon, and Bloomberg  We are grateful to Aaron Jaech, Hannah Rashkin, Mandar Joshi, Aniruddha Kembhavi, and anonymous reviewers for their helpful comments 

ground_truth: Variational LSTM, CharCNN, Pointer Sentinel-LSTM, RHN, NAS Cell, SRU, QRNN, RAN, 4-layer skip-connection LSTM, AWD-LSTM, Quantized LSTM

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what was the baseline?
Context 3: Introduction
Machine translation has made remarkable progress, and studies claiming it to reach a human parity are starting to appear BIBREF0  However, when evaluating translations of the whole documents rather than isolated sentences, human raters show a stronger preference for human over machine translation BIBREF1  These findings emphasize the need to shift towards context-aware machine translation both from modeling and evaluation perspective 
Most previous work on context-aware NMT assumed that either all the bilingual data is available at the document level BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10 or at least its fraction BIBREF11  But in practical scenarios, document-level parallel data is often scarce, which is one of the challenges when building a context-aware system 
We introduce an approach to context-aware machine translation using only monolingual document-level data  In our setting, a separate monolingual sequence-to-sequence model (DocRepair) is used to correct sentence-level translations of adjacent sentences  The key idea is to use monolingual data to imitate typical inconsistencies between context-agnostic translations of isolated sentences  The DocRepair model is trained to map inconsistent groups of sentences into consistent ones  The consistent groups come from the original training data; the inconsistent groups are obtained by sampling round-trip translations for each isolated sentence 
To validate the performance of our model, we use three kinds of evaluation: the BLEU score, contrastive evaluation of translation of several discourse phenomena BIBREF11, and human evaluation  We show strong improvements for all metrics 
We analyze which discourse phenomena are hard to capture using monolingual data only  Using contrastive test sets for targeted evaluation of several contextual phenomena, we compare the performance of the models trained on round-trip translations and genuine document-level parallel data  Among the four phenomena in the test sets we use (deixis, lexical cohesion, VP ellipsis and ellipsis which affects NP inflection) we find VP ellipsis to be the hardest phenomenon to be captured using round-trip translations 
Our key contributions are as follows:
we introduce the first approach to context-aware machine translation using only monolingual document-level data;
our approach shows substantial improvements in translation quality as measured by BLEU, targeted contrastive evaluation of several discourse phenomena and human evaluation;
we show which discourse phenomena are hard to capture using monolingual data only 
Our Approach: Document-level Repair
We propose a monolingual DocRepair model to correct inconsistencies between sentence-level translations of a context-agnostic MT system  It does not use any states of a trained MT model whose outputs it corrects and therefore can in principle be trained to correct translations from any black-box MT system 
The DocRepair model requires only monolingual document-level data in the target language  It is a monolingual sequence-to-sequence model that maps inconsistent groups of sentences into consistent ones  Consistent groups come from monolingual document-level data  To obtain inconsistent groups, each sentence in a group is replaced with its round-trip translation produced in isolation from context  More formally, forming a training minibatch for the DocRepair model involves the following steps (see also Figure FIGREF9):
sample several groups of sentences from the monolingual data;
for each sentence in a group, (i) translate it using a target-to-source MT model, (ii) sample a translation of this back-translated sentence in the source language using a source-to-target MT model;
using these round-trip translations of isolated sentences, form an inconsistent version of the initial groups;
use inconsistent groups as input for the DocRepair model, consistent ones as output 
At test time, the process of getting document-level translations is two-step (Figure FIGREF10):
produce translations of isolated sentences using a context-agnostic MT model;
apply the DocRepair model to a sequence of context-agnostic translations to correct inconsistencies between translations 
In the scope of the current work, the DocRepair model is the standard sequence-to-sequence Transformer  Sentences in a group are concatenated using a reserved token-separator between sentences  The Transformer is trained to correct these long inconsistent pseudo-sentences into consistent ones  The token-separator is then removed from corrected translations 
Evaluation of Contextual Phenomena
We use contrastive test sets for evaluation of discourse phenomena for English-Russian by BIBREF11  These test sets allow for testing different kinds of phenomena which, as we show, can be captured from monolingual data with varying success  In this section, we provide test sets statistics and briefly describe the tested phenomena  For more details, the reader is referred to BIBREF11 
Evaluation of Contextual Phenomena ::: Test sets
There are four test sets in the suite  Each test set contains contrastive examples  It is specifically designed to test the ability of a system to adapt to contextual information and handle the phenomenon under consideration  Each test instance consists of a true example (a sequence of sentences and their reference translation from the data) and several contrastive translations which differ from the true one only in one specific aspect  All contrastive translations are correct and plausible translations at the sentence level, and only context reveals the inconsistencies between them  The system is asked to score each candidate translation, and we compute the system accuracy as the proportion of times the true translation is preferred to the contrastive ones  Test set statistics are shown in Table TABREF15  The suites for deixis and lexical cohesion are split into development and test sets, with 500 examples from each used for validation purposes and the rest for testing  Convergence of both consistency scores on these development sets and BLEU score on a general development set are used as early stopping criteria in models training  For ellipsis, there is no dedicated development set, so we evaluate on all the ellipsis data and do not use it for development 
Evaluation of Contextual Phenomena ::: Phenomena overview
Deixis Deictic words or phrases, are referential expressions whose denotation depends on context  This includes personal deixis (“I”, “you”), place deixis (“here”, “there”), and discourse deixis, where parts of the discourse are referenced (“that's a good question”)  The test set examples are all related to person deixis, specifically the T-V distinction between informal and formal you (Latin “tu” and “vos”) in the Russian translations, and test for consistency in this respect 
Ellipsis Ellipsis is the omission from a clause of one or more words that are nevertheless understood in the context of the remaining elements  In machine translation, elliptical constructions in the source language pose a problem in two situations  First, if the target language does not allow the same types of ellipsis, requiring the elided material to be predicted from context  Second, if the elided material affects the syntax of the sentence  For example, in Russian the grammatical function of a noun phrase, and thus its inflection, may depend on the elided verb, or, conversely, the verb inflection may depend on the elided subject 
There are two different test sets for ellipsis  One contains examples where a morphological form of a noun group in the last sentence can not be understood without context beyond the sentence level (“ellipsis (infl )” in Table TABREF15)  Another includes cases of verb phrase ellipsis in English, which does not exist in Russian, thus requires predicting the verb when translating into Russian (“ellipsis (VP)” in Table TABREF15) 
Lexical cohesion The test set focuses on reiteration of named entities  Where several translations of a named entity are possible, a model has to prefer consistent translations over inconsistent ones 
Experimental Setup ::: Data preprocessing
We use the publicly available OpenSubtitles2018 corpus BIBREF12 for English and Russian  For a fair comparison with previous work, we train the baseline MT system on the data released by BIBREF11  Namely, our MT system is trained on 6m instances  These are sentence pairs with a relative time overlap of subtitle frames between source and target language subtitles of at least $0 9$ 
We gathered 30m groups of 4 consecutive sentences as our monolingual data  We used only documents not containing groups of sentences from general development and test sets as well as from contrastive test sets  The main results we report are for the model trained on all 30m fragments 
We use the tokenization provided by the corpus and use multi-bleu perl on lowercased data to compute BLEU score  We use beam search with a beam of 4 
Sentences were encoded using byte-pair encoding BIBREF13, with source and target vocabularies of about 32000 tokens  Translation pairs were batched together by approximate sequence length  Each training batch contained a set of translation pairs containing approximately 15000 source tokens  It has been shown that Transformer's performance depends heavily on batch size BIBREF14, and we chose a large batch size to ensure the best performance  In training context-aware models, for early stopping we use both convergence in BLEU score on the general development set and scores on the consistency development sets  After training, we average the 5 latest checkpoints 
Experimental Setup ::: Models
The baseline model, the model used for back-translation, and the DocRepair model are all Transformer base models BIBREF15  More precisely, the number of layers is $N=6$ with $h = 8$ parallel attention layers, or heads  The dimensionality of input and output is $d_{model} = 512$, and the inner-layer of a feed-forward networks has dimensionality $d_{ff}=2048$  We use regularization as described in BIBREF15 
As a second baseline, we use the two-pass CADec model BIBREF11  The first pass produces sentence-level translations  The second pass takes both the first-pass translation and representations of the context sentences as input and returns contextualized translations  CADec requires document-level parallel training data, while DocRepair only needs monolingual training data 
Experimental Setup ::: Generating round-trip translations
On the selected 6m instances we train sentence-level translation models in both directions  To create training data for DocRepair, we proceed as follows  The Russian monolingual data is first translated into English, using the Russian$\rightarrow $English model and beam search with beam size of 4  Then, we use the English$\rightarrow $Russian model to sample translations with temperature of $0{ }5$  For each sentence, we precompute 20 sampled translations and randomly choose one of them when forming a training minibatch for DocRepair  Also, in training, we replace each token in the input with a random one with the probability of $10\%$ 
Experimental Setup ::: Optimizer
As in BIBREF15, we use the Adam optimizer BIBREF16, the parameters are $\beta _1 = 0{ }9$, $\beta _2 = 0{ }98$ and $\varepsilon = 10^{-9}$  We vary the learning rate over the course of training using the formula:
where $warmup\_steps = 16000$ and $scale=4$ 
Results ::: General results
The BLEU scores are provided in Table TABREF24 (we evaluate translations of 4-sentence fragments)  To see which part of the improvement is due to fixing agreement between sentences rather than simply sentence-level post-editing, we train the same repair model at the sentence level  Each sentence in a group is now corrected separately, then they are put back together in a group  One can see that most of the improvement comes from accounting for extra-sentential dependencies  DocRepair outperforms the baseline and CADec by 0 7 BLEU, and its sentence-level repair version by 0 5 BLEU 
Results ::: Consistency results
Scores on the phenomena test sets are provided in Table TABREF26  For deixis, lexical cohesion and ellipsis (infl ) we see substantial improvements over both the baseline and CADec  The largest improvement over CADec (22 5 percentage points) is for lexical cohesion  However, there is a drop of almost 5 percentage points for VP ellipsis  We hypothesize that this is because it is hard to learn to correct inconsistencies in translations caused by VP ellipsis relying on monolingual data alone  Figure FIGREF27(a) shows an example of inconsistency caused by VP ellipsis in English  There is no VP ellipsis in Russian, and when translating auxiliary “did” the model has to guess the main verb  Figure FIGREF27(b) shows steps of generating round-trip translations for the target side of the previous example  When translating from Russian, main verbs are unlikely to be translated as the auxiliary “do” in English, and hence the VP ellipsis is rarely present on the English side  This implies the model trained using the round-trip translations will not be exposed to many VP ellipsis examples in training  We discuss this further in Section SECREF34 
Table TABREF28 provides scores for deixis and lexical cohesion separately for different distances between sentences requiring consistency  It can be seen, that the performance of DocRepair degrades less than that of CADec when the distance between sentences requiring consistency gets larger 
Results ::: Human evaluation
We conduct a human evaluation on random 700 examples from our general test set  We picked only examples where a DocRepair translation is not a full copy of the baseline one 
The annotators were provided an original group of sentences in English and two translations: baseline context-agnostic one and the one corrected by the DocRepair model  Translations were presented in random order with no indication which model they came from  The task is to pick one of the three options: (1) the first translation is better, (2) the second translation is better, (3) the translations are of equal quality  The annotators were asked to avoid the third answer if they are able to give preference to one of the translations  No other guidelines were given 
The results are provided in Table TABREF30  In about $52\%$ of the cases annotators marked translations as having equal quality  Among the cases where one of the translations was marked better than the other, the DocRepair translation was marked better in $73\%$ of the cases  This shows a strong preference of the annotators for corrected translations over the baseline ones 
Varying Training Data
In this section, we discuss the influence of the training data chosen for document-level models  In all experiments, we used the DocRepair model 
Varying Training Data ::: The amount of training data
Table TABREF33 provides BLEU and consistency scores for the DocRepair model trained on different amount of data  We see that even when using a dataset of moderate size (e g , 5m fragments) we can achieve performance comparable to the model trained on a large amount of data (30m fragments)  Moreover, we notice that deixis scores are less sensitive to the amount of training data than lexical cohesion and ellipsis scores  The reason might be that, as we observed in our previous work BIBREF11, inconsistencies in translations due to the presence of deictic words and phrases are more frequent in this dataset than other types of inconsistencies  Also, as we show in Section SECREF7, this is the phenomenon the model learns faster in training 
Varying Training Data ::: One-way vs round-trip translations
In this section, we discuss the limitations of using only monolingual data to model inconsistencies between sentence-level translations  In Section SECREF25 we observed a drop in performance on VP ellipsis for DocRepair compared to CADec, which was trained on parallel data  We hypothesized that this is due to the differences between one-way and round-trip translations, and now we test this hypothesis  To do so, we fix the dataset and vary the way in which the input for DocRepair is generated: round-trip or one-way translations  The latter assumes that document-level data is parallel, and translations are sampled from the source side of the sentences in a group rather than from their back-translations  For parallel data, we take 1 5m parallel instances which were used for CADec training and add 1m instances from our monolingual data  For segments in the parallel part, we either sample translations from the source side or use round-trip translations  The results are provided in Table TABREF35 
The model trained on one-way translations is slightly better than the one trained on round-trip translations  As expected, VP ellipsis is the hardest phenomena to be captured using round-trip translations, and the DocRepair model trained on one-way translated data gains 6% accuracy on this test set  This shows that the DocRepair model benefits from having access to non-synthetic English data  This results in exposing DocRepair at training time to Russian translations which suffer from the same inconsistencies as the ones it will have to correct at test time 
Varying Training Data ::: Filtering: monolingual (no filtering) or parallel
Note that the scores of the DocRepair model trained on 2 5m instances randomly chosen from monolingual data (Table TABREF33) are different from the ones for the model trained on 2 5m instances combined from parallel and monolingual data (Table TABREF35)  For convenience, we show these two in Table TABREF36 
The domain, the dataset these two data samples were gathered from, and the way we generated training data for DocRepair (round-trip translations) are all the same  The only difference lies in how the data was filtered  For parallel data, as in the previous work BIBREF6, we picked only sentence pairs with large relative time overlap of subtitle frames between source-language and target-language subtitles  This is necessary to ensure the quality of translation data: one needs groups of consecutive sentences in the target language where every sentence has a reliable translation 
Table TABREF36 shows that the quality of the model trained on data which came from the parallel part is worse than the one trained on monolingual data  This indicates that requiring each sentence in a group to have a reliable translation changes the distribution of the data, which might be not beneficial for translation quality and provides extra motivation for using monolingual data 
Learning Dynamics
Let us now look into how the process of DocRepair training progresses  Figure FIGREF38 shows how the BLEU scores with the reference translation and with the baseline context-agnostic translation (i e  the input for the DocRepair model) are changing during training  First, the model quickly learns to copy baseline translations: the BLEU score with the baseline is very high  Then it gradually learns to change them, which leads to an improvement in BLEU with the reference translation and a drop in BLEU with the baseline  Importantly, the model is reluctant to make changes: the BLEU score between translations of the converged model and the baseline is 82 5  We count the number of changed sentences in every 4-sentence fragment in the test set and plot the histogram in Figure FIGREF38  In over than 20$\%$ of the cases the model has not changed base translations at all  In almost $40\%$, it modified only one sentence and left the remaining 3 sentences unchanged  The model changed more than half sentences in a group in only $14\%$ of the cases  Several examples of the DocRepair translations are shown in Figure FIGREF43 
Figure FIGREF42 shows how consistency scores are changing in training  For deixis, the model achieves the final quality quite quickly; for the rest, it needs a large number of training steps to converge 
Related Work
Our work is most closely related to two lines of research: automatic post-editing (APE) and document-level machine translation 
Related Work ::: Automatic post-editing
Our model can be regarded as an automatic post-editing system – a system designed to fix systematic MT errors that is decoupled from the main MT system  Automatic post-editing has a long history, including rule-based BIBREF17, statistical BIBREF18 and neural approaches BIBREF19, BIBREF20, BIBREF21 
In terms of architectures, modern approaches use neural sequence-to-sequence models, either multi-source architectures that consider both the original source and the baseline translation BIBREF19, BIBREF20, or monolingual repair systems, as in BIBREF21, which is concurrent work to ours  True post-editing datasets are typically small and expensive to create BIBREF22, hence synthetic training data has been created that uses original monolingual data as output for the sequence-to-sequence model, paired with an automatic back-translation BIBREF23 and/or round-trip translation as its input(s) BIBREF19, BIBREF21 
While previous work on automatic post-editing operated on the sentence level, the main novelty of this work is that our DocRepair model operates on groups of sentences and is thus able to fix consistency errors caused by the context-agnostic baseline MT system  We consider this strategy of sentence-level baseline translation and context-aware monolingual repair attractive when parallel document-level data is scarce 
For training, the DocRepair model only requires monolingual document-level data  While we create synthetic training data via round-trip translation similarly to earlier work BIBREF19, BIBREF21, note that we purposefully use sentence-level MT systems for this to create the types of consistency errors that we aim to fix with the context-aware DocRepair model  Not all types of consistency errors that we want to fix emerge from a round-trip translation, so access to parallel document-level data can be useful (Section SECREF34) 
Related Work ::: Document-level NMT
Neural models of MT that go beyond the sentence-level are an active research area BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF10, BIBREF9, BIBREF11  Typically, the main MT system is modified to take additional context as its input  One limitation of these approaches is that they assume that parallel document-level training data is available 
Closest to our work are two-pass models for document-level NMT BIBREF24, BIBREF11, where a second, context-aware model takes the translation and hidden representations of the sentence-level first-pass model as its input  The second-pass model can in principle be trained on a subset of the parallel training data BIBREF11, somewhat relaxing the assumption that all training data is at the document level 
Our work is different from this previous work in two main respects  Firstly, we show that consistency can be improved with only monolingual document-level training data  Secondly, the DocRepair model is decoupled from the first-pass MT system, which improves its portability 
Conclusions
We introduce the first approach to context-aware machine translation using only monolingual document-level data  We propose a monolingual DocRepair model to correct inconsistencies between sentence-level translations  The model performs automatic post-editing on a sequence of sentence-level translations, refining translations of sentences in context of each other  Our approach results in substantial improvements in translation quality as measured by BLEU, targeted contrastive evaluation of several discourse phenomena and human evaluation  Moreover, we perform error analysis and detect which discourse phenomena are hard to capture using only monolingual document-level data  While in the current work we used text fragments of 4 sentences, in future work we would like to consider longer contexts 
Acknowledgments
We would like to thank the anonymous reviewers for their comments  The authors also thank David Talbot and Yandex Machine Translation team for helpful discussions and inspiration  Ivan Titov acknowledges support of the European Research Council (ERC StG BroadSem 678254) and the Dutch National Science Foundation (NWO VIDI 639 022 518)  Rico Sennrich acknowledges support from the Swiss National Science Foundation (105212_169888), the European Union’s Horizon 2020 research and innovation programme (grant agreement no 825460), and the Royal Society (NAF\R1\180122) 

ground_truth:  MT system on the data released by BIBREF11

Transformer base, two-pass CADec model