prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the corpus used for the task?
Context 1: Introduction
Natural languages evolve and words have always been subject to semantic change over time BIBREF1  With the rise of large digitized text resources recent NLP technologies have made it possible to capture such change with vector space models BIBREF2, BIBREF3, BIBREF4, BIBREF5, topic models BIBREF6, BIBREF7, BIBREF8, and sense clustering models BIBREF9  However, many approaches for detecting LSC differ profoundly from each other and therefore drawing comparisons between them can be challenging BIBREF10  Not only do architectures for detecting LSC vary, their performance is also often evaluated without access to evaluation data or too sparse data sets  In cases where evaluation data is available, oftentimes LSCD systems are not evaluated on the same data set which hinders the research community to draw comparisons 
For this reason we report the results of the first shared task on unsupervised lexical semantic change detection in German that is based on an annotated data set to guarantee objective reasoning throughout different approaches  The task was organized as part of the seminar 'Lexical Semantic Change Detection' at the IMS Stuttgart in the summer term of 2019 
Task
The goal of the shared task was to create an architecture to detect semantic change and to rank words according to their degree of change between two different time periods  Given two corpora Ca and Cb, the target words had to be ranked according to their degree of lexical semantic change between Ca and Cb as annotated by human judges  A competition was set up on Codalab and teams mostly consisting of 2 people were formed to take part in the task  There was one group consisting of 3 team members and two individuals who entered the task on their own  In total there were 12 LSCD systems participating in the shared task 
The shared task was divided into three phases, i e , development, testing and analysis phase  In the development phase each team implemented a first version of their model based on a trial data set and submitted it subsequently  In the testing phase the testing data was made public and participants applied their models to the test data with a restriction of possible result uploads to 30  The leaderboard was public at all times  Eventually, the analysis phase was entered and the models of the testing phase were evaluated in terms of the predictions they made and parameters could be tuned further  The models and results will be discussed in detail in sections 7 and 8 
Corpora
The task, as framed above, requires to detect the semantic change between two corpora  The two corpora used in the shared task correspond to the diachronic corpus pair from BIBREF0: DTA18 and DTA19  They consist of subparts of DTA corpus BIBREF11 which is a freely available lemmatized, POS-tagged and spelling-normalized diachronic corpus of German containing texts from the 16th to the 20th century  DTA18 contains 26 million sentences published between 1750-1799 and DTA19 40 million between 1850-1899  The corpus version used in the task has the following format: "year [tab] lemma1 lemma2 lemma3    " 
Evaluation
The Diachronic Usage Relatedness (DURel) gold standard data set includes 22 target words and their varying degrees of semantic change BIBREF12  For each of these target words a random sample of use pairs from the DTA corpus was retrieved and annotated  The annotators were required to rate the pairs according to their semantic relatedness on a scale from 1 to 4 (unrelated - identical meanings) for two time periods  The average Spearman's $\rho $ between the five annotators was 0 66 for 1,320 use paris  The resulting word ranking of the DURel data set is determined by the mean usage relatedness across two time periods and is used as the benchmark to compare the models’ performances in the shared task 
Evaluation ::: Metric
The output of a system with the target words in the predicted order is compared to the gold ranking of the DURel data set  As the metric to assess how well the model's output fits the gold ranking Spearman's $\rho $ was used  The higher Spearman's rank-order correlation the better the system's performance 
Evaluation ::: Baselines
Models were compared to two baselines for the shared task:
log-transformed normalized frequency difference (FD)
count vectors with column intersection and cosine distance (CNT + CI + CD)
The window size for CNT + CI + CD was 10  Find more information on these models in BIBREF0 
Participating Systems
Participants mostly rely on the models compared in BIBREF0 and apply modifications to improve them  In particular, most teams make use of skip-gram with negative sampling (SGNS) based on BIBREF13 to learn the semantic spaces of the two time periods and orthogonal procrustes (OP) to align these vector spaces, similar to the approach by BIBREF14  Different meaning representations such as sense clusters are used as well  As measure to detect the degree of LSC all teams except one choose cosine distance (CD)  This team uses Jensen-Shannon distance (JSD) instead, which computes the distance between probability distributions BIBREF15 
The models of each team will be briefly introduced in this section 
Participating Systems ::: sorensbn
Team sorensbn makes use of SGNS + OP + CD to detect LSC  They use similar hyperparameters as in BIBREF0 to tune the SGNS model  They use an open-sourced noise-aware implementation to improve the OP alignment BIBREF16 
Participating Systems ::: tidoe
Team tidoe builds on SGNS + OP + CD, but they add a transformation step to receive binarized representations of matrices BIBREF17  This step is taken to counter the bias that can occur in vector-space models based on frequencies BIBREF18 
Participating Systems ::: in vain
The team applies a model based on SGNS with vector initialization alignment and cosine distance (SGNS + VI + CD)  Vector initialization is an alignment strategy where the vector space learning model for $t_2$ is initialized with the vectors from $t_1$ BIBREF19  Since SGNS + VI + OP does not perform as well as other models in BIBREF0, they alter the vector initialization process by initializing on the complete model instead of only the word matrix of $t_1$ to receive improved results 
Participating Systems ::: Evilly
In line with previous approaches, team Evilly builds upon SGNS + OP + CD  They alter the OP step by using only high-frequency words for alignment 
Participating Systems ::: DAF
Team DAF uses an architecture based on learning vectors with fastText, alignment with unsupervised and supervised variations of OP, and CD, using the MUSE package BIBREF20, BIBREF21  For the supervised alignment stop words are used  The underlying assumption is that stop words serve as functional units of language and their usage should be consistent over time 
Participating Systems ::: SnakesOnAPlane
The team learns vector spaces with count vectors, positive pointwise mutual information (PPMI), SGNS and uses column intersection (CI) and OP as alignment techniques where applicable  Then they compare two distance measures (CD and JSD) for the different models CNT + CI, PPMI + CI and SGNS + OP to identify which measure performs better for these models  They also experiment with different ways to remove negative values from SGNS vectors, which is needed for JSD 
Participating Systems ::: TeamKulkarni15
TeamKulkarni15 uses SGNS + OP + CD with the modification of local alignment with k nearest neighbors, since other models often use global alignment that can be prone to noise BIBREF22 
Participating Systems ::: Bashmaistori
They use word injection (WI) alignment on PPMI vectors with CD  This approach avoids the complex alignment procedure for embeddings and is applicable to embeddings and count-based methods  They compare two implementations of word injection BIBREF23, BIBREF0 as these showed different results on different data sets 
Participating Systems ::: giki
Team giki uses PPMI + CI + CD to detect LSC  They state that a word sense is determined by its context, but relevant context words can also be found outside a predefined window  Therefore, they use tf-idf to select relevant context BIBREF24 
Participating Systems ::: Edu-Phil
Similar to team DAF they also use fastText + OP + CD  Their hypothesis is that fastText may increase the performance for less frequent words in the corpus since generating word embeddings in fasttext is based on character n-grams 
Participating Systems ::: orangefoxes
They use the model by BIBREF5 which is based on SGNS, but avoids alignment by treating time as a vector that may be combined with word vectors to get time-specific word vectors 
Participating Systems ::: Loud Whisper
Loud Whisper base their approach on BIBREF9 which is a graph-based sense clustering model  They process the data set to receive bigrams, create a co-occurence graph representation and after clustering assess the type of change per word by comparing the results against an intersection table  Their motivation is not only to use a graph-based approach, but to extend the approach by enabling change detection for all parts of speech as opposed to the original model 
Results and Discussion
Table TABREF8 shows the results of the shared task  All teams receive better results than baseline 1 (FD), of which a total of 8 teams outperform baseline 2 (CNT + CI + CD)  The 4 top scores with $\rho $ $>$ 0 7 are either modified versions of SGNS + OP + CD or use SGNS + VI + CD  The following 4 scores in the range of 0 5 $<$ $\rho $ $<$ 0 6 are generated by the models fastText + OP + CD, SGNS + OP + CD/JSD, and PPMI + WI + CD 
Contrary to the results by BIBREF0 the modified version of vector initialization shows high performance similar to OP alignment, as previously reported by BIBREF14  Some modifications to the SGNS + OP + CD approach are able to yield better results than others, e g  noise-aware alignment and binarized matrices as compared to frequency-driven OP alignment or local alignment with KNN  Team SnakesOnAPlane compare two distance measures and their results show that JSD ($\rho $ $=$  561) performs minimally worse than CD ($\rho $ $=$  565) as the semantic change measure for their model 
The overall best-performing model is Skip-Gram with orthogonal alignment and cosine distance (SGNS + OP + CD) with similar hyperparameters as in the model architecture described previously BIBREF0  Said architecture was used as the basis for the two best performing models  Team tidoe reports that binarizing matrices leads to a generally worse performance ($\rho $ $=$  811) compared to the unmodified version of SGNS + OP + CD ($\rho $ $=$ 0 9)  The noise aware alignment approach applied by team sorensbn obtains a higher score ($\rho $ $=$  854) compared to the result reported by tidoe, but is unable to exceed the performance of the unmodified SNGS + OP + CD for the same set of hyperparameters (window size = 10, negative sampling = 1; subsampling = None)  Of the 8 scores above the second baseline, 5 use an architecture that builds upon SGNS + OP + CD  Whereas in the lower score segment $\rho $ $<$ 0 5 none of the models use SGNS + OP + CD  These findings are in line with the results reported by BIBREF0, however the overall best results are lower in this shared task, which is expected from the smaller number of parameter combinations explored  Additionally, in the shared task the objective was to report the best score and not to calculate the mean which makes it more difficult to compare the robustness of the models presented here 
Context 2: Introduction
Recently, deep neural network has been widely employed in various recognition tasks  Increasing the depth of neural network is a effective way to improve the performance, and convolutional neural network (CNN) has benefited from it in visual recognition task BIBREF0   Deeper long short-term memory (LSTM) recurrent neural networks (RNNs) are also applied in large vocabulary continuous speech recognition (LVCSR) task, because LSTM networks have shown better performance than Fully-connected feed-forward deep neural network BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4  
Training neural network becomes more challenge when it goes deep  A conceptual tool called linear classifier probe is introduced to better understand the dynamics inside a neural network BIBREF5   The discriminating features of linear classifier is the hidden units of a intermediate layer  For deep neural networks, it is observed that deeper layer's accuracy is lower than that of shallower layers  Therefore, the tool shows the difficulty of deep neural model training visually 
Layer-wise pre-training is a successful method to train very deep neural networks BIBREF6   The convergence becomes harder with increasing the number of layers, even though the model is initialized with Xavier or its variants BIBREF7 , BIBREF8   But the deeper network which is initialized with a shallower trained network could converge well 
The size of LVCSR training dataset goes larger and training with only one GPU becomes high time consumption inevitably  Therefore, parallel training with multi-GPUs is more suitable for LVCSR system  Mini-batch based stochastic gradient descent (SGD) is the most popular method in neural network training procedure  Asynchronous SGD is a successful effort for parallel training based on it BIBREF9 , BIBREF10   It can many times speed up the training time without decreasing the accuracy  Besides, synchronous SGD is another effective effort, where the parameter server waits for every works to finish their computation and sent their local models to it, and then it sends updated model back to all workers BIBREF11   Synchronous SGD converges well in parallel training with data parallelism, and is also easy to implement 
In order to further improve the performance of deep neural network with parallel training, several methods are proposed  Model averaging method achieves linear speedup, as the final model is averaged from all parameters of local models in different workers BIBREF12 , BIBREF13 , but the accuracy decreases compared with single GPU training  Moreover, blockwise model-updating filter (BMUF) provides another almost linear speedup approach with multi-GPUs on the basis of model averaging  It can achieve improvement or no-degradation of recognition performance compared with mini-batch SGD on single GPU BIBREF14  
Moving averaged (MA) approaches are also proposed for parallel training  It is demonstrated that the moving average of the parameters obtained by SGD performs as well as the parameters that minimize the empirical cost, and moving average parameters can be used as the estimator of them, if the size of training data is large enough BIBREF15   One pass learning is then proposed, which is the combination of learning rate schedule and averaged SGD using moving average BIBREF16   Exponential moving average (EMA) is proposed as a non-interference method BIBREF17   EMA model is not broadcasted to workers to update their local models, and it is applied as the final model of entire training process  EMA method is utilized with model averaging and BMUF to further decrease the character error rate (CER)  It is also easy to implement in existing parallel training systems 
Frame stacking can also speed up the training time BIBREF18   The super frame is stacked by several regular frames, and it contains the information of them  Thus, the network can see multiple frames at a time, as the super frame is new input  Frame stacking can also lead to faster decoding 
For streaming voice search service, it needs to display intermediate recognition results while users are still speaking  As a result, the system needs to fulfill high real-time requirement, and we prefer unidirectional LSTM network rather than bidirectional one  High real-time requirement means low real time factor (RTF), but the RTF of deep LSTM model is higher inevitably  The dilemma of recognition accuracy and real-time requirement is an obstacle to the employment of deep LSTM network  Deep model outperforms because it contains more knowledge, but it is also cumbersome  As a result, the knowledge of deep model can be distilled to a shallow model BIBREF19   It provided a effective way to employ the deep model to the real-time system 
In this paper, we explore a entire deep LSTM RNN training framework, and employ it to real-time application  The deep learning systems benefit highly from a large quantity of labeled training data  Our first and basic speech recognition system is trained on 17000 hours of Shenma voice search dataset  It is a generic dataset sampled from diverse aspects of search queries  The requirement of speech recognition system also addressed by specific scenario, such as map and navigation task  The labeled dataset is too expensive, and training a new model with new large dataset from the beginning costs lots of time  Thus, it is natural to think of transferring the knowledge from basic model to new scenario's model  Transfer learning expends less data and less training time than full training  In this paper, we also introduce a novel transfer learning strategy with segmental Minimum Bayes-Risk (sMBR)  As a result, transfer training with only 1000 hours data can match equivalent performance for full training with 7300 hours data 
Our deep LSTM training framework for LVCSR is presented in Section 2  Section 3 describes how the very deep models does apply in real world applications, and how to transfer the model to another task  The framework is analyzed and discussed in Section 4, and followed by the conclusion in Section 5 
Layer-wise Training with Soft Target and Hard Target
Gradient-based optimization of deep LSTM network with random initialization get stuck in poor solution easily  Xavier initialization can partially solve this problem BIBREF7 , so this method is the regular initialization method of all training procedure  However, it does not work well when it is utilized to initialize very deep model directly, because of vanishing or exploding gradients  Instead, layer-wise pre-training method is a effective way to train the weights of very deep architecture BIBREF6 , BIBREF20   In layer-wise pre-training procedure, a one-layer LSTM model is firstly trained with normalized initialization  Sequentially, two-layers LSTM model's first layer is initialized by trained one-layer model, and its second layer is regularly initialized  In this way, a deep architecture is layer-by-layer trained, and it can converge well 
In conventional layer-wise pre-training, only parameters of shallower network are transfered to deeper one, and the learning targets are still the alignments generated by HMM-GMM system  The targets are vectors that only one state's probability is one, and the others' are zeros  They are known as hard targets, and they carry limited knowledge as only one state is active  In contrast, the knowledge of shallower network should be also transfered to deeper one  It is obtained by the softmax layer of existing model typically, so each state has a probability rather than only zero or one, and called as soft target  As a result, the deeper network which is student network learns the parameters and knowledge from shallower one which is called teacher network  When training the student network from the teacher network, the final alignment is the combination of hard target and soft target in our layer-wise training phase  The final alignment provides various knowledge which transfered from teacher network and extracted from true labels  If only soft target is learned, student network perform no better than teacher network, but it could outperform teacher network as it also learns true labels 
The deeper network spends less time to getting the same level of original network than the network trained from the beginning, as a period of low performance is skipped  Therefore, training with hard and soft target is a time saving method  For large training dataset, training with the whole dataset still spends too much time  A network firstly trained with only a small part of dataset could go deeper as well, and so the training time reducing rapidly  When the network is deep enough, it then trained on the entire dataset to get further improvement  There is no gap of accuracy between these two approaches, but latter one saves much time 
Differential Saturation Check
The objects of conventional saturation check are gradients and the cell activations BIBREF4   Gradients are clipped to range [-5, 5], while the cell activations clipped to range [-50, 50]  Apart from them, the differentials of recurrent layers is also limited  If the differentials go beyond the range, corresponding back propagation is skipped, while if the gradients and cell activations go beyond the bound, values are set as the boundary values  The differentials which are too large or too small will lead to the gradients easily vanishing, and it demonstrates the failure of this propagation  As a result, the parameters are not updated, and next propagation  
Sequence Discriminative Training
Cross-entropy (CE) is widely used in speech recognition training system as a frame-wise discriminative training criterion  However, it is not well suited to speech recognition, because speech recognition training is a sequential learning problem  In contrast, sequence discriminative training criterion has shown to further improve performance of neural network first trained with cross-entropy BIBREF21 , BIBREF22 , BIBREF23   We choose state-level minimum bayes risk (sMBR) BIBREF21 among a number of sequence discriminative criterion is proposed, such as maximum mutual information (MMI) BIBREF24 and minimum phone error (MPE) BIBREF25   MPE and sMBR are designed to minimize the expected error of different granularity of labels, while CE aims to minimizes expected frame error, and MMI aims to minimizes expected sentence error  State-level information is focused on by sMBR 
a frame-level accurate model is firstly trained by CE loss function, and then sMBR loss function is utilized for further training to get sequence-level accuracy  Only a part of training dataset is needed in sMBR training phase on the basis of whole dataset CE training 
Parallel Training
It is demonstrated that training with larger dataset can improve recognition accuracy  However, larger dataset means more training samples and more model parameters  Therefore, parallel training with multiple GPUs is essential, and it makes use of data parallelism BIBREF9   The entire training data is partitioned into several split without overlapping and they are distributed to different GPUs  Each GPU trains with one split of training dataset locally  All GPUs synchronize their local models with model average method after a mini-batch optimization BIBREF12 , BIBREF13  
Model average method achieves linear speedup in training phase, but the recognition accuracy decreases compared with single GPU training  Block-wise model updating filter (BMUF) is another successful effort in parallel training with linear speedup as well  It can achieve no-degradation of recognition accuracy with multi-GPUs BIBREF14   In the model average method, aggregated model INLINEFORM0 is computed and broadcasted to GPUs  On the basis of it, BMUF proposed a novel model updating strategy: INLINEFORM1 INLINEFORM2
Where INLINEFORM0 denotes model update, and INLINEFORM1 is the global-model update  There are two parameters in BMUF, block momentum INLINEFORM2 , and block learning rate INLINEFORM3   Then, the global model is updated as INLINEFORM4
Consequently, INLINEFORM0 is broadcasted to all GPUs to initial their local models, instead of INLINEFORM1 in model average method 
Averaged SGD is proposed to further accelerate the convergence speed of SGD  Averaged SGD leverages the moving average (MA) INLINEFORM0 as the estimator of INLINEFORM1 BIBREF15 : INLINEFORM2
Where INLINEFORM0 is computed by model averaging or BMUF  It is shown that INLINEFORM1 can well converge to INLINEFORM2 , with the large enough training dataset in single GPU training  It can be considered as a non-interference strategy that INLINEFORM3 does not participate the main optimization process, and only takes effect after the end of entire optimization  However, for the parallel training implementation, each INLINEFORM4 is computed by model averaging and BMUF with multiple models, and moving average model INLINEFORM5 does not well converge, compared with single GPU training 
Model averaging based methods are employed in parallel training of large scale dataset, because of their faster convergence, and especially no-degradation implementation of BMUF  But combination of model averaged based methods and moving average does not match the expectation of further enhance performance and it is presented as INLINEFORM0
The weight of each INLINEFORM0 is equal in moving average method regardless the effect of temporal order  But INLINEFORM1 closer to the end of training achieve higher accuracy in the model averaging based approach, and thus it should be with more proportion in final INLINEFORM2   As a result, exponential moving average(EMA) is appropriate, which the weight for each older parameters decrease exponentially, and never reaching zero  After moving average based methods, the EMA parameters are updated recursively as INLINEFORM3
Here INLINEFORM0 represents the degree of weight decrease, and called exponential updating rate  EMA is also a non-interference training strategy that is implemented easily, as the updated model is not broadcasted  Therefore, there is no need to add extra learning rate updating approach, as it can be appended to existing training procedure directly
Context 3: Introduction
We posses a wealth of prior knowledge about many natural language processing tasks  For example, in text categorization, we know that words such as NBA, player, and basketball are strong indicators of the sports category BIBREF0 , and words like terrible, boring, and messing indicate a negative polarity while words like perfect, exciting, and moving suggest a positive polarity in sentiment classification 
A key problem arisen here, is how to leverage such knowledge to guide the learning process, an interesting problem for both NLP and machine learning communities  Previous studies addressing the problem fall into several lines  First, to leverage prior knowledge to label data BIBREF1 , BIBREF2   Second, to encode prior knowledge with a prior on parameters, which can be commonly seen in many Bayesian approaches BIBREF3 , BIBREF4   Third, to formalise prior knowledge with additional variables and dependencies BIBREF5   Last, to use prior knowledge to control the distributions over latent output variables BIBREF6 , BIBREF7 , BIBREF8 , which makes the output variables easily interpretable 
However, a crucial problem, which has rarely been addressed, is the bias in the prior knowledge that we supply to the learning model  Would the model be robust or sensitive to the prior knowledge? Or, which kind of knowledge is appropriate for the task? Let's see an example: we may be a baseball fan but unfamiliar with hockey so that we can provide a few number of feature words of baseball, but much less of hockey for a baseball-hockey classification task  Such prior knowledge may mislead the model with heavy bias to baseball  If the model cannot handle this situation appropriately, the performance may be undesirable 
In this paper, we investigate into the problem in the framework of Generalized Expectation Criteria BIBREF7   The study aims to reveal the factors of reducing the sensibility of the prior knowledge and therefore to make the model more robust and practical  To this end, we introduce auxiliary regularization terms in which our prior knowledge is formalized as distribution over output variables  Recall the example just mentioned, though we do not have enough knowledge to provide features for class hockey, it is easy for us to provide some neutral words, namely words that are not strong indicators of any class, like player here  As one of the factors revealed in this paper, supplying neutral feature words can boost the performance remarkably, making the model more robust 
More attractively, we do not need manual annotation to label these neutral feature words in our proposed approach 
More specifically, we explore three regularization terms to address the problem: (1) a regularization term associated with neutral features; (2) the maximum entropy of class distribution regularization term; and (3) the KL divergence between reference and predicted class distribution  For the first manner, we simply use the most common features as neutral features and assume the neutral features are distributed uniformly over class labels  For the second and third one, we assume we have some knowledge about the class distribution which will be detailed soon later 
To summarize, the main contributions of this work are as follows:
The rest of the paper is structured as follows: In Section 2, we briefly describe the generalized expectation criteria and present the proposed regularization terms  In Section 3, we conduct extensive experiments to justify the proposed methods  We survey related work in Section 4, and summarize our work in Section 5 
Method
We address the robustness problem on top of GE-FL BIBREF0 , a GE method which leverages labeled features as prior knowledge  A labeled feature is a strong indicator of a specific class and is manually provided to the classifier  For example, words like amazing, exciting can be labeled features for class positive in sentiment classification 
Generalized Expectation Criteria
Generalized expectation (GE) criteria BIBREF7 provides us a natural way to directly constrain the model in the preferred direction  For example, when we know the proportion of each class of the dataset in a classification task, we can guide the model to predict out a pre-specified class distribution 
Formally, in a parameter estimation objective function, a GE term expresses preferences on the value of some constraint functions about the model's expectation  Given a constraint function $G({\rm x}, y)$ , a conditional model distribution $p_\theta (y|\rm x)$ , an empirical distribution $\tilde{p}({\rm x})$ over input samples and a score function $S$ , a GE term can be expressed as follows:
$$S(E_{\tilde{p}({\rm x})}[E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]])$$   (Eq  4)
Learning from Labeled Features
Druck et al  ge-fl proposed GE-FL to learn from labeled features using generalized expectation criteria  When given a set of labeled features $K$ , the reference distribution over classes of these features is denoted by $\hat{p}(y| x_k), k \in K$   GE-FL introduces the divergence between this reference distribution and the model predicted distribution $p_\theta (y | x_k)$ , as a term of the objective function:
$$\mathcal {O} = \sum _{k \in K} KL(\hat{p}(y|x_k) || p_\theta (y | x_k)) + \sum _{y,i} \frac{\theta _{yi}^2}{2 \sigma ^2}$$   (Eq  6)
where $\theta _{yi}$ is the model parameter which indicates the importance of word $i$ to class $y$   The predicted distribution $p_\theta (y | x_k)$ can be expressed as follows: $ p_\theta (y | x_k) = \frac{1}{C_k} \sum _{\rm x} p_\theta (y|{\rm x})I(x_k) $
in which $I(x_k)$ is 1 if feature $k$ occurs in instance ${\rm x}$ and 0 otherwise, $C_k = \sum _{\rm x} I(x_k)$ is the number of instances with a non-zero value of feature $k$ , and $p_\theta (y|{\rm x})$ takes a softmax form as follows: $ p_\theta (y|{\rm x}) = \frac{1}{Z(\rm x)}\exp (\sum _i \theta _{yi}x_i)  $
To solve the optimization problem, L-BFGS can be used for parameter estimation 
In the framework of GE, this term can be obtained by setting the constraint function $G({\rm x}, y) = \frac{1}{C_k} \vec{I} (y)I(x_k)$ , where $\vec{I}(y)$ is an indicator vector with 1 at the index corresponding to label $y$ and 0 elsewhere 
Regularization Terms
GE-FL reduces the heavy load of instance annotation and performs well when we provide prior knowledge with no bias  In our experiments, we observe that comparable numbers of labeled features for each class have to be supplied  But as mentioned before, it is often the case that we are not able to provide enough knowledge for some of the classes  For the baseball-hockey classification task, as shown before, GE-FL will predict most of the instances as baseball  In this section, we will show three terms to make the model more robust 
Neutral features are features that are not informative indicator of any classes, for instance, word player to the baseball-hockey classification task  Such features are usually frequent words across all categories  When we set the preference distribution of the neutral features to be uniform distributed, these neutral features will prevent the model from biasing to the class that has a dominate number of labeled features 
Formally, given a set of neutral features $K^{^{\prime }}$ , the uniform distribution is $\hat{p}_u(y|x_k) = \frac{1}{|C|}, k \in K^{^{\prime }}$ , where $|C|$ is the number of classes  The objective function with the new term becomes
$$\mathcal {O}_{NE} = \mathcal {O} + \sum _{k \in K^{^{\prime }}} KL(\hat{p}_u(y|x_k) || p_\theta (y | x_k)) $$   (Eq  9)
Note that we do not need manual annotation to provide neutral features  One simple way is to take the most common features as neutral features  Experimental results show that this strategy works successfully 
Another way to prevent the model from drifting from the desired direction is to constrain the predicted class distribution on unlabeled data  When lacking knowledge about the class distribution of the data, one feasible way is to take maximum entropy principle, as below:
$$\mathcal {O}_{ME} = \mathcal {O} + \lambda \sum _{y} p(y) \log p(y)$$   (Eq  11)
where $p(y)$ is the predicted class distribution, given by $ p(y) = \frac{1}{|X|} \sum _{\rm x} p_\theta (y | \rm x)  $ To control the influence of this term on the overall objective function, we can tune $\lambda $ according to the difference in the number of labeled features of each class  In this paper, we simply set $\lambda $ to be proportional to the total number of labeled features, say $\lambda = \beta |K|$  
This maximum entropy term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$   Therefore, $E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]$ is just the model distribution $p_\theta (y|{\rm x})$ and its expectation with the empirical distribution $\tilde{p}(\rm x)$ is simply the average over input samples, namely $p(y)$   When $S$ takes the maximum entropy form, we can derive the objective function as above 
Sometimes, we have already had much knowledge about the corpus, and can estimate the class distribution roughly without labeling instances  Therefore, we introduce the KL divergence between the predicted and reference class distributions into the objective function 
Given the preference class distribution $\hat{p}(y)$ , we modify the objective function as follows:
$$\mathcal {O}_{KL} &= \mathcal {O} + \lambda KL(\hat{p}(y) || p(y))$$   (Eq  13)
Similarly, we set $\lambda = \beta |K|$  
This divergence term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$ and setting the score function to $S(\hat{p}, p) = \sum _i \hat{p}_i \log \frac{\hat{p}_i}{p_i}$ , where $p$ and $\hat{p}$ are distributions  Note that this regularization term involves the reference class distribution which will be discussed later 
Experiments
In this section, we first justify the approach when there exists unbalance in the number of labeled features or in class distribution  Then, to test the influence of $\lambda $ , we conduct some experiments with the method which incorporates the KL divergence of class distribution  Last, we evaluate our approaches in 9 commonly used text classification datasets  We set $\lambda = 5|K|$ by default in all experiments unless there is explicit declaration  The baseline we choose here is GE-FL BIBREF0 , a method based on generalization expectation criteria 
Data Preparation
We evaluate our methods on several commonly used datasets whose themes range from sentiment, web-page, science to medical and healthcare  We use bag-of-words feature and remove stopwords in the preprocess stage  Though we have labels of all documents, we do not use them during the learning process, instead, we use the label of features 
The movie dataset, in which the task is to classify the movie reviews as positive or negtive, is used for testing the proposed approaches with unbalanced labeled features, unbalanced datasets or different $\lambda $ parameters  All unbalanced datasets are constructed based on the movie dataset by randomly removing documents of the positive class  For each experiment, we conduct 10-fold cross validation 
As described in BIBREF0 , there are two ways to obtain labeled features  The first way is to use information gain  We first calculate the mutual information of all features according to the labels of the documents and select the top 20 as labeled features for each class as a feature pool  Note that using information gain requires the document label, but this is only to simulate how we human provide prior knowledge to the model  The second way is to use LDA BIBREF9 to select features  We use the same selection process as BIBREF0 , where they first train a LDA on the dataset, and then select the most probable features of each topic (sorted by $P(w_i|t_j)$ , the probability of word $w_i$ given topic $t_j$ ) 
Similar to BIBREF10 , BIBREF0 , we estimate the reference distribution of the labeled features using a heuristic strategy  If there are $|C|$ classes in total, and $n$ classes are associated with a feature $k$ , the probability that feature $k$ is related with any one of the $n$ classes is $\frac{0 9}{n}$ and with any other class is $\frac{0 1}{|C| - n}$  
Neutral features are the most frequent words after removing stop words, and their reference distributions are uniformly distributed  We use the top 10 frequent words as neutral features in all experiments 
With Unbalanced Labeled Features
In this section, we evaluate our approach when there is unbalanced knowledge on the categories to be classified  The labeled features are obtained through information gain  Two settings are chosen:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other  The original balanced movie dataset is used (positive:negative=1:1) 
(b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4) 
As shown in Figure 1 , Maximum entropy principle shows improvement only on the balanced case

ground_truth: DTA18, DTA19

Diachronic Usage Relatedness (DURel) gold standard data set

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. which lstm models did they compare with?
Context 2: 
Deployment
There is a high real time requirement in real world application, especially in online voice search system  Shenma voice search is one of the most popular mobile search engines in China, and it is a streaming service that intermediate recognition results displayed while users are still speaking  Unidirectional LSTM network is applied, rather than bidirectional one, because it is well suited to real-time streaming speech recognition 
Distillation
It is demonstrated that deep neural network architecture can achieve improvement in LVCSR  However, it also leads to much more computation and higher RTF, so that the recognition result can not be displayed in real time  It should be noted that deeper neural network contains more knowledge, but it is also cumbersome  the knowledge is key to improve the performance  If it can be transfered from cumbersome model to a small model, the recognition ability can also be transfered to the small model  Knowledge transferring to small model is called distillation BIBREF19   The small model can perform as well as cumbersome model, after distilling  It provide a way to utilize high-performance but high RTF model in real time system  The class probability produced by the cumbersome model is regarded as soft target, and the generalization ability of cumbersome model is transfered to small model with it  Distillation is model's knowledge transferring approach, so there is no need to use the hard target, which is different with the layer-wise training method 
9-layers unidirectional LSTM model achieves outstanding performance, but meanwhile it is too computationally expensive to allow deployment in real time recognition system  In order to ensure real-time of the system, the number of layers needs to be reduced  The shallower network can learn the knowledge of deeper network with distillation  It is found that RTF of 2-layers network is acceptable, so the knowledge is distilled from 9-layers well-trained model to 2-layers model  Table TABREF16 shows that distillation from 9-layers to 2-layers brings RTF decrease of relative 53%, while CER only increases 5%  The knowledge of deep network is almost transfered with distillation, Distillation brings promising RTF reduction, but only little knowledge of deep network is lost  Moreover, CER of 2-layers distilled LSTM decreases relative 14%, compared with 2-layers regular-trained LSTM 
Transfer Learning with sMBR
For a certain specific scenario, the model trained with the data recorded from it has better adaptation than the model trained with generic scenario  But it spends too much time training a model from the beginning, if there is a well-trained model for generic scenarios  Moreover, labeling a large quantity of training data in new scenario is both costly and time consuming  If a model transfer trained with smaller dataset can obtained the similar recognition accuracy compared with the model directly trained with larger dataset, it is no doubt that transfer learning is more practical  Since specific scenario is a subset of generic scenario, some knowledge can be shared between them  Besides, generic scenario consists of various conditions, so its model has greater robustness  As a result, not only shared knowledge but also robustness can be transfered from the model of generic scenario to the model of specific one 
As the model well trained from generic scenario achieves good performance in frame level classification, sequence discriminative training is required to adapt new model to specific scenario additionally  Moreover, it does not need alignment from HMM-GMM system, and it also saves amount of time to prepare alignment 
Training Data
A large quantity of labeled data is needed for training a more accurate acoustic model  We collect the 17000 hours labeled data from Shenma voice search, which is one of the most popular mobile search engines in China  The dataset is created from anonymous online users' search queries in Mandarin, and all audio file's sampling rate is 16kHz, recorded by mobile phones  This dataset consists of many different conditions, such as diverse noise even low signal-to-noise, babble, dialects, accents, hesitation and so on 
In the Amap, which is one of the most popular web mapping and navigation services in China, users can search locations and navigate to locations they want though voice search  To present the performance of transfer learning with sequence discriminative training, the model trained from Shenma voice search which is greneric scenario transfer its knowledge to the model of Amap voice search  7300 hours labeled data is collected in the similar way of Shenma voice search data collection 
Two dataset is divided into training set, validation set and test set separately, and the quantity of them is shown in Table TABREF10   The three sets are split according to speakers, in order to avoid utterances of same speaker appearing in three sets simultaneously  The test sets of Shenma and Amap voice search are called Shenma Test and Amap Test 
Experimental setup
LSTM RNNs outperform conventional RNNs for speech recognition system, especially deep LSTM RNNs, because of its long-range dependencies more accurately for temporal sequence conditions BIBREF26 , BIBREF23   Shenma and Amap voice search is a streaming service that intermediate recognition results displayed while users are still speaking  So as for online recognition in real time, we prefer unidirectional LSTM model rather than bidirectional one  Thus, the training system is unidirectional LSTM-based 
A 26-dimensional filter bank and 2-dimensional pitch feature is extracted for each frame, and is concatenated with first and second order difference as the final input of the network  The super frame are stacked by 3 frames without overlapping  The architecture we trained consists of two LSTM layers with sigmoid activation function, followed by a full-connection layer  The out layer is a softmax layer with 11088 hidden markov model (HMM) tied-states as output classes, the loss function is cross-entropy (CE)  The performance metric of the system in Mandarin is reported with character error rate (CER)  The alignment of frame-level ground truth is obtained by GMM-HMM system  Mini-batched SGD is utilized with momentum trick and the network is trained for a total of 4 epochs  The block learning rate and block momentum of BMUF are set as 1 and 0 9  5-gram language model is leveraged in decoder, and the vocabulary size is as large as 760000  Differentials of recurrent layers is limited to range [-10000,10000], while gradients are clipped to range [-5, 5] and cell activations clipped to range [-50, 50]  After training with CE loss, sMBR loss is employed to further improve the performance 
It has shown that BMUF outperforms traditional model averaging method, and it is utilized at the synchronization phase  After synchronizing with BMUF, EMA method further updates the model in non-interference way  The training system is deployed on the MPI-based HPC cluster where 8 GPUs  Each GPU processes non-overlap subset split from the entire large scale dataset in parallel 
Local models from distributed workers synchronize with each other in decentralized way  In the traditional model averaging and BMUF method, a parameter server waits for all workers to send their local models, aggregate them, and send the updated model to all workers  Computing resource of workers is wasted until aggregation of the parameter server done  Decentralized method makes full use of computing resource, and we employ the MPI-based Mesh AllReduce method  It is mesh topology as shown in Figure FIGREF12   There is no centralized parameter server, and peer to peer communication is used to transmit local models between workers  Local model INLINEFORM0 of INLINEFORM1 -th worker in INLINEFORM2 workers cluster is split to INLINEFORM3 pieces INLINEFORM4 , and send to corresponding worker  In the aggregation phase, INLINEFORM5 -th worker computed INLINEFORM6 splits of model INLINEFORM7 and send updated model INLINEFORM8 back to workers  As a result, all workers participate in aggregation and no computing resource is dissipated  It is significant to promote training efficiency, when the size of neural network model is too large  The EMA model is also updated additionally, but not broadcasting it 
Results
In order to evaluate our system, several sets of experiments are performed  The Shenma test set including about 9000 samples and Amap test set including about 7000 samples contain various real world conditions  It simulates the majority of user scenarios, and can well evaluates the performance of a trained model  Firstly, we show the results of models trained with EMA method  Secondly, for real world applications, very deep LSTM is distilled to a shallow one, so as for lower RTF  The model of Amap is also needed to train for map and navigation scenarios  The performance of transfer learning from Shenma voice search to Amap voice search is also presented 
Layer-wise Training
In layer-wise training, the deeper model learns both parameters and knowledge from the shallower model  The deeper model is initialized by the shallower one, and its alignment is the combination of hard target and soft target of shallower one  Two targets have the same weights in our framework  The teacher model is trained with CE  For each layer-wise trained CE model, corresponding sMBR model is also trained, as sMBR could achieve additional improvement  In our framework, 1000 hours data is randomly selected from the total dataset for sMBR training  There is no obvious performance enhancement when the size of sMBR training dataset increases 
For very deep unidirectional LSTM initialized with Xavier initialization algorithm, 6-layers model converges well, but there is no further improvement with increasing the number of layers  Therefore, the first 6 layers of 7-layers model is initialized by 6-layers model, and soft target is provided by 6-layers model  Consequently, deeper LSTM is also trained in the same way  It should be noticed that the teacher model of 9-layers model is the 8-layers model trained by sMBR, while the other teacher model is CE model  As shown in Table TABREF15 , the layer-wise trained models always performs better than the models with Xavier initialization, as the model is deep  Therefore, for the last layer training, we choose 8-layers sMBR model as the teacher model instead of CE model  A comparison between 6-layers and 9-layers sMBR models shows that 3 additional layers of layer-wise training brings relative 12 6% decreasing of CER  It is also significant that the averaged CER of sMBR models with different layers decreases absolute 0 73% approximately compared with CE models, so the improvement of sequence discriminative learning is promising 
Transfer Learning
2-layers distilled model of Shenma voice search has shown a impressive performance on Shenma Test, and we call it Shenma model  It is trained for generic search scenario, but it has less adaptation for specific scenario like Amap voice search  Training with very large dataset using CE loss is regarded as improvement of frame level recognition accuracy, and sMBR with less dataset further improves accuracy as sequence discriminative training  If robust model of generic scenario is trained, there is no need to train a model with very large dataset, and sequence discriminative training with less dataset is enough  Therefore, on the basis of Shenma model, it is sufficient to train a new Amap model with small dataset using sMBR  As shown in Table TABREF18 , Shenma model presents the worst performance among three methods, since it does not trained for Amap scenario  2-layers Shenma model further trained with sMBR achieves about 8 1% relative reduction, compared with 2-layers regular-trained Amap model  Both training sMBR datasets contain the same 1000 hours data  As a result, with the Shenma model, only about 14% data usage achieves lower CER, and it leads to great time and cost saving with less labeled data  Besides, transfer learning with sMBR does not use the alignment from the HMM-GMM system, so it also saves huge amount of time 
Conclusion
We have presented a whole deep unidirectional LSTM parallel training system for LVCSR  The recognition performance improves when the network goes deep  Distillation makes it possible that deep LSTM model transfer its knowledge to shallow model with little loss  The model could be distilled to 2-layers model with very low RTF, so that it can display the immediate recognition results  As a result, its CER decrease relatively 14%, compared with the 2-layers regular trained model  In addition, transfer learning with sMBR is also proposed  If a great model has well trained from generic scenario, only 14% of the size of training dataset is needed to train a more accuracy acoustic model for specific scenario  Our future work includes 1) finding more effective methods to reduce the CER by increasing the number of layers; 2) applying this training framework to Connectionist Temporal Classification (CTC) and attention-based neural networks 
Context 3:  An obvious reason is that maximum entropy only favors uniform distribution 
Incorporating Neutral features performs similarly to maximum entropy since we assume that neutral words are uniformly distributed  Its accuracy decreases slowly when the number of labeled features becomes larger ( $t>4$ ) (Figure 1 (a)), suggesting that the model gradually biases to the class with more labeled features, just like GE-FL 
Incorporating the KL divergence of class distribution performs much better than GE-FL on both balanced and unbalanced datasets  This shows that it is effective to control the unbalance in labeled features and in the dataset 
With Balanced Labeled Features
We also compare with the baseline when the labeled features are balanced  Similar to the experiment above, the labeled features are obtained by information gain  Two settings are experimented with:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for each class, and conduct comparisons on the original balanced movie dataset (positive:negtive=1:1) 
(b) Similar to (a), but the class distribution is unbalanced, by randomly removing 75% positive documents (positive:negative=1:4) 
Results are shown in Figure 2   When the dataset is balanced (Figure 2 (a)), there is little difference between GE-FL and our methods  The reason is that the proposed regularization terms provide no additional knowledge to the model and there is no bias in the labeled features  On the unbalanced dataset (Figure 2 (b)), incorporating KL divergence is much better than GE-FL since we provide additional knowledge(the true class distribution), but maximum entropy and neutral features are much worse because forcing the model to approach the uniform distribution misleads it 
With Unbalanced Class Distributions
Our methods are also evaluated on datasets with different unbalanced class distributions  We manually construct several movie datasets with class distributions of 1:2, 1:3, 1:4 by randomly removing 50%, 67%, 75% positive documents  The original balanced movie dataset is used as a control group  We test with both balanced and unbalanced labeled features  For the balanced case, we randomly select 10 features from the feature pool for each class, and for the unbalanced case, we select 10 features for one class, and 1 feature for the other  Results are shown in Figure 3  
Figure 3 (a) shows that when the dataset and the labeled features are both balanced, there is little difference between our methods and GE-FL(also see Figure 2 (a))  But when the class distribution becomes more unbalanced, the difference becomes more remarkable  Performance of neutral features and maximum entropy decrease significantly but incorporating KL divergence increases remarkably  This suggests if we have more accurate knowledge about class distribution, KL divergence can guide the model to the right direction 
Figure 3 (b) shows that when the labeled features are unbalanced, our methods significantly outperforms GE-FL  Incorporating KL divergence is robust enough to control unbalance both in the dataset and in labeled features while the other three methods are not so competitive 
The Influence of λ\lambda 
We present the influence of $\lambda $ on the method that incorporates KL divergence in this section  Since we simply set $\lambda = \beta |K|$ , we just tune $\beta $ here  Note that when $\beta = 0$ , the newly introduced regularization term is disappeared, and thus the model is actually GE-FL  Again, we test the method with different $\lambda $ in two settings:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other class  The original balanced movie dataset is used (positive:negative=1:1) 
(b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4) 
Results are shown in Figure 4   As expected, $\lambda $ reflects how strong the regularization is  The model tends to be closer to our preferences with the increasing of $\lambda $ on both cases 
Using LDA Selected Features
We compare our methods with GE-FL on all the 9 datasets in this section  Instead of using features obtained by information gain, we use LDA to select labeled features  Unlike information gain, LDA does not employ any instance labels to find labeled features  In this setting, we can build classification models without any instance annotation, but just with labeled features 
Table 1 shows that our three methods significantly outperform GE-FL  Incorporating neutral features performs better than GE-FL on 7 of the 9 datasets, maximum entropy is better on 8 datasets, and KL divergence better on 7 datasets 
LDA selects out the most predictive features as labeled features without considering the balance among classes  GE-FL does not exert any control on such an issue, so the performance is severely suffered  Our methods introduce auxiliary regularization terms to control such a bias problem and thus promote the model significantly 
Related Work
There have been much work that incorporate prior knowledge into learning, and two related lines are surveyed here  One is to use prior knowledge to label unlabeled instances and then apply a standard learning algorithm  The other is to constrain the model directly with prior knowledge 
Liu et al text manually labeled features which are highly predictive to unsupervised clustering assignments and use them to label unlabeled data  Chang et al guiding proposed constraint driven learning  They first used constraints and the learned model to annotate unlabeled instances, and then updated the model with the newly labeled data  Daumé daume2008cross proposed a self training method in which several models are trained on the same dataset, and only unlabeled instances that satisfy the cross task knowledge constraints are used in the self training process 
MaCallum et al gec proposed generalized expectation(GE) criteria which formalised the knowledge as constraint terms about the expectation of the model into the objective function Graça et al pr proposed posterior regularization(PR) framework which projects the model's posterior onto a set of distributions that satisfy the auxiliary constraints  Druck et al ge-fl explored constraints of labeled features in the framework of GE by forcing the model's predicted feature distribution to approach the reference distribution  Andrzejewski et al andrzejewski2011framework proposed a framework in which general domain knowledge can be easily incorporated into LDA  Altendorf et al altendorf2012learning explored monotonicity constraints to improve the accuracy while learning from sparse data  Chen et al chen2013leveraging tried to learn comprehensible topic models by leveraging multi-domain knowledge 
Mann and McCallum simple,generalized incorporated not only labeled features but also other knowledge like class distribution into the objective function of GE-FL  But they discussed only from the semi-supervised perspective and did not investigate into the robustness problem, unlike what we addressed in this paper 
There are also some active learning methods trying to use prior knowledge  Raghavan et al feedback proposed to use feedback on instances and features interlacedly, and demonstrated that feedback on features boosts the model much  Druck et al active proposed an active learning method which solicits labels on features rather than on instances and then used GE-FL to train the model 
Conclusion and Discussions
This paper investigates into the problem of how to leverage prior knowledge robustly in learning models  We propose three regularization terms on top of generalized expectation criteria  As demonstrated by the experimental results, the performance can be considerably improved when taking into account these factors  Comparative results show that our proposed methods is more effective and works more robustly against baselines  To the best of our knowledge, this is the first work to address the robustness problem of leveraging knowledge, and may inspire other research 
We then present more detailed discussions about the three regularization methods  Incorporating neutral features is the simplest way of regularization, which doesn't require any modification of GE-FL but just finding out some common features  But as Figure 1 (a) shows, only using neutral features are not strong enough to handle extremely unbalanced labeled features 
The maximum entropy regularization term shows the strong ability of controlling unbalance 
This method doesn't need any extra knowledge, and is thus suitable when we know nothing about the corpus  But this method assumes that the categories are uniformly distributed, which may not be the case in practice, and it will have a degraded performance if the assumption is violated (see Figure 1 (b), Figure 2 (b), Figure 3 (a)) 
The KL divergence performs much better on unbalanced corpora than other methods  The reason is that KL divergence utilizes the reference class distribution and doesn't make any assumptions  The fact suggests that additional knowledge does benefit the model 
However, the KL divergence term requires providing the true class distribution  Sometimes, we may have the exact knowledge about the true distribution, but sometimes we may not  Fortunately, the model is insensitive to the true distribution and therefore a rough estimation of the true distribution is sufficient  In our experiments, when the true class distribution is 1:2, where the reference class distribution is set to 1:1 5/1:2/1:2 5, the accuracy is 0 755/0 756/0 760 respectively  This provides us the possibility to perform simple computing on the corpus to obtain the distribution in reality  Or, we can set the distribution roughly with domain expertise 

ground_truth: Unidirectional LSTM networks with 2, 6, 7, 8, and  9 layers.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What NLP tasks do they consider?
Context 4: 5 after embedding layer, the output between LSTM layers, and the output of final LSTM layer  (2) Advanced dropout: We use the same dropout techniques with the same dropout values as in BIBREF0   We call this model as AWD-PRU 
Results
Table TABREF23 compares the performance of the PRU with state-of-the-art methods  We can see that the PRU achieves the best performance with fewer parameters 
PRUs achieve either the same or better performance than LSTMs  In particular, the performance of PRUs improves with the increasing value of INLINEFORM0   At INLINEFORM1 , PRUs outperform LSTMs by about 4 points on the PTB dataset and by about 3 points on the WT-2 dataset  This is explained in part by the regularization effect of the grouped linear transformation (Figure FIGREF1 )  With grouped linear and pyramidal transformations, PRUs learn rich representations at very high dimensional space while learning fewer parameters  On the other hand, LSTMs overfit to the training data at such high dimensions and learn INLINEFORM2 to INLINEFORM3 more parameters than PRUs 
With the advanced dropouts, the performance of PRUs improves by about 4 points on the PTB dataset and 7 points on the WT-2 dataset  This further improves with finetuning on the PTB (about 2 points) and WT-2 (about 1 point) datasets 
For similar number of parameters, the PRU with standard dropout outperforms most of the state-of-the-art methods by large margin on the PTB dataset (e g  RAN BIBREF7 by 16 points with 4M less parameters, QRNN BIBREF33 by 16 points with 1M more parameters, and NAS BIBREF31 by 1 58 points with 6M less parameters)  With advanced dropouts, the PRU delivers the best performance  On both datasets, the PRU improves the perplexity by about 1 point while learning 15-20% fewer parameters 
PRU is a drop-in replacement for LSTM, therefore, it can improve language models with modern inference techniques such as dynamic evaluation BIBREF21   When we evaluate PRU-based language models (only with standard dropout) with dynamic evaluation on the PTB test set, the perplexity of PRU ( INLINEFORM0 ) improves from 62 42 to 55 23 while the perplexity of an LSTM ( INLINEFORM1 ) with similar settings improves from 66 29 to 58 79; suggesting that modern inference techniques are equally applicable to PRU-based language models 
Analysis
It is shown above that the PRU can learn representations at higher dimensionality with more generalization power, resulting in performance gains for language modeling  A closer analysis of the impact of the PRU in a language modeling system reveals several factors that help explain how the PRU achieves these gains 
As exemplified in Table TABREF34 , the PRU tends toward more confident decisions, placing more of the probability mass on the top next-word prediction than the LSTM  To quantify this effect, we calculate the entropy of the next-token distribution for both the PRU and the LSTM using 3687 contexts from the PTB validation set  Figure FIGREF32 shows a histogram of the entropies of the distribution, where bins of size 0 23 are used to effect categories  We see that the PRU more often produces lower entropy distributions corresponding to higher confidences for next-token choices  This is evidenced by the mass of the red PRU curve lying in the lower entropy ranges compared to the blue LSTM's curve  The PRU can produce confident decisions in part because more information is encoded in the higher dimensional context vectors 
The PRU has the ability to model individual words at different resolutions through the pyramidal transform; which provides multiple paths for the gradient to the embedding layer (similar to multi-task learning) and improves the flow of information  When considering the embeddings by part of speech, we find that the pyramid level 1 embeddings exhibit higher variance than the LSTM across all POS categories (Figure FIGREF33 ), and that pyramid level 2 embeddings show extremely low variance  We hypothesize that the LSTM must encode both coarse group similarities and individual word differences into the same vector space, reducing the space between individual words of the same category  The PRU can rely on the subsampled embeddings to account for coarse-grained group similarities, allowing for finer individual word distinctions in the embedding layer  This hypothesis is strengthened by the entropy results described above: a model which can make finer distinctions between individual words can more confidently assign probability mass  A model that cannot make these distinctions, such as the LSTM, must spread its probability mass across a larger class of similar words 
Saliency analysis using gradients help identify relevant words in a test sequence that contribute to the prediction BIBREF34 , BIBREF35 , BIBREF36   These approaches compute the relevance as the squared norm of the gradients obtained through back-propagation  Table TABREF34 visualizes the heatmaps for different sequences  PRUs, in general, give more relevance to contextual words than LSTMs, such as southeast (sample 1), cost (sample 2), face (sample 4), and introduced (sample 5), which help in making more confident decisions  Furthermore, when gradients during back-propagation are visualized BIBREF37 (Table TABREF34 ), we find that PRUs have better gradient coverage than LSTMs, suggesting PRUs use more features than LSTMs that contributes to the decision  This also suggests that PRUs update more parameters at each iteration which results in faster training  Language model in BIBREF0 takes 500 and 750 epochs to converge with PRU and LSTM as a recurrent unit, respectively 
Ablation studies
In this section, we provide a systematic analysis of our design choices  Our training methodology is the same as described in Section SECREF19 with the standard dropouts  For a thorough understanding of our design choices, we use a language model with a single layer of PRU and fix the size of embedding and hidden layers to 600  The word-level perplexities are reported on the validation sets of the PTB and the WT-2 datasets 
The two hyper-parameters that control the trade-off between performance and number of parameters in PRUs are the number of pyramidal levels INLINEFORM0 and groups INLINEFORM1   Figure FIGREF35 provides a trade-off between perplexity and recurrent unit (RU) parameters 
Variable INLINEFORM0 and fixed INLINEFORM1 : When we increase the number of pyramidal levels INLINEFORM2 at a fixed value of INLINEFORM3 , the performance of the PRU drops by about 1 to 4 points while reducing the total number of recurrent unit parameters by up to 15%  We note that the PRU with INLINEFORM4 at INLINEFORM5 delivers similar performance as the LSTM while learning about 15% fewer recurrent unit parameters 
Fixed INLINEFORM0 and variable INLINEFORM1 : When we vary the value of INLINEFORM2 at fixed number of pyramidal levels INLINEFORM3 , the total number of recurrent unit parameters decreases significantly with a minimal impact on the perplexity  For example, PRUs with INLINEFORM4 and INLINEFORM5 learns 77% fewer recurrent unit parameters while its perplexity (lower is better) increases by about 12% in comparison to LSTMs  Moreover, the decrease in number of parameters at higher value of INLINEFORM6 enables PRUs to learn the representations in high dimensional space with better generalizability (Table TABREF23 ) 
Table TABREF43 shows the impact of different transformations of the input vector INLINEFORM0 and the context vector INLINEFORM1   We make following observations: (1) Using the pyramidal transformation for the input vectors improves the perplexity by about 1 point on both the PTB and WT-2 datasets while reducing the number of recurrent unit parameters by about 14% (see R1 and R4)  We note that the performance of the PRU drops by up to 1 point when residual connections are not used (R4 and R6)  (2) Using the grouped linear transformation for context vectors reduces the total number of recurrent unit parameters by about 75% while the performance drops by about 11% (see R3 and R4)  When we use the pyramidal transformation instead of the linear transformation, the performance drops by up to 2% while there is no significant drop in the number of parameters (R4 and R5) 
We set sub-sampling kernel INLINEFORM0 (Eq  EQREF12 ) with stride INLINEFORM1 and size of 3 ( INLINEFORM2 ) in four different ways: (1) Skip: We skip every other element in the input vector  (2) Convolution: We initialize the elements of INLINEFORM3 randomly from normal distribution and learn them during training the model  We limit the output values between -1 and 1 using INLINEFORM4 activation function to make training stable  (3) Avg  pool: We initialize the elements of INLINEFORM5 to INLINEFORM6   (4) Max pool: We select the maximum value in the kernel window INLINEFORM7  
Table TABREF45 compares the performance of the PRU with different sampling methods  Average pooling performs the best while skipping give comparable performance  Both of these methods enable the network to learn richer word representations while representing the input vector in different forms, thus delivering higher performance  Surprisingly, a convolution-based sub-sampling method does not perform as well as the averaging method  The INLINEFORM0 function used after convolution limits the range of output values which are further limited by the LSTM gating structure, thereby impeding in the flow of information inside the cell  Max pooling forces the network to learn representations from high magnitude elements, thus distinguishing features between elements vanishes, resulting in poor performance 
Conclusion
We introduce the Pyramidal Recurrent Unit, which better model contextual information by admitting higher dimensional representations with good generalizability  When applied to the task of language modeling, PRUs improve perplexity across several settings, including recent state-of-the-art systems  Our analysis shows that the PRU improves the flow of gradient and expand the word embedding subspace, resulting in more confident decisions  Here we have shown improvements for language modeling  In future, we plan to study the performance of PRUs on different tasks, including machine translation and question answering  In addition, we will study the performance of the PRU on language modeling with more recent inference techniques, such as dynamic evaluation and mixture of softmax 
Acknowledgments
This research was supported by NSF (IIS 1616112, III 1703166), Allen Distinguished Investigator Award, and gifts from Allen Institute for AI, Google, Amazon, and Bloomberg  We are grateful to Aaron Jaech, Hannah Rashkin, Mandar Joshi, Aniruddha Kembhavi, and anonymous reviewers for their helpful comments 

ground_truth: text classification for themes including sentiment, web-page, science, medical and healthcare