prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How are weights dynamically adjusted?
Context 1: Introduction
Data imbalance is a common issue in a variety of NLP tasks such as tagging and machine reading comprehension  Table TABREF3 gives concrete examples: for the Named Entity Recognition (NER) task BIBREF2, BIBREF3, most tokens are backgrounds with tagging class $O$  Specifically, the number of tokens tagging class $O$ is 5 times as many as those with entity labels for the CoNLL03 dataset and 8 times for the OntoNotes5 0 dataset; Data-imbalanced issue is more severe for MRC tasks BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8 with the value of negative-positive ratio being 50-200 
Data imbalance results in the following two issues: (1) the training-test discrepancy: Without balancing the labels, the learning process tends to converge to a point that strongly biases towards class with the majority label  This actually creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function while at test time, F1 score concerns more about positive examples; (2) the overwhelming effect of easy-negative examples  As pointed out by meng2019dsreg, significantly large number of negative examples also means that the number of easy-negative example is large  The huge number of easy examples tends to overwhelm the training, making the model not sufficiently learned to distinguish between positive examples and hard-negative examples  The cross-entropy objective (CE for short) or maximum likelihood (MLE) objective, which is widely adopted as the training objective for data-imbalanced NLP tasks BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15, handles neither of the issues 
To handle the first issue, we propose to replace CE or MLE with losses based on the Sørensen–Dice coefficient BIBREF0 or Tversky index BIBREF1  The Sørensen–Dice coefficient, dice loss for short, is the harmonic mean of precision and recall  It attaches equal importance to false positives (FPs) and false negatives (FNs) and is thus more immune to data-imbalanced datasets  Tversky index extends dice loss by using a weight that trades precision and recall, which can be thought as the approximation of the $F_{\beta }$ score, and thus comes with more flexibility  Therefore, We use dice loss or Tversky index to replace CE loss to address the first issue 
Only using dice loss or Tversky index is not enough since they are unable to address the dominating influence of easy-negative examples  This is intrinsically because dice loss is actually a hard version of the F1 score  Taking the binary classification task as an example, at test time, an example will be classified as negative as long as its probability is smaller than 0 5, but training will push the value to 0 as much as possible  This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily  Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones  Inspired by the idea of focal loss BIBREF16 in computer vision, we propose a dynamic weight adjusting strategy, which associates each training example with a weight in proportion to $(1-p)$, and this weight dynamically changes as training proceeds  This strategy helps to deemphasize confident examples during training as their $p$ approaches the value of 1, makes the model attentive to hard-negative examples, and thus alleviates the dominating effect of easy-negative examples 
Combing both strategies, we observe significant performance boosts on a wide range of data imbalanced NLP tasks  Notably, we are able to achieve SOTA results on CTB5 (97 92, +1 86), CTB6 (96 57, +1 80) and UD1 4 (96 98, +2 19) for the POS task; SOTA results on CoNLL03 (93 33, +0 29), OntoNotes5 0 (92 07, +0 96)), MSRA 96 72(+0 97) and OntoNotes4 0 (84 47,+2 36) for the NER task; along with competitive results on the tasks of machine reading comprehension and paraphrase identification 
The rest of this paper is organized as follows: related work is presented in Section 2  We describe different training objectives in Section 3  Experimental results are presented in Section 4  We perform ablation studies in Section 5, followed by a brief conclusion in Section 6 
Related Work ::: Data Resample
The idea of weighting training examples has a long history  Importance sampling BIBREF17 assigns weights to different samples and changes the data distribution  Boosting algorithms such as AdaBoost BIBREF18 select harder examples to train subsequent classifiers  Similarly, hard example mining BIBREF19 downsamples the majority class and exploits the most difficult examples  Oversampling BIBREF20, BIBREF21 is used to balance the data distribution  Another line of data resampling is to dynamically control the weights of examples as training proceeds  For example, focal loss BIBREF16 used a soft weighting scheme that emphasizes harder examples during training  In self-paced learning BIBREF22, example weights are obtained through optimizing the weighted training loss which encourages learning easier examples first  At each training step, self-paced learning algorithm optimizes model parameters and example weights jointly  Other works BIBREF23, BIBREF24 adjusted the weights of different training examples based on training loss  Besides, recent work BIBREF25, BIBREF26 proposed to learn a separate network to predict sample weights 
Related Work ::: Data Imbalance Issue in Object Detection
The background-object label imbalance issue is severe and thus well studied in the field of object detection BIBREF27, BIBREF28, BIBREF29, BIBREF30, BIBREF31  The idea of hard negative mining (HNM) BIBREF30 has gained much attention recently  shrivastava2016ohem proposed the online hard example mining (OHEM) algorithm in an iterative manner that makes training progressively more difficult, and pushes the model to learn better  ssd2016liu sorted all of the negative samples based on the confidence loss and picking the training examples with the negative-positive ratio at 3:1  pang2019rcnn proposed a novel method called IoU-balanced sampling and aploss2019chen designed a ranking model to replace the conventional classification task with a average-precision loss to alleviate the class imbalance issue  The efforts made on object detection have greatly inspired us to solve the data imbalance issue in NLP 
Losses ::: Notation
For illustration purposes, we use the binary classification task to demonstrate how different losses work  The mechanism can be easily extended to multi-class classification 
Let $\lbrace x_i\rbrace $ denote a set of instances  Each $x_i$ is associated with a golden label vector $y_i = [y_{i0},y_{i1} ]$, where $y_{i1}\in \lbrace 0,1\rbrace $ and $y_{i0}\in \lbrace 0,1\rbrace $ respectively denote the positive and negative classes, and thus $y_i$ can be either $[0,1]$ or $[0,1]$  Let $p_i = [p_{i0},p_{i1} ]$ denote the probability vector, and $p_{i1}$ and $p_{i0}$ respectively denote the probability that a model assigns the positive and negative label to $x_i$ 
Losses ::: Cross Entropy Loss
The vanilla cross entropy (CE) loss is given by:
As can be seen from Eq DISPLAY_FORM8, each $x_i$ contributes equally to the final objective  Two strategies are normally used to address the the case where we wish that not all $x_i$ are treated equal: associating different classes with different weighting factor $\alpha $ or resampling the datasets  For the former, Eq DISPLAY_FORM8 is adjusted as follows:
where $\alpha _i\in [0,1]$ may be set by the inverse class frequency or treated as a hyperparameter to set by cross validation  In this work, we use $\lg (\frac{n-n_t}{n_t}+K)$ to calculate the coefficient $\alpha $, where $n_t$ is the number of samples with class $t$ and $n$ is the total number of samples in the training set  $K$ is a hyperparameter to tune  The data resampling strategy constructs a new dataset by sampling training examples from the original dataset based on human-designed criteria, e g , extract equal training samples from each class  Both strategies are equivalent to changing the data distribution and thus are of the same nature  Empirically, these two methods are not widely used due to the trickiness of selecting $\alpha $ especially for multi-class classification tasks and that inappropriate selection can easily bias towards rare classes BIBREF32 
Losses ::: Dice coefficient and Tversky index
Sørensen–Dice coefficient BIBREF0, BIBREF33, dice coefficient (DSC) for short, is a F1-oriented statistic used to gauge the similarity of two sets  Given two sets $A$ and $B$, the dice coefficient between them is given as follows:
In our case, $A$ is the set that contains of all positive examples predicted by a specific model, and $B$ is the set of all golden positive examples in the dataset  When applied to boolean data with the definition of true positive (TP), false positive (FP), and false negative (FN), it can be then written as follows:
For an individual example $x_i$, its corresponding DSC loss is given as follows:
As can be seen, for a negative example with $y_{i1}=0$, it does not contribute to the objective  For smoothing purposes, it is common to add a $\gamma $ factor to both the nominator and the denominator, making the form to be as follows:
As can be seen, negative examples, with $y_{i1}$ being 0 and DSC being $\frac{\gamma }{ p_{i1}+\gamma }$, also contribute to the training  Additionally, milletari2016v proposed to change the denominator to the square form for faster convergence, which leads to the following dice loss (DL):
Another version of DL is to directly compute set-level dice coefficient instead of the sum of individual dice coefficient  We choose the latter due to ease of optimization 
Tversky index (TI), which can be thought as the approximation of the $F_{\beta }$ score, extends dice coefficient to a more general case  Given two sets $A$ and $B$, tversky index is computed as follows:
Tversky index offers the flexibility in controlling the tradeoff between false-negatives and false-positives  It degenerates to DSC if $\alpha =\beta =0 5$  The Tversky loss (TL) for the training set $\lbrace x_i,y_i\rbrace $ is thus as follows:
Losses ::: Self-adusting Dice Loss
Consider a simple case where the dataset consists of only one example $x_i$, which is classified as positive as long as $p_{i1}$ is larger than 0 5  The computation of $F1$ score is actually as follows:
Comparing Eq DISPLAY_FORM14 with Eq DISPLAY_FORM22, we can see that Eq DISPLAY_FORM14 is actually a soft form of $F1$, using a continuous $p$ rather than the binary $\mathbb {I}( p_{i1}>0 5)$  This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily  Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones, which has a huge negative effect on the final F1 performance 
To address this issue, we propose to multiply the soft probability $p$ with a decaying factor $(1-p)$, changing Eq DISPLAY_FORM22 to the following form:
One can think $(1-p_{i1})$ as a weight associated with each example, which changes as training proceeds  The intuition of changing $p_{i1}$ to $(1-p_{i1}) p_{i1}$ is to push down the weight of easy examples  For easy examples whose probability are approaching 0 or 1, $(1-p_{i1}) p_{i1}$ makes the model attach significantly less focus to them  Figure FIGREF23 gives gives an explanation from the perspective in derivative: the derivative of $\frac{(1-p)p}{1+(1-p)p}$ with respect to $p$ approaches 0 immediately after $p$ approaches 0, which means the model attends less to examples once they are correctly classified 
A close look at Eq DISPLAY_FORM14 reveals that it actually mimics the idea of focal loss (FL for short) BIBREF16 for object detection in vision  Focal loss was proposed for one-stage object detector to handle foreground-background tradeoff encountered during training  It down-weights the loss assigned to well-classified examples by adding a $(1-p)^{\beta }$ factor, leading the final loss to be $(1-p)^{\beta }\log p$ 
In Table TABREF18, we show the losses used in our experiments, which is described in the next section 
Experiments
We evaluate the proposed method on four NLP tasks: part-of-speech tagging, named entity recognition, machine reading comprehension and paraphrase identification  Baselines in our experiments are optimized by using the standard cross-entropy training objective 
Experiments ::: Part-of-Speech Tagging
Part-of-speech tagging (POS) is the task of assigning a label (e g , noun, verb, adjective) to each word in a given text  In this paper, we choose BERT as the backbone and conduct experiments on three Chinese POS datasets  We report the span-level micro-averaged precision, recall and F1 for evaluation  Hyperparameters are tuned on the corresponding development set of each dataset 
Experiments ::: Part-of-Speech Tagging ::: Datasets
We conduct experiments on the widely used Chinese Treebank 5 0, 6 0 as well as UD1 4 
CTB5 is a Chinese dataset for tagging and parsing, which contains 507,222 words, 824,983 characters and 18,782 sentences extracted from newswire sources 
CTB6 is an extension of CTB5, containing 781,351 words, 1,285,149 characters and 28,295 sentences 
UD is the abbreviation of Universal Dependencies, which is a framework for consistent annotation of grammar (parts of speech, morphological features, and syntactic dependencies) across different human languages  In this work, we use UD1 4 for Chinese POS tagging 
Experiments ::: Part-of-Speech Tagging ::: Baselines
We use the following baselines:
Joint-POS: shao2017character jointly learns Chinese word segmentation and POS 
Lattice-LSTM: lattice2018zhang constructs a word-character lattice 
Bert-Tagger: devlin2018bert treats part-of-speech as a tagging task 
Experiments ::: Part-of-Speech Tagging ::: Results
Table presents the experimental results on the POS task  As can be seen, the proposed DSC loss outperforms the best baseline results by a large margin, i e , outperforming BERT-tagger by +1 86 in terms of F1 score on CTB5, +1 80 on CTB6 and +2 19 on UD1 4  As far as we are concerned, we are achieving SOTA performances on the three datasets  Weighted cross entropy and focal loss only gain a little performance improvement on CTB5 and CTB6, and the dice loss obtains huge gain on CTB5 but not on CTB6, which indicates the three losses are not consistently robust in resolving the data imbalance issue  The proposed DSC loss performs robustly on all the three datasets 
Experiments ::: Named Entity Recognition
Named entity recognition (NER) refers to the task of detecting the span and semantic category of entities from a chunk of text  Our implementation uses the current state-of-the-art BERT-MRC model proposed by xiaoya2019ner as a backbone  For English datasets, we use BERT$_\text{Large}$ English checkpoints, while for Chinese we use the official Chinese checkpoints  We report span-level micro-averaged precision, recall and F1-score  Hyperparameters are tuned on the development set of each dataset 
Experiments ::: Named Entity Recognition ::: Datasets
For the NER task, we consider both Chinese datasets, i e , OntoNotes4 0 BIBREF34 and MSRA BIBREF35, and English datasets, i e , CoNLL2003 BIBREF36 and OntoNotes5 0 BIBREF37 
CoNLL2003 is an English dataset with 4 entity types: Location, Organization, Person and Miscellaneous  We followed data processing protocols in BIBREF14 
English OntoNotes5 0 consists of texts from a wide variety of sources and contains 18 entity types  We use the standard train/dev/test split of CoNLL2012 shared task 
Chinese MSRA performs as a Chinese benchmark dataset containing 3 entity types  Data in MSRA is collected from news domain  Since the development set is not provided in the original MSRA dataset, we randomly split the training set into training and development splits by 9:1  We use the official test set for evaluation 
Chinese OntoNotes4 0 is a Chinese dataset and consists of texts from news domain, which has 18 entity types  In this paper, we take the same data split as wu2019glyce did 
Experiments ::: Named Entity Recognition ::: Baselines
We use the following baselines:
ELMo: a tagging model from peters2018deep 
Lattice-LSTM: lattice2018zhang constructs a word-character lattice, only used in Chinese datasets 
CVT: from kevin2018cross, which uses Cross-View Training(CVT) to improve the representations of a Bi-LSTM encoder 
Bert-Tagger: devlin2018bert treats NER as a tagging task 
Glyce-BERT: wu2019glyce combines glyph information with BERT pretraining 
BERT-MRC: The current SOTA model for both Chinese and English NER datasets proposed by xiaoya2019ner, which formulate NER as machine reading comprehension task 
Experiments ::: Named Entity Recognition ::: Results
Table shows experimental results on NER datasets  For English datasets including CoNLL2003 and OntoNotes5 0, our proposed method outperforms BERT-MRCBIBREF38 by +0 29 and +0 96 respectively  We observe huge performance boosts on Chinese datasets, achieving F1 improvements by +0 97 and +2 36 on MSRA and OntoNotes4 0, respectively  As far as we are concerned, we are setting new SOTA performances on all of the four NER datasets 
Experiments ::: Machine Reading Comprehension
Machine reading comprehension (MRC) BIBREF39, BIBREF40, BIBREF41, BIBREF40, BIBREF42, BIBREF15 has become a central task in natural language understanding  MRC in the SQuAD-style is to predict the answer span in the passage given a question and the passage  In this paper, we choose the SQuAD-style MRC task and report Extract Match (EM) in addition to F1 score on validation set  All hyperparameters are tuned on the development set of each dataset 
Experiments ::: Machine Reading Comprehension ::: Datasets
The following five datasets are used for MRC task: SQuAD v1 1, SQuAD v2 0 BIBREF4, BIBREF6 and Quoref BIBREF8 
SQuAD v1 1 and SQuAD v2 0 are the most widely used QA benchmarks  SQuAD1 1 is a collection of 100K crowdsourced question-answer pairs, and SQuAD2 0 extends SQuAD1 1 allowing no short answer exists in the provided passage 
Quoref is a QA dataset which tests the coreferential reasoning capability of reading comprehension systems, containing 24K questions over 4 7K paragraphs from Wikipedia 
Experiments ::: Machine Reading Comprehension ::: Baselines
We use the following baselines:
QANet: qanet2018 builds a model based on convolutions and self-attention  Convolution to model local interactions and self-attention to model global interactions 
BERT: devlin2018bert treats NER as a tagging task 
XLNet: xlnet2019 proposes a generalized autoregressive pretraining method that enables learning bidirectional contexts 
Experiments ::: Machine Reading Comprehension ::: Results
Table shows the experimental results for MRC tasks  With either BERT or XLNet, our proposed DSC loss obtains significant performance boost on both EM and F1  For SQuADv1 1, our proposed method outperforms XLNet by +1 25 in terms of F1 score and +0 84 in terms of EM and achieves 87 65 on EM and 89 51 on F1 for SQuAD v2 0  Moreover, on QuoRef, the proposed method surpasses XLNet results by +1 46 on EM and +1 41 on F1  Another observation is that, XLNet outperforms BERT by a huge margin, and the proposed DSC loss can obtain further performance improvement by an average score above 1 0 in terms of both EM and F1, which indicates the DSC loss is complementary to the model structures 
Experiments ::: Paraphrase Identification
Paraphrases are textual expressions that have the same semantic meaning using different surface words  Paraphrase identification (PI) is the task of identifying whether two sentences have the same meaning or not  We use BERT BIBREF11 and XLNet BIBREF43 as backbones and report F1 score for comparison  Hyperparameters are tuned on the development set of each dataset 
Experiments ::: Paraphrase Identification ::: Datasets
We conduct experiments on two widely used datasets for PI task: MRPC BIBREF44 and QQP 
MRPC is a corpus of sentence pairs automatically extracted from online news sources, with human annotations of whether the sentence pairs are semantically equivalent  The MRPC dataset has imbalanced classes (68% positive, 32% for negative) 
QQP is a collection of question pairs from the community question-answering website Quora  The class distribution in QQP is also unbalanced (37% positive, 63% negative) 
Experiments ::: Paraphrase Identification ::: Results
Table shows the results for PI task  We find that replacing the training objective with DSC introduces performance boost for both BERT and XLNet  Using DSC loss improves the F1 score by +0 58 for MRPC and +0 73 for QQP 
Ablation Studies ::: The Effect of Dice Loss on Accuracy-oriented Tasks
We argue that the most commonly used cross-entropy objective is actually accuracy-oriented, whereas the proposed dice loss (DL) performs as a hard version of F1-score  To explore the effect of the dice loss on accuracy-oriented tasks such as text classification, we conduct experiments on the Stanford Sentiment Treebank sentiment classification datasets including SST-2 and SST-5  We fine-tune BERT$_\text{Large}$ with different training objectives  Experiment results for SST are shown in   For SST-5, BERT with CE achieves 55 57 in terms of accuracy, with DL and DSC losses slightly degrade the accuracy performance and achieve 54 63 and 55 19, respectively  For SST-2, BERT with CE achieves 94 9 in terms of accuracy  The same as SST-5, we observe a slight performance drop with DL and DSC, which means that the dice loss actually works well for F1 but not for accuracy 
Ablation Studies ::: The Effect of Hyperparameters in Tversky index
As mentioned in Section SECREF10, Tversky index (TI) offers the flexibility in controlling the tradeoff between false-negatives and false-positives  In this subsection, we explore the effect of hyperparameters (i e , $\alpha $ and $\beta $) in TI to test how they manipulate the tradeoff  We conduct experiments on the Chinese OntoNotes4 0 NER dataset and English QuoRef MRC dataset to examine the influence of tradeoff between precision and recall  Experiment results are shown in Table   The highest F1 for Chinese OntoNotes4 0 is 84 67 when $\alpha $ is set to 0 6 while for QuoRef, the highest F1 is 68 44 when $\alpha $ is set to 0 4  In addition, we can observe that the performance varies a lot as $\alpha $ changes in distinct datasets, which shows that the hyperparameters $\alpha ,\beta $ play an important role in the proposed method 
Conclusion
In this paper, we alleviate the severe data imbalance issue in NLP tasks  We propose to use dice loss in replacement of the standard cross-entropy loss, which performs as a soft version of F1 score  Using dice loss can help narrow the gap between training objectives and evaluation metrics  Empirically, we show that the proposed training objective leads to significant performance boost for part-of-speech, named entity recognition, machine reading comprehension and paraphrase identification tasks 
Context 2: Introduction
We posses a wealth of prior knowledge about many natural language processing tasks  For example, in text categorization, we know that words such as NBA, player, and basketball are strong indicators of the sports category BIBREF0 , and words like terrible, boring, and messing indicate a negative polarity while words like perfect, exciting, and moving suggest a positive polarity in sentiment classification 
A key problem arisen here, is how to leverage such knowledge to guide the learning process, an interesting problem for both NLP and machine learning communities  Previous studies addressing the problem fall into several lines  First, to leverage prior knowledge to label data BIBREF1 , BIBREF2   Second, to encode prior knowledge with a prior on parameters, which can be commonly seen in many Bayesian approaches BIBREF3 , BIBREF4   Third, to formalise prior knowledge with additional variables and dependencies BIBREF5   Last, to use prior knowledge to control the distributions over latent output variables BIBREF6 , BIBREF7 , BIBREF8 , which makes the output variables easily interpretable 
However, a crucial problem, which has rarely been addressed, is the bias in the prior knowledge that we supply to the learning model  Would the model be robust or sensitive to the prior knowledge? Or, which kind of knowledge is appropriate for the task? Let's see an example: we may be a baseball fan but unfamiliar with hockey so that we can provide a few number of feature words of baseball, but much less of hockey for a baseball-hockey classification task  Such prior knowledge may mislead the model with heavy bias to baseball  If the model cannot handle this situation appropriately, the performance may be undesirable 
In this paper, we investigate into the problem in the framework of Generalized Expectation Criteria BIBREF7   The study aims to reveal the factors of reducing the sensibility of the prior knowledge and therefore to make the model more robust and practical  To this end, we introduce auxiliary regularization terms in which our prior knowledge is formalized as distribution over output variables  Recall the example just mentioned, though we do not have enough knowledge to provide features for class hockey, it is easy for us to provide some neutral words, namely words that are not strong indicators of any class, like player here  As one of the factors revealed in this paper, supplying neutral feature words can boost the performance remarkably, making the model more robust 
More attractively, we do not need manual annotation to label these neutral feature words in our proposed approach 
More specifically, we explore three regularization terms to address the problem: (1) a regularization term associated with neutral features; (2) the maximum entropy of class distribution regularization term; and (3) the KL divergence between reference and predicted class distribution  For the first manner, we simply use the most common features as neutral features and assume the neutral features are distributed uniformly over class labels  For the second and third one, we assume we have some knowledge about the class distribution which will be detailed soon later 
To summarize, the main contributions of this work are as follows:
The rest of the paper is structured as follows: In Section 2, we briefly describe the generalized expectation criteria and present the proposed regularization terms  In Section 3, we conduct extensive experiments to justify the proposed methods  We survey related work in Section 4, and summarize our work in Section 5 
Method
We address the robustness problem on top of GE-FL BIBREF0 , a GE method which leverages labeled features as prior knowledge  A labeled feature is a strong indicator of a specific class and is manually provided to the classifier  For example, words like amazing, exciting can be labeled features for class positive in sentiment classification 
Generalized Expectation Criteria
Generalized expectation (GE) criteria BIBREF7 provides us a natural way to directly constrain the model in the preferred direction  For example, when we know the proportion of each class of the dataset in a classification task, we can guide the model to predict out a pre-specified class distribution 
Formally, in a parameter estimation objective function, a GE term expresses preferences on the value of some constraint functions about the model's expectation  Given a constraint function $G({\rm x}, y)$ , a conditional model distribution $p_\theta (y|\rm x)$ , an empirical distribution $\tilde{p}({\rm x})$ over input samples and a score function $S$ , a GE term can be expressed as follows:
$$S(E_{\tilde{p}({\rm x})}[E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]])$$   (Eq  4)
Learning from Labeled Features
Druck et al  ge-fl proposed GE-FL to learn from labeled features using generalized expectation criteria  When given a set of labeled features $K$ , the reference distribution over classes of these features is denoted by $\hat{p}(y| x_k), k \in K$   GE-FL introduces the divergence between this reference distribution and the model predicted distribution $p_\theta (y | x_k)$ , as a term of the objective function:
$$\mathcal {O} = \sum _{k \in K} KL(\hat{p}(y|x_k) || p_\theta (y | x_k)) + \sum _{y,i} \frac{\theta _{yi}^2}{2 \sigma ^2}$$   (Eq  6)
where $\theta _{yi}$ is the model parameter which indicates the importance of word $i$ to class $y$   The predicted distribution $p_\theta (y | x_k)$ can be expressed as follows: $ p_\theta (y | x_k) = \frac{1}{C_k} \sum _{\rm x} p_\theta (y|{\rm x})I(x_k) $
in which $I(x_k)$ is 1 if feature $k$ occurs in instance ${\rm x}$ and 0 otherwise, $C_k = \sum _{\rm x} I(x_k)$ is the number of instances with a non-zero value of feature $k$ , and $p_\theta (y|{\rm x})$ takes a softmax form as follows: $ p_\theta (y|{\rm x}) = \frac{1}{Z(\rm x)}\exp (\sum _i \theta _{yi}x_i)  $
To solve the optimization problem, L-BFGS can be used for parameter estimation 
In the framework of GE, this term can be obtained by setting the constraint function $G({\rm x}, y) = \frac{1}{C_k} \vec{I} (y)I(x_k)$ , where $\vec{I}(y)$ is an indicator vector with 1 at the index corresponding to label $y$ and 0 elsewhere 
Regularization Terms
GE-FL reduces the heavy load of instance annotation and performs well when we provide prior knowledge with no bias  In our experiments, we observe that comparable numbers of labeled features for each class have to be supplied  But as mentioned before, it is often the case that we are not able to provide enough knowledge for some of the classes  For the baseball-hockey classification task, as shown before, GE-FL will predict most of the instances as baseball  In this section, we will show three terms to make the model more robust 
Neutral features are features that are not informative indicator of any classes, for instance, word player to the baseball-hockey classification task  Such features are usually frequent words across all categories  When we set the preference distribution of the neutral features to be uniform distributed, these neutral features will prevent the model from biasing to the class that has a dominate number of labeled features 
Formally, given a set of neutral features $K^{^{\prime }}$ , the uniform distribution is $\hat{p}_u(y|x_k) = \frac{1}{|C|}, k \in K^{^{\prime }}$ , where $|C|$ is the number of classes  The objective function with the new term becomes
$$\mathcal {O}_{NE} = \mathcal {O} + \sum _{k \in K^{^{\prime }}} KL(\hat{p}_u(y|x_k) || p_\theta (y | x_k)) $$   (Eq  9)
Note that we do not need manual annotation to provide neutral features  One simple way is to take the most common features as neutral features  Experimental results show that this strategy works successfully 
Another way to prevent the model from drifting from the desired direction is to constrain the predicted class distribution on unlabeled data  When lacking knowledge about the class distribution of the data, one feasible way is to take maximum entropy principle, as below:
$$\mathcal {O}_{ME} = \mathcal {O} + \lambda \sum _{y} p(y) \log p(y)$$   (Eq  11)
where $p(y)$ is the predicted class distribution, given by $ p(y) = \frac{1}{|X|} \sum _{\rm x} p_\theta (y | \rm x)  $ To control the influence of this term on the overall objective function, we can tune $\lambda $ according to the difference in the number of labeled features of each class  In this paper, we simply set $\lambda $ to be proportional to the total number of labeled features, say $\lambda = \beta |K|$  
This maximum entropy term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$   Therefore, $E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]$ is just the model distribution $p_\theta (y|{\rm x})$ and its expectation with the empirical distribution $\tilde{p}(\rm x)$ is simply the average over input samples, namely $p(y)$   When $S$ takes the maximum entropy form, we can derive the objective function as above 
Sometimes, we have already had much knowledge about the corpus, and can estimate the class distribution roughly without labeling instances  Therefore, we introduce the KL divergence between the predicted and reference class distributions into the objective function 
Given the preference class distribution $\hat{p}(y)$ , we modify the objective function as follows:
$$\mathcal {O}_{KL} &= \mathcal {O} + \lambda KL(\hat{p}(y) || p(y))$$   (Eq  13)
Similarly, we set $\lambda = \beta |K|$  
This divergence term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$ and setting the score function to $S(\hat{p}, p) = \sum _i \hat{p}_i \log \frac{\hat{p}_i}{p_i}$ , where $p$ and $\hat{p}$ are distributions  Note that this regularization term involves the reference class distribution which will be discussed later 
Experiments
In this section, we first justify the approach when there exists unbalance in the number of labeled features or in class distribution  Then, to test the influence of $\lambda $ , we conduct some experiments with the method which incorporates the KL divergence of class distribution  Last, we evaluate our approaches in 9 commonly used text classification datasets  We set $\lambda = 5|K|$ by default in all experiments unless there is explicit declaration  The baseline we choose here is GE-FL BIBREF0 , a method based on generalization expectation criteria 
Data Preparation
We evaluate our methods on several commonly used datasets whose themes range from sentiment, web-page, science to medical and healthcare  We use bag-of-words feature and remove stopwords in the preprocess stage  Though we have labels of all documents, we do not use them during the learning process, instead, we use the label of features 
The movie dataset, in which the task is to classify the movie reviews as positive or negtive, is used for testing the proposed approaches with unbalanced labeled features, unbalanced datasets or different $\lambda $ parameters  All unbalanced datasets are constructed based on the movie dataset by randomly removing documents of the positive class  For each experiment, we conduct 10-fold cross validation 
As described in BIBREF0 , there are two ways to obtain labeled features  The first way is to use information gain  We first calculate the mutual information of all features according to the labels of the documents and select the top 20 as labeled features for each class as a feature pool  Note that using information gain requires the document label, but this is only to simulate how we human provide prior knowledge to the model  The second way is to use LDA BIBREF9 to select features  We use the same selection process as BIBREF0 , where they first train a LDA on the dataset, and then select the most probable features of each topic (sorted by $P(w_i|t_j)$ , the probability of word $w_i$ given topic $t_j$ ) 
Similar to BIBREF10 , BIBREF0 , we estimate the reference distribution of the labeled features using a heuristic strategy  If there are $|C|$ classes in total, and $n$ classes are associated with a feature $k$ , the probability that feature $k$ is related with any one of the $n$ classes is $\frac{0 9}{n}$ and with any other class is $\frac{0 1}{|C| - n}$  
Neutral features are the most frequent words after removing stop words, and their reference distributions are uniformly distributed  We use the top 10 frequent words as neutral features in all experiments 
With Unbalanced Labeled Features
In this section, we evaluate our approach when there is unbalanced knowledge on the categories to be classified  The labeled features are obtained through information gain  Two settings are chosen:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other  The original balanced movie dataset is used (positive:negative=1:1) 
(b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4) 
As shown in Figure 1 , Maximum entropy principle shows improvement only on the balanced case

ground_truth: One can think $(1-p_{i1})$ as a weight associated with each example, which changes as training proceeds. The intuition of changing $p_{i1}$ to $(1-p_{i1}) p_{i1}$ is to push down the weight of easy examples. For easy examples whose probability are approaching 0 or 1, $(1-p_{i1}) p_{i1}$ makes the model attach significantly less focus to them. Figure FIGREF23 gives gives an explanation from the perspective in derivative: the derivative of $\frac{(1-p)p}{1+(1-p)p}$ with respect to $p$ approaches 0 immediately after $p$ approaches 0, which means the model attends less to examples once they are correctly classified.

associates each training example with a weight in proportion to $(1-p)$, and this weight dynamically changes as training proceeds

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What NLP tasks do they consider?
Context 2:  An obvious reason is that maximum entropy only favors uniform distribution 
Incorporating Neutral features performs similarly to maximum entropy since we assume that neutral words are uniformly distributed  Its accuracy decreases slowly when the number of labeled features becomes larger ( $t>4$ ) (Figure 1 (a)), suggesting that the model gradually biases to the class with more labeled features, just like GE-FL 
Incorporating the KL divergence of class distribution performs much better than GE-FL on both balanced and unbalanced datasets  This shows that it is effective to control the unbalance in labeled features and in the dataset 
With Balanced Labeled Features
We also compare with the baseline when the labeled features are balanced  Similar to the experiment above, the labeled features are obtained by information gain  Two settings are experimented with:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for each class, and conduct comparisons on the original balanced movie dataset (positive:negtive=1:1) 
(b) Similar to (a), but the class distribution is unbalanced, by randomly removing 75% positive documents (positive:negative=1:4) 
Results are shown in Figure 2   When the dataset is balanced (Figure 2 (a)), there is little difference between GE-FL and our methods  The reason is that the proposed regularization terms provide no additional knowledge to the model and there is no bias in the labeled features  On the unbalanced dataset (Figure 2 (b)), incorporating KL divergence is much better than GE-FL since we provide additional knowledge(the true class distribution), but maximum entropy and neutral features are much worse because forcing the model to approach the uniform distribution misleads it 
With Unbalanced Class Distributions
Our methods are also evaluated on datasets with different unbalanced class distributions  We manually construct several movie datasets with class distributions of 1:2, 1:3, 1:4 by randomly removing 50%, 67%, 75% positive documents  The original balanced movie dataset is used as a control group  We test with both balanced and unbalanced labeled features  For the balanced case, we randomly select 10 features from the feature pool for each class, and for the unbalanced case, we select 10 features for one class, and 1 feature for the other  Results are shown in Figure 3  
Figure 3 (a) shows that when the dataset and the labeled features are both balanced, there is little difference between our methods and GE-FL(also see Figure 2 (a))  But when the class distribution becomes more unbalanced, the difference becomes more remarkable  Performance of neutral features and maximum entropy decrease significantly but incorporating KL divergence increases remarkably  This suggests if we have more accurate knowledge about class distribution, KL divergence can guide the model to the right direction 
Figure 3 (b) shows that when the labeled features are unbalanced, our methods significantly outperforms GE-FL  Incorporating KL divergence is robust enough to control unbalance both in the dataset and in labeled features while the other three methods are not so competitive 
The Influence of λ\lambda 
We present the influence of $\lambda $ on the method that incorporates KL divergence in this section  Since we simply set $\lambda = \beta |K|$ , we just tune $\beta $ here  Note that when $\beta = 0$ , the newly introduced regularization term is disappeared, and thus the model is actually GE-FL  Again, we test the method with different $\lambda $ in two settings:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other class  The original balanced movie dataset is used (positive:negative=1:1) 
(b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4) 
Results are shown in Figure 4   As expected, $\lambda $ reflects how strong the regularization is  The model tends to be closer to our preferences with the increasing of $\lambda $ on both cases 
Using LDA Selected Features
We compare our methods with GE-FL on all the 9 datasets in this section  Instead of using features obtained by information gain, we use LDA to select labeled features  Unlike information gain, LDA does not employ any instance labels to find labeled features  In this setting, we can build classification models without any instance annotation, but just with labeled features 
Table 1 shows that our three methods significantly outperform GE-FL  Incorporating neutral features performs better than GE-FL on 7 of the 9 datasets, maximum entropy is better on 8 datasets, and KL divergence better on 7 datasets 
LDA selects out the most predictive features as labeled features without considering the balance among classes  GE-FL does not exert any control on such an issue, so the performance is severely suffered  Our methods introduce auxiliary regularization terms to control such a bias problem and thus promote the model significantly 
Related Work
There have been much work that incorporate prior knowledge into learning, and two related lines are surveyed here  One is to use prior knowledge to label unlabeled instances and then apply a standard learning algorithm  The other is to constrain the model directly with prior knowledge 
Liu et al text manually labeled features which are highly predictive to unsupervised clustering assignments and use them to label unlabeled data  Chang et al guiding proposed constraint driven learning  They first used constraints and the learned model to annotate unlabeled instances, and then updated the model with the newly labeled data  Daumé daume2008cross proposed a self training method in which several models are trained on the same dataset, and only unlabeled instances that satisfy the cross task knowledge constraints are used in the self training process 
MaCallum et al gec proposed generalized expectation(GE) criteria which formalised the knowledge as constraint terms about the expectation of the model into the objective function Graça et al pr proposed posterior regularization(PR) framework which projects the model's posterior onto a set of distributions that satisfy the auxiliary constraints  Druck et al ge-fl explored constraints of labeled features in the framework of GE by forcing the model's predicted feature distribution to approach the reference distribution  Andrzejewski et al andrzejewski2011framework proposed a framework in which general domain knowledge can be easily incorporated into LDA  Altendorf et al altendorf2012learning explored monotonicity constraints to improve the accuracy while learning from sparse data  Chen et al chen2013leveraging tried to learn comprehensible topic models by leveraging multi-domain knowledge 
Mann and McCallum simple,generalized incorporated not only labeled features but also other knowledge like class distribution into the objective function of GE-FL  But they discussed only from the semi-supervised perspective and did not investigate into the robustness problem, unlike what we addressed in this paper 
There are also some active learning methods trying to use prior knowledge  Raghavan et al feedback proposed to use feedback on instances and features interlacedly, and demonstrated that feedback on features boosts the model much  Druck et al active proposed an active learning method which solicits labels on features rather than on instances and then used GE-FL to train the model 
Conclusion and Discussions
This paper investigates into the problem of how to leverage prior knowledge robustly in learning models  We propose three regularization terms on top of generalized expectation criteria  As demonstrated by the experimental results, the performance can be considerably improved when taking into account these factors  Comparative results show that our proposed methods is more effective and works more robustly against baselines  To the best of our knowledge, this is the first work to address the robustness problem of leveraging knowledge, and may inspire other research 
We then present more detailed discussions about the three regularization methods  Incorporating neutral features is the simplest way of regularization, which doesn't require any modification of GE-FL but just finding out some common features  But as Figure 1 (a) shows, only using neutral features are not strong enough to handle extremely unbalanced labeled features 
The maximum entropy regularization term shows the strong ability of controlling unbalance 
This method doesn't need any extra knowledge, and is thus suitable when we know nothing about the corpus  But this method assumes that the categories are uniformly distributed, which may not be the case in practice, and it will have a degraded performance if the assumption is violated (see Figure 1 (b), Figure 2 (b), Figure 3 (a)) 
The KL divergence performs much better on unbalanced corpora than other methods  The reason is that KL divergence utilizes the reference class distribution and doesn't make any assumptions  The fact suggests that additional knowledge does benefit the model 
However, the KL divergence term requires providing the true class distribution  Sometimes, we may have the exact knowledge about the true distribution, but sometimes we may not  Fortunately, the model is insensitive to the true distribution and therefore a rough estimation of the true distribution is sufficient  In our experiments, when the true class distribution is 1:2, where the reference class distribution is set to 1:1 5/1:2/1:2 5, the accuracy is 0 755/0 756/0 760 respectively  This provides us the possibility to perform simple computing on the corpus to obtain the distribution in reality  Or, we can set the distribution roughly with domain expertise 

ground_truth: text classification for themes including sentiment, web-page, science, medical and healthcare

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. which lstm models did they compare with?
Context 3: Introduction
Recently, deep neural network has been widely employed in various recognition tasks  Increasing the depth of neural network is a effective way to improve the performance, and convolutional neural network (CNN) has benefited from it in visual recognition task BIBREF0   Deeper long short-term memory (LSTM) recurrent neural networks (RNNs) are also applied in large vocabulary continuous speech recognition (LVCSR) task, because LSTM networks have shown better performance than Fully-connected feed-forward deep neural network BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4  
Training neural network becomes more challenge when it goes deep  A conceptual tool called linear classifier probe is introduced to better understand the dynamics inside a neural network BIBREF5   The discriminating features of linear classifier is the hidden units of a intermediate layer  For deep neural networks, it is observed that deeper layer's accuracy is lower than that of shallower layers  Therefore, the tool shows the difficulty of deep neural model training visually 
Layer-wise pre-training is a successful method to train very deep neural networks BIBREF6   The convergence becomes harder with increasing the number of layers, even though the model is initialized with Xavier or its variants BIBREF7 , BIBREF8   But the deeper network which is initialized with a shallower trained network could converge well 
The size of LVCSR training dataset goes larger and training with only one GPU becomes high time consumption inevitably  Therefore, parallel training with multi-GPUs is more suitable for LVCSR system  Mini-batch based stochastic gradient descent (SGD) is the most popular method in neural network training procedure  Asynchronous SGD is a successful effort for parallel training based on it BIBREF9 , BIBREF10   It can many times speed up the training time without decreasing the accuracy  Besides, synchronous SGD is another effective effort, where the parameter server waits for every works to finish their computation and sent their local models to it, and then it sends updated model back to all workers BIBREF11   Synchronous SGD converges well in parallel training with data parallelism, and is also easy to implement 
In order to further improve the performance of deep neural network with parallel training, several methods are proposed  Model averaging method achieves linear speedup, as the final model is averaged from all parameters of local models in different workers BIBREF12 , BIBREF13 , but the accuracy decreases compared with single GPU training  Moreover, blockwise model-updating filter (BMUF) provides another almost linear speedup approach with multi-GPUs on the basis of model averaging  It can achieve improvement or no-degradation of recognition performance compared with mini-batch SGD on single GPU BIBREF14  
Moving averaged (MA) approaches are also proposed for parallel training  It is demonstrated that the moving average of the parameters obtained by SGD performs as well as the parameters that minimize the empirical cost, and moving average parameters can be used as the estimator of them, if the size of training data is large enough BIBREF15   One pass learning is then proposed, which is the combination of learning rate schedule and averaged SGD using moving average BIBREF16   Exponential moving average (EMA) is proposed as a non-interference method BIBREF17   EMA model is not broadcasted to workers to update their local models, and it is applied as the final model of entire training process  EMA method is utilized with model averaging and BMUF to further decrease the character error rate (CER)  It is also easy to implement in existing parallel training systems 
Frame stacking can also speed up the training time BIBREF18   The super frame is stacked by several regular frames, and it contains the information of them  Thus, the network can see multiple frames at a time, as the super frame is new input  Frame stacking can also lead to faster decoding 
For streaming voice search service, it needs to display intermediate recognition results while users are still speaking  As a result, the system needs to fulfill high real-time requirement, and we prefer unidirectional LSTM network rather than bidirectional one  High real-time requirement means low real time factor (RTF), but the RTF of deep LSTM model is higher inevitably  The dilemma of recognition accuracy and real-time requirement is an obstacle to the employment of deep LSTM network  Deep model outperforms because it contains more knowledge, but it is also cumbersome  As a result, the knowledge of deep model can be distilled to a shallow model BIBREF19   It provided a effective way to employ the deep model to the real-time system 
In this paper, we explore a entire deep LSTM RNN training framework, and employ it to real-time application  The deep learning systems benefit highly from a large quantity of labeled training data  Our first and basic speech recognition system is trained on 17000 hours of Shenma voice search dataset  It is a generic dataset sampled from diverse aspects of search queries  The requirement of speech recognition system also addressed by specific scenario, such as map and navigation task  The labeled dataset is too expensive, and training a new model with new large dataset from the beginning costs lots of time  Thus, it is natural to think of transferring the knowledge from basic model to new scenario's model  Transfer learning expends less data and less training time than full training  In this paper, we also introduce a novel transfer learning strategy with segmental Minimum Bayes-Risk (sMBR)  As a result, transfer training with only 1000 hours data can match equivalent performance for full training with 7300 hours data 
Our deep LSTM training framework for LVCSR is presented in Section 2  Section 3 describes how the very deep models does apply in real world applications, and how to transfer the model to another task  The framework is analyzed and discussed in Section 4, and followed by the conclusion in Section 5 
Layer-wise Training with Soft Target and Hard Target
Gradient-based optimization of deep LSTM network with random initialization get stuck in poor solution easily  Xavier initialization can partially solve this problem BIBREF7 , so this method is the regular initialization method of all training procedure  However, it does not work well when it is utilized to initialize very deep model directly, because of vanishing or exploding gradients  Instead, layer-wise pre-training method is a effective way to train the weights of very deep architecture BIBREF6 , BIBREF20   In layer-wise pre-training procedure, a one-layer LSTM model is firstly trained with normalized initialization  Sequentially, two-layers LSTM model's first layer is initialized by trained one-layer model, and its second layer is regularly initialized  In this way, a deep architecture is layer-by-layer trained, and it can converge well 
In conventional layer-wise pre-training, only parameters of shallower network are transfered to deeper one, and the learning targets are still the alignments generated by HMM-GMM system  The targets are vectors that only one state's probability is one, and the others' are zeros  They are known as hard targets, and they carry limited knowledge as only one state is active  In contrast, the knowledge of shallower network should be also transfered to deeper one  It is obtained by the softmax layer of existing model typically, so each state has a probability rather than only zero or one, and called as soft target  As a result, the deeper network which is student network learns the parameters and knowledge from shallower one which is called teacher network  When training the student network from the teacher network, the final alignment is the combination of hard target and soft target in our layer-wise training phase  The final alignment provides various knowledge which transfered from teacher network and extracted from true labels  If only soft target is learned, student network perform no better than teacher network, but it could outperform teacher network as it also learns true labels 
The deeper network spends less time to getting the same level of original network than the network trained from the beginning, as a period of low performance is skipped  Therefore, training with hard and soft target is a time saving method  For large training dataset, training with the whole dataset still spends too much time  A network firstly trained with only a small part of dataset could go deeper as well, and so the training time reducing rapidly  When the network is deep enough, it then trained on the entire dataset to get further improvement  There is no gap of accuracy between these two approaches, but latter one saves much time 
Differential Saturation Check
The objects of conventional saturation check are gradients and the cell activations BIBREF4   Gradients are clipped to range [-5, 5], while the cell activations clipped to range [-50, 50]  Apart from them, the differentials of recurrent layers is also limited  If the differentials go beyond the range, corresponding back propagation is skipped, while if the gradients and cell activations go beyond the bound, values are set as the boundary values  The differentials which are too large or too small will lead to the gradients easily vanishing, and it demonstrates the failure of this propagation  As a result, the parameters are not updated, and next propagation  
Sequence Discriminative Training
Cross-entropy (CE) is widely used in speech recognition training system as a frame-wise discriminative training criterion  However, it is not well suited to speech recognition, because speech recognition training is a sequential learning problem  In contrast, sequence discriminative training criterion has shown to further improve performance of neural network first trained with cross-entropy BIBREF21 , BIBREF22 , BIBREF23   We choose state-level minimum bayes risk (sMBR) BIBREF21 among a number of sequence discriminative criterion is proposed, such as maximum mutual information (MMI) BIBREF24 and minimum phone error (MPE) BIBREF25   MPE and sMBR are designed to minimize the expected error of different granularity of labels, while CE aims to minimizes expected frame error, and MMI aims to minimizes expected sentence error  State-level information is focused on by sMBR 
a frame-level accurate model is firstly trained by CE loss function, and then sMBR loss function is utilized for further training to get sequence-level accuracy  Only a part of training dataset is needed in sMBR training phase on the basis of whole dataset CE training 
Parallel Training
It is demonstrated that training with larger dataset can improve recognition accuracy  However, larger dataset means more training samples and more model parameters  Therefore, parallel training with multiple GPUs is essential, and it makes use of data parallelism BIBREF9   The entire training data is partitioned into several split without overlapping and they are distributed to different GPUs  Each GPU trains with one split of training dataset locally  All GPUs synchronize their local models with model average method after a mini-batch optimization BIBREF12 , BIBREF13  
Model average method achieves linear speedup in training phase, but the recognition accuracy decreases compared with single GPU training  Block-wise model updating filter (BMUF) is another successful effort in parallel training with linear speedup as well  It can achieve no-degradation of recognition accuracy with multi-GPUs BIBREF14   In the model average method, aggregated model INLINEFORM0 is computed and broadcasted to GPUs  On the basis of it, BMUF proposed a novel model updating strategy: INLINEFORM1 INLINEFORM2
Where INLINEFORM0 denotes model update, and INLINEFORM1 is the global-model update  There are two parameters in BMUF, block momentum INLINEFORM2 , and block learning rate INLINEFORM3   Then, the global model is updated as INLINEFORM4
Consequently, INLINEFORM0 is broadcasted to all GPUs to initial their local models, instead of INLINEFORM1 in model average method 
Averaged SGD is proposed to further accelerate the convergence speed of SGD  Averaged SGD leverages the moving average (MA) INLINEFORM0 as the estimator of INLINEFORM1 BIBREF15 : INLINEFORM2
Where INLINEFORM0 is computed by model averaging or BMUF  It is shown that INLINEFORM1 can well converge to INLINEFORM2 , with the large enough training dataset in single GPU training  It can be considered as a non-interference strategy that INLINEFORM3 does not participate the main optimization process, and only takes effect after the end of entire optimization  However, for the parallel training implementation, each INLINEFORM4 is computed by model averaging and BMUF with multiple models, and moving average model INLINEFORM5 does not well converge, compared with single GPU training 
Model averaging based methods are employed in parallel training of large scale dataset, because of their faster convergence, and especially no-degradation implementation of BMUF  But combination of model averaged based methods and moving average does not match the expectation of further enhance performance and it is presented as INLINEFORM0
The weight of each INLINEFORM0 is equal in moving average method regardless the effect of temporal order  But INLINEFORM1 closer to the end of training achieve higher accuracy in the model averaging based approach, and thus it should be with more proportion in final INLINEFORM2   As a result, exponential moving average(EMA) is appropriate, which the weight for each older parameters decrease exponentially, and never reaching zero  After moving average based methods, the EMA parameters are updated recursively as INLINEFORM3
Here INLINEFORM0 represents the degree of weight decrease, and called exponential updating rate  EMA is also a non-interference training strategy that is implemented easily, as the updated model is not broadcasted  Therefore, there is no need to add extra learning rate updating approach, as it can be appended to existing training procedure directly 
Deployment
There is a high real time requirement in real world application, especially in online voice search system  Shenma voice search is one of the most popular mobile search engines in China, and it is a streaming service that intermediate recognition results displayed while users are still speaking  Unidirectional LSTM network is applied, rather than bidirectional one, because it is well suited to real-time streaming speech recognition 
Distillation
It is demonstrated that deep neural network architecture can achieve improvement in LVCSR  However, it also leads to much more computation and higher RTF, so that the recognition result can not be displayed in real time  It should be noted that deeper neural network contains more knowledge, but it is also cumbersome  the knowledge is key to improve the performance  If it can be transfered from cumbersome model to a small model, the recognition ability can also be transfered to the small model  Knowledge transferring to small model is called distillation BIBREF19   The small model can perform as well as cumbersome model, after distilling  It provide a way to utilize high-performance but high RTF model in real time system  The class probability produced by the cumbersome model is regarded as soft target, and the generalization ability of cumbersome model is transfered to small model with it  Distillation is model's knowledge transferring approach, so there is no need to use the hard target, which is different with the layer-wise training method 
9-layers unidirectional LSTM model achieves outstanding performance, but meanwhile it is too computationally expensive to allow deployment in real time recognition system  In order to ensure real-time of the system, the number of layers needs to be reduced  The shallower network can learn the knowledge of deeper network with distillation  It is found that RTF of 2-layers network is acceptable, so the knowledge is distilled from 9-layers well-trained model to 2-layers model  Table TABREF16 shows that distillation from 9-layers to 2-layers brings RTF decrease of relative 53%, while CER only increases 5%  The knowledge of deep network is almost transfered with distillation, Distillation brings promising RTF reduction, but only little knowledge of deep network is lost  Moreover, CER of 2-layers distilled LSTM decreases relative 14%, compared with 2-layers regular-trained LSTM 
Transfer Learning with sMBR
For a certain specific scenario, the model trained with the data recorded from it has better adaptation than the model trained with generic scenario  But it spends too much time training a model from the beginning, if there is a well-trained model for generic scenarios  Moreover, labeling a large quantity of training data in new scenario is both costly and time consuming  If a model transfer trained with smaller dataset can obtained the similar recognition accuracy compared with the model directly trained with larger dataset, it is no doubt that transfer learning is more practical  Since specific scenario is a subset of generic scenario, some knowledge can be shared between them  Besides, generic scenario consists of various conditions, so its model has greater robustness  As a result, not only shared knowledge but also robustness can be transfered from the model of generic scenario to the model of specific one 
As the model well trained from generic scenario achieves good performance in frame level classification, sequence discriminative training is required to adapt new model to specific scenario additionally  Moreover, it does not need alignment from HMM-GMM system, and it also saves amount of time to prepare alignment 
Training Data
A large quantity of labeled data is needed for training a more accurate acoustic model  We collect the 17000 hours labeled data from Shenma voice search, which is one of the most popular mobile search engines in China  The dataset is created from anonymous online users' search queries in Mandarin, and all audio file's sampling rate is 16kHz, recorded by mobile phones  This dataset consists of many different conditions, such as diverse noise even low signal-to-noise, babble, dialects, accents, hesitation and so on 
In the Amap, which is one of the most popular web mapping and navigation services in China, users can search locations and navigate to locations they want though voice search  To present the performance of transfer learning with sequence discriminative training, the model trained from Shenma voice search which is greneric scenario transfer its knowledge to the model of Amap voice search  7300 hours labeled data is collected in the similar way of Shenma voice search data collection 
Two dataset is divided into training set, validation set and test set separately, and the quantity of them is shown in Table TABREF10   The three sets are split according to speakers, in order to avoid utterances of same speaker appearing in three sets simultaneously  The test sets of Shenma and Amap voice search are called Shenma Test and Amap Test 
Experimental setup
LSTM RNNs outperform conventional RNNs for speech recognition system, especially deep LSTM RNNs, because of its long-range dependencies more accurately for temporal sequence conditions BIBREF26 , BIBREF23   Shenma and Amap voice search is a streaming service that intermediate recognition results displayed while users are still speaking  So as for online recognition in real time, we prefer unidirectional LSTM model rather than bidirectional one  Thus, the training system is unidirectional LSTM-based 
A 26-dimensional filter bank and 2-dimensional pitch feature is extracted for each frame, and is concatenated with first and second order difference as the final input of the network  The super frame are stacked by 3 frames without overlapping  The architecture we trained consists of two LSTM layers with sigmoid activation function, followed by a full-connection layer  The out layer is a softmax layer with 11088 hidden markov model (HMM) tied-states as output classes, the loss function is cross-entropy (CE)  The performance metric of the system in Mandarin is reported with character error rate (CER)  The alignment of frame-level ground truth is obtained by GMM-HMM system  Mini-batched SGD is utilized with momentum trick and the network is trained for a total of 4 epochs  The block learning rate and block momentum of BMUF are set as 1 and 0 9  5-gram language model is leveraged in decoder, and the vocabulary size is as large as 760000  Differentials of recurrent layers is limited to range [-10000,10000], while gradients are clipped to range [-5, 5] and cell activations clipped to range [-50, 50]  After training with CE loss, sMBR loss is employed to further improve the performance 
It has shown that BMUF outperforms traditional model averaging method, and it is utilized at the synchronization phase  After synchronizing with BMUF, EMA method further updates the model in non-interference way  The training system is deployed on the MPI-based HPC cluster where 8 GPUs  Each GPU processes non-overlap subset split from the entire large scale dataset in parallel 
Local models from distributed workers synchronize with each other in decentralized way  In the traditional model averaging and BMUF method, a parameter server waits for all workers to send their local models, aggregate them, and send the updated model to all workers  Computing resource of workers is wasted until aggregation of the parameter server done  Decentralized method makes full use of computing resource, and we employ the MPI-based Mesh AllReduce method  It is mesh topology as shown in Figure FIGREF12   There is no centralized parameter server, and peer to peer communication is used to transmit local models between workers  Local model INLINEFORM0 of INLINEFORM1 -th worker in INLINEFORM2 workers cluster is split to INLINEFORM3 pieces INLINEFORM4 , and send to corresponding worker  In the aggregation phase, INLINEFORM5 -th worker computed INLINEFORM6 splits of model INLINEFORM7 and send updated model INLINEFORM8 back to workers  As a result, all workers participate in aggregation and no computing resource is dissipated  It is significant to promote training efficiency, when the size of neural network model is too large  The EMA model is also updated additionally, but not broadcasting it 
Results
In order to evaluate our system, several sets of experiments are performed  The Shenma test set including about 9000 samples and Amap test set including about 7000 samples contain various real world conditions  It simulates the majority of user scenarios, and can well evaluates the performance of a trained model  Firstly, we show the results of models trained with EMA method  Secondly, for real world applications, very deep LSTM is distilled to a shallow one, so as for lower RTF  The model of Amap is also needed to train for map and navigation scenarios  The performance of transfer learning from Shenma voice search to Amap voice search is also presented 
Layer-wise Training
In layer-wise training, the deeper model learns both parameters and knowledge from the shallower model  The deeper model is initialized by the shallower one, and its alignment is the combination of hard target and soft target of shallower one  Two targets have the same weights in our framework  The teacher model is trained with CE  For each layer-wise trained CE model, corresponding sMBR model is also trained, as sMBR could achieve additional improvement  In our framework, 1000 hours data is randomly selected from the total dataset for sMBR training  There is no obvious performance enhancement when the size of sMBR training dataset increases 
For very deep unidirectional LSTM initialized with Xavier initialization algorithm, 6-layers model converges well, but there is no further improvement with increasing the number of layers  Therefore, the first 6 layers of 7-layers model is initialized by 6-layers model, and soft target is provided by 6-layers model  Consequently, deeper LSTM is also trained in the same way  It should be noticed that the teacher model of 9-layers model is the 8-layers model trained by sMBR, while the other teacher model is CE model  As shown in Table TABREF15 , the layer-wise trained models always performs better than the models with Xavier initialization, as the model is deep  Therefore, for the last layer training, we choose 8-layers sMBR model as the teacher model instead of CE model  A comparison between 6-layers and 9-layers sMBR models shows that 3 additional layers of layer-wise training brings relative 12 6% decreasing of CER  It is also significant that the averaged CER of sMBR models with different layers decreases absolute 0 73% approximately compared with CE models, so the improvement of sequence discriminative learning is promising 
Transfer Learning
2-layers distilled model of Shenma voice search has shown a impressive performance on Shenma Test, and we call it Shenma model  It is trained for generic search scenario, but it has less adaptation for specific scenario like Amap voice search  Training with very large dataset using CE loss is regarded as improvement of frame level recognition accuracy, and sMBR with less dataset further improves accuracy as sequence discriminative training  If robust model of generic scenario is trained, there is no need to train a model with very large dataset, and sequence discriminative training with less dataset is enough  Therefore, on the basis of Shenma model, it is sufficient to train a new Amap model with small dataset using sMBR  As shown in Table TABREF18 , Shenma model presents the worst performance among three methods, since it does not trained for Amap scenario  2-layers Shenma model further trained with sMBR achieves about 8 1% relative reduction, compared with 2-layers regular-trained Amap model  Both training sMBR datasets contain the same 1000 hours data  As a result, with the Shenma model, only about 14% data usage achieves lower CER, and it leads to great time and cost saving with less labeled data  Besides, transfer learning with sMBR does not use the alignment from the HMM-GMM system, so it also saves huge amount of time 
Conclusion
We have presented a whole deep unidirectional LSTM parallel training system for LVCSR  The recognition performance improves when the network goes deep  Distillation makes it possible that deep LSTM model transfer its knowledge to shallow model with little loss  The model could be distilled to 2-layers model with very low RTF, so that it can display the immediate recognition results  As a result, its CER decrease relatively 14%, compared with the 2-layers regular trained model  In addition, transfer learning with sMBR is also proposed  If a great model has well trained from generic scenario, only 14% of the size of training dataset is needed to train a more accuracy acoustic model for specific scenario  Our future work includes 1) finding more effective methods to reduce the CER by increasing the number of layers; 2) applying this training framework to Connectionist Temporal Classification (CTC) and attention-based neural networks 

ground_truth: Unidirectional LSTM networks with 2, 6, 7, 8, and  9 layers.