prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. which lstm models did they compare with?
Context 1: Introduction
Recently, deep neural network has been widely employed in various recognition tasks  Increasing the depth of neural network is a effective way to improve the performance, and convolutional neural network (CNN) has benefited from it in visual recognition task BIBREF0   Deeper long short-term memory (LSTM) recurrent neural networks (RNNs) are also applied in large vocabulary continuous speech recognition (LVCSR) task, because LSTM networks have shown better performance than Fully-connected feed-forward deep neural network BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4  
Training neural network becomes more challenge when it goes deep  A conceptual tool called linear classifier probe is introduced to better understand the dynamics inside a neural network BIBREF5   The discriminating features of linear classifier is the hidden units of a intermediate layer  For deep neural networks, it is observed that deeper layer's accuracy is lower than that of shallower layers  Therefore, the tool shows the difficulty of deep neural model training visually 
Layer-wise pre-training is a successful method to train very deep neural networks BIBREF6   The convergence becomes harder with increasing the number of layers, even though the model is initialized with Xavier or its variants BIBREF7 , BIBREF8   But the deeper network which is initialized with a shallower trained network could converge well 
The size of LVCSR training dataset goes larger and training with only one GPU becomes high time consumption inevitably  Therefore, parallel training with multi-GPUs is more suitable for LVCSR system  Mini-batch based stochastic gradient descent (SGD) is the most popular method in neural network training procedure  Asynchronous SGD is a successful effort for parallel training based on it BIBREF9 , BIBREF10   It can many times speed up the training time without decreasing the accuracy  Besides, synchronous SGD is another effective effort, where the parameter server waits for every works to finish their computation and sent their local models to it, and then it sends updated model back to all workers BIBREF11   Synchronous SGD converges well in parallel training with data parallelism, and is also easy to implement 
In order to further improve the performance of deep neural network with parallel training, several methods are proposed  Model averaging method achieves linear speedup, as the final model is averaged from all parameters of local models in different workers BIBREF12 , BIBREF13 , but the accuracy decreases compared with single GPU training  Moreover, blockwise model-updating filter (BMUF) provides another almost linear speedup approach with multi-GPUs on the basis of model averaging  It can achieve improvement or no-degradation of recognition performance compared with mini-batch SGD on single GPU BIBREF14  
Moving averaged (MA) approaches are also proposed for parallel training  It is demonstrated that the moving average of the parameters obtained by SGD performs as well as the parameters that minimize the empirical cost, and moving average parameters can be used as the estimator of them, if the size of training data is large enough BIBREF15   One pass learning is then proposed, which is the combination of learning rate schedule and averaged SGD using moving average BIBREF16   Exponential moving average (EMA) is proposed as a non-interference method BIBREF17   EMA model is not broadcasted to workers to update their local models, and it is applied as the final model of entire training process  EMA method is utilized with model averaging and BMUF to further decrease the character error rate (CER)  It is also easy to implement in existing parallel training systems 
Frame stacking can also speed up the training time BIBREF18   The super frame is stacked by several regular frames, and it contains the information of them  Thus, the network can see multiple frames at a time, as the super frame is new input  Frame stacking can also lead to faster decoding 
For streaming voice search service, it needs to display intermediate recognition results while users are still speaking  As a result, the system needs to fulfill high real-time requirement, and we prefer unidirectional LSTM network rather than bidirectional one  High real-time requirement means low real time factor (RTF), but the RTF of deep LSTM model is higher inevitably  The dilemma of recognition accuracy and real-time requirement is an obstacle to the employment of deep LSTM network  Deep model outperforms because it contains more knowledge, but it is also cumbersome  As a result, the knowledge of deep model can be distilled to a shallow model BIBREF19   It provided a effective way to employ the deep model to the real-time system 
In this paper, we explore a entire deep LSTM RNN training framework, and employ it to real-time application  The deep learning systems benefit highly from a large quantity of labeled training data  Our first and basic speech recognition system is trained on 17000 hours of Shenma voice search dataset  It is a generic dataset sampled from diverse aspects of search queries  The requirement of speech recognition system also addressed by specific scenario, such as map and navigation task  The labeled dataset is too expensive, and training a new model with new large dataset from the beginning costs lots of time  Thus, it is natural to think of transferring the knowledge from basic model to new scenario's model  Transfer learning expends less data and less training time than full training  In this paper, we also introduce a novel transfer learning strategy with segmental Minimum Bayes-Risk (sMBR)  As a result, transfer training with only 1000 hours data can match equivalent performance for full training with 7300 hours data 
Our deep LSTM training framework for LVCSR is presented in Section 2  Section 3 describes how the very deep models does apply in real world applications, and how to transfer the model to another task  The framework is analyzed and discussed in Section 4, and followed by the conclusion in Section 5 
Layer-wise Training with Soft Target and Hard Target
Gradient-based optimization of deep LSTM network with random initialization get stuck in poor solution easily  Xavier initialization can partially solve this problem BIBREF7 , so this method is the regular initialization method of all training procedure  However, it does not work well when it is utilized to initialize very deep model directly, because of vanishing or exploding gradients  Instead, layer-wise pre-training method is a effective way to train the weights of very deep architecture BIBREF6 , BIBREF20   In layer-wise pre-training procedure, a one-layer LSTM model is firstly trained with normalized initialization  Sequentially, two-layers LSTM model's first layer is initialized by trained one-layer model, and its second layer is regularly initialized  In this way, a deep architecture is layer-by-layer trained, and it can converge well 
In conventional layer-wise pre-training, only parameters of shallower network are transfered to deeper one, and the learning targets are still the alignments generated by HMM-GMM system  The targets are vectors that only one state's probability is one, and the others' are zeros  They are known as hard targets, and they carry limited knowledge as only one state is active  In contrast, the knowledge of shallower network should be also transfered to deeper one  It is obtained by the softmax layer of existing model typically, so each state has a probability rather than only zero or one, and called as soft target  As a result, the deeper network which is student network learns the parameters and knowledge from shallower one which is called teacher network  When training the student network from the teacher network, the final alignment is the combination of hard target and soft target in our layer-wise training phase  The final alignment provides various knowledge which transfered from teacher network and extracted from true labels  If only soft target is learned, student network perform no better than teacher network, but it could outperform teacher network as it also learns true labels 
The deeper network spends less time to getting the same level of original network than the network trained from the beginning, as a period of low performance is skipped  Therefore, training with hard and soft target is a time saving method  For large training dataset, training with the whole dataset still spends too much time  A network firstly trained with only a small part of dataset could go deeper as well, and so the training time reducing rapidly  When the network is deep enough, it then trained on the entire dataset to get further improvement  There is no gap of accuracy between these two approaches, but latter one saves much time 
Differential Saturation Check
The objects of conventional saturation check are gradients and the cell activations BIBREF4   Gradients are clipped to range [-5, 5], while the cell activations clipped to range [-50, 50]  Apart from them, the differentials of recurrent layers is also limited  If the differentials go beyond the range, corresponding back propagation is skipped, while if the gradients and cell activations go beyond the bound, values are set as the boundary values  The differentials which are too large or too small will lead to the gradients easily vanishing, and it demonstrates the failure of this propagation  As a result, the parameters are not updated, and next propagation  
Sequence Discriminative Training
Cross-entropy (CE) is widely used in speech recognition training system as a frame-wise discriminative training criterion  However, it is not well suited to speech recognition, because speech recognition training is a sequential learning problem  In contrast, sequence discriminative training criterion has shown to further improve performance of neural network first trained with cross-entropy BIBREF21 , BIBREF22 , BIBREF23   We choose state-level minimum bayes risk (sMBR) BIBREF21 among a number of sequence discriminative criterion is proposed, such as maximum mutual information (MMI) BIBREF24 and minimum phone error (MPE) BIBREF25   MPE and sMBR are designed to minimize the expected error of different granularity of labels, while CE aims to minimizes expected frame error, and MMI aims to minimizes expected sentence error  State-level information is focused on by sMBR 
a frame-level accurate model is firstly trained by CE loss function, and then sMBR loss function is utilized for further training to get sequence-level accuracy  Only a part of training dataset is needed in sMBR training phase on the basis of whole dataset CE training 
Parallel Training
It is demonstrated that training with larger dataset can improve recognition accuracy  However, larger dataset means more training samples and more model parameters  Therefore, parallel training with multiple GPUs is essential, and it makes use of data parallelism BIBREF9   The entire training data is partitioned into several split without overlapping and they are distributed to different GPUs  Each GPU trains with one split of training dataset locally  All GPUs synchronize their local models with model average method after a mini-batch optimization BIBREF12 , BIBREF13  
Model average method achieves linear speedup in training phase, but the recognition accuracy decreases compared with single GPU training  Block-wise model updating filter (BMUF) is another successful effort in parallel training with linear speedup as well  It can achieve no-degradation of recognition accuracy with multi-GPUs BIBREF14   In the model average method, aggregated model INLINEFORM0 is computed and broadcasted to GPUs  On the basis of it, BMUF proposed a novel model updating strategy: INLINEFORM1 INLINEFORM2
Where INLINEFORM0 denotes model update, and INLINEFORM1 is the global-model update  There are two parameters in BMUF, block momentum INLINEFORM2 , and block learning rate INLINEFORM3   Then, the global model is updated as INLINEFORM4
Consequently, INLINEFORM0 is broadcasted to all GPUs to initial their local models, instead of INLINEFORM1 in model average method 
Averaged SGD is proposed to further accelerate the convergence speed of SGD  Averaged SGD leverages the moving average (MA) INLINEFORM0 as the estimator of INLINEFORM1 BIBREF15 : INLINEFORM2
Where INLINEFORM0 is computed by model averaging or BMUF  It is shown that INLINEFORM1 can well converge to INLINEFORM2 , with the large enough training dataset in single GPU training  It can be considered as a non-interference strategy that INLINEFORM3 does not participate the main optimization process, and only takes effect after the end of entire optimization  However, for the parallel training implementation, each INLINEFORM4 is computed by model averaging and BMUF with multiple models, and moving average model INLINEFORM5 does not well converge, compared with single GPU training 
Model averaging based methods are employed in parallel training of large scale dataset, because of their faster convergence, and especially no-degradation implementation of BMUF  But combination of model averaged based methods and moving average does not match the expectation of further enhance performance and it is presented as INLINEFORM0
The weight of each INLINEFORM0 is equal in moving average method regardless the effect of temporal order  But INLINEFORM1 closer to the end of training achieve higher accuracy in the model averaging based approach, and thus it should be with more proportion in final INLINEFORM2   As a result, exponential moving average(EMA) is appropriate, which the weight for each older parameters decrease exponentially, and never reaching zero  After moving average based methods, the EMA parameters are updated recursively as INLINEFORM3
Here INLINEFORM0 represents the degree of weight decrease, and called exponential updating rate  EMA is also a non-interference training strategy that is implemented easily, as the updated model is not broadcasted  Therefore, there is no need to add extra learning rate updating approach, as it can be appended to existing training procedure directly 
Deployment
There is a high real time requirement in real world application, especially in online voice search system  Shenma voice search is one of the most popular mobile search engines in China, and it is a streaming service that intermediate recognition results displayed while users are still speaking  Unidirectional LSTM network is applied, rather than bidirectional one, because it is well suited to real-time streaming speech recognition 
Distillation
It is demonstrated that deep neural network architecture can achieve improvement in LVCSR  However, it also leads to much more computation and higher RTF, so that the recognition result can not be displayed in real time  It should be noted that deeper neural network contains more knowledge, but it is also cumbersome  the knowledge is key to improve the performance  If it can be transfered from cumbersome model to a small model, the recognition ability can also be transfered to the small model  Knowledge transferring to small model is called distillation BIBREF19   The small model can perform as well as cumbersome model, after distilling  It provide a way to utilize high-performance but high RTF model in real time system  The class probability produced by the cumbersome model is regarded as soft target, and the generalization ability of cumbersome model is transfered to small model with it  Distillation is model's knowledge transferring approach, so there is no need to use the hard target, which is different with the layer-wise training method 
9-layers unidirectional LSTM model achieves outstanding performance, but meanwhile it is too computationally expensive to allow deployment in real time recognition system  In order to ensure real-time of the system, the number of layers needs to be reduced  The shallower network can learn the knowledge of deeper network with distillation  It is found that RTF of 2-layers network is acceptable, so the knowledge is distilled from 9-layers well-trained model to 2-layers model  Table TABREF16 shows that distillation from 9-layers to 2-layers brings RTF decrease of relative 53%, while CER only increases 5%  The knowledge of deep network is almost transfered with distillation, Distillation brings promising RTF reduction, but only little knowledge of deep network is lost  Moreover, CER of 2-layers distilled LSTM decreases relative 14%, compared with 2-layers regular-trained LSTM 
Transfer Learning with sMBR
For a certain specific scenario, the model trained with the data recorded from it has better adaptation than the model trained with generic scenario  But it spends too much time training a model from the beginning, if there is a well-trained model for generic scenarios  Moreover, labeling a large quantity of training data in new scenario is both costly and time consuming  If a model transfer trained with smaller dataset can obtained the similar recognition accuracy compared with the model directly trained with larger dataset, it is no doubt that transfer learning is more practical  Since specific scenario is a subset of generic scenario, some knowledge can be shared between them  Besides, generic scenario consists of various conditions, so its model has greater robustness  As a result, not only shared knowledge but also robustness can be transfered from the model of generic scenario to the model of specific one 
As the model well trained from generic scenario achieves good performance in frame level classification, sequence discriminative training is required to adapt new model to specific scenario additionally  Moreover, it does not need alignment from HMM-GMM system, and it also saves amount of time to prepare alignment 
Training Data
A large quantity of labeled data is needed for training a more accurate acoustic model  We collect the 17000 hours labeled data from Shenma voice search, which is one of the most popular mobile search engines in China  The dataset is created from anonymous online users' search queries in Mandarin, and all audio file's sampling rate is 16kHz, recorded by mobile phones  This dataset consists of many different conditions, such as diverse noise even low signal-to-noise, babble, dialects, accents, hesitation and so on 
In the Amap, which is one of the most popular web mapping and navigation services in China, users can search locations and navigate to locations they want though voice search  To present the performance of transfer learning with sequence discriminative training, the model trained from Shenma voice search which is greneric scenario transfer its knowledge to the model of Amap voice search  7300 hours labeled data is collected in the similar way of Shenma voice search data collection 
Two dataset is divided into training set, validation set and test set separately, and the quantity of them is shown in Table TABREF10   The three sets are split according to speakers, in order to avoid utterances of same speaker appearing in three sets simultaneously  The test sets of Shenma and Amap voice search are called Shenma Test and Amap Test 
Experimental setup
LSTM RNNs outperform conventional RNNs for speech recognition system, especially deep LSTM RNNs, because of its long-range dependencies more accurately for temporal sequence conditions BIBREF26 , BIBREF23   Shenma and Amap voice search is a streaming service that intermediate recognition results displayed while users are still speaking  So as for online recognition in real time, we prefer unidirectional LSTM model rather than bidirectional one  Thus, the training system is unidirectional LSTM-based 
A 26-dimensional filter bank and 2-dimensional pitch feature is extracted for each frame, and is concatenated with first and second order difference as the final input of the network  The super frame are stacked by 3 frames without overlapping  The architecture we trained consists of two LSTM layers with sigmoid activation function, followed by a full-connection layer  The out layer is a softmax layer with 11088 hidden markov model (HMM) tied-states as output classes, the loss function is cross-entropy (CE)  The performance metric of the system in Mandarin is reported with character error rate (CER)  The alignment of frame-level ground truth is obtained by GMM-HMM system  Mini-batched SGD is utilized with momentum trick and the network is trained for a total of 4 epochs  The block learning rate and block momentum of BMUF are set as 1 and 0 9  5-gram language model is leveraged in decoder, and the vocabulary size is as large as 760000  Differentials of recurrent layers is limited to range [-10000,10000], while gradients are clipped to range [-5, 5] and cell activations clipped to range [-50, 50]  After training with CE loss, sMBR loss is employed to further improve the performance 
It has shown that BMUF outperforms traditional model averaging method, and it is utilized at the synchronization phase  After synchronizing with BMUF, EMA method further updates the model in non-interference way  The training system is deployed on the MPI-based HPC cluster where 8 GPUs  Each GPU processes non-overlap subset split from the entire large scale dataset in parallel 
Local models from distributed workers synchronize with each other in decentralized way  In the traditional model averaging and BMUF method, a parameter server waits for all workers to send their local models, aggregate them, and send the updated model to all workers  Computing resource of workers is wasted until aggregation of the parameter server done  Decentralized method makes full use of computing resource, and we employ the MPI-based Mesh AllReduce method  It is mesh topology as shown in Figure FIGREF12   There is no centralized parameter server, and peer to peer communication is used to transmit local models between workers  Local model INLINEFORM0 of INLINEFORM1 -th worker in INLINEFORM2 workers cluster is split to INLINEFORM3 pieces INLINEFORM4 , and send to corresponding worker  In the aggregation phase, INLINEFORM5 -th worker computed INLINEFORM6 splits of model INLINEFORM7 and send updated model INLINEFORM8 back to workers  As a result, all workers participate in aggregation and no computing resource is dissipated  It is significant to promote training efficiency, when the size of neural network model is too large  The EMA model is also updated additionally, but not broadcasting it 
Results
In order to evaluate our system, several sets of experiments are performed  The Shenma test set including about 9000 samples and Amap test set including about 7000 samples contain various real world conditions  It simulates the majority of user scenarios, and can well evaluates the performance of a trained model  Firstly, we show the results of models trained with EMA method  Secondly, for real world applications, very deep LSTM is distilled to a shallow one, so as for lower RTF  The model of Amap is also needed to train for map and navigation scenarios  The performance of transfer learning from Shenma voice search to Amap voice search is also presented 
Layer-wise Training
In layer-wise training, the deeper model learns both parameters and knowledge from the shallower model  The deeper model is initialized by the shallower one, and its alignment is the combination of hard target and soft target of shallower one  Two targets have the same weights in our framework  The teacher model is trained with CE  For each layer-wise trained CE model, corresponding sMBR model is also trained, as sMBR could achieve additional improvement  In our framework, 1000 hours data is randomly selected from the total dataset for sMBR training  There is no obvious performance enhancement when the size of sMBR training dataset increases 
For very deep unidirectional LSTM initialized with Xavier initialization algorithm, 6-layers model converges well, but there is no further improvement with increasing the number of layers  Therefore, the first 6 layers of 7-layers model is initialized by 6-layers model, and soft target is provided by 6-layers model  Consequently, deeper LSTM is also trained in the same way  It should be noticed that the teacher model of 9-layers model is the 8-layers model trained by sMBR, while the other teacher model is CE model  As shown in Table TABREF15 , the layer-wise trained models always performs better than the models with Xavier initialization, as the model is deep  Therefore, for the last layer training, we choose 8-layers sMBR model as the teacher model instead of CE model  A comparison between 6-layers and 9-layers sMBR models shows that 3 additional layers of layer-wise training brings relative 12 6% decreasing of CER  It is also significant that the averaged CER of sMBR models with different layers decreases absolute 0 73% approximately compared with CE models, so the improvement of sequence discriminative learning is promising 
Transfer Learning
2-layers distilled model of Shenma voice search has shown a impressive performance on Shenma Test, and we call it Shenma model  It is trained for generic search scenario, but it has less adaptation for specific scenario like Amap voice search  Training with very large dataset using CE loss is regarded as improvement of frame level recognition accuracy, and sMBR with less dataset further improves accuracy as sequence discriminative training  If robust model of generic scenario is trained, there is no need to train a model with very large dataset, and sequence discriminative training with less dataset is enough  Therefore, on the basis of Shenma model, it is sufficient to train a new Amap model with small dataset using sMBR  As shown in Table TABREF18 , Shenma model presents the worst performance among three methods, since it does not trained for Amap scenario  2-layers Shenma model further trained with sMBR achieves about 8 1% relative reduction, compared with 2-layers regular-trained Amap model  Both training sMBR datasets contain the same 1000 hours data  As a result, with the Shenma model, only about 14% data usage achieves lower CER, and it leads to great time and cost saving with less labeled data  Besides, transfer learning with sMBR does not use the alignment from the HMM-GMM system, so it also saves huge amount of time 
Conclusion
We have presented a whole deep unidirectional LSTM parallel training system for LVCSR  The recognition performance improves when the network goes deep  Distillation makes it possible that deep LSTM model transfer its knowledge to shallow model with little loss  The model could be distilled to 2-layers model with very low RTF, so that it can display the immediate recognition results  As a result, its CER decrease relatively 14%, compared with the 2-layers regular trained model  In addition, transfer learning with sMBR is also proposed  If a great model has well trained from generic scenario, only 14% of the size of training dataset is needed to train a more accuracy acoustic model for specific scenario  Our future work includes 1) finding more effective methods to reduce the CER by increasing the number of layers; 2) applying this training framework to Connectionist Temporal Classification (CTC) and attention-based neural networks 
Context 2: Introduction
The cognitive processes involved in human language comprehension are complex and only partially identified  According to the dual-stream model of speech comprehension BIBREF1 , sound waves are first converted to phoneme-like features and further processed by a ventral stream that maps those features onto words and semantic structures, and a dorsal stream that (among other things) supports audio-short term memory  The mapping of words onto meaning is thought to be subserved by widely distributed regions of the brain that specialize in particular modalities — for example visual aspects of the word banana reside in the occipital lobe of the brain and are activated when the word banana is heard BIBREF2 — and the different representation modalities are thought to be integrated into a single coherent latent representation in the anterior temporal lobe BIBREF3   While this part of meaning representation in human language comprehension is somewhat understood, much less is known about how the meanings of words are integrated together to form the meaning of sentences and discourses  One tool researchers use to study the integration of meaning across words is electroencephelography (EEG), which measures the electrical activity of large numbers of neurons acting in concert  EEG has the temporal resolution necessary to study the processes involved in meaning integration, and certain stereotyped electrical responses to word presentations, known as event-related potentials (ERPs), have been identified with some of the processes thought to contribute to comprehension 
In this work, we consider six ERP components that have been associated in the cognitive neuroscience and psycholinguistics literature with language processing and which we analyze in the data from BIBREF0 (see Figure FIGREF1 for spatial and temporal definitions of these ERP components)  Three of these — the N400, EPNP, and PNP responses — are primarily considered markers for semantic processing, while the other three — the P600, ELAN, and LAN responses — are primarily considered markers for syntactic processing  However, the neat division of the ERP responses into either semantic or syntactic categories is controversial  The N400 response has been very well studied (for an overview see BIBREF4 ) and it is well established that it is associated with semantic complexity, but the features of language that trigger the other ERP responses we consider here are poorly understood  We propose to use a neural network pretrained as a language model to probe what features of language drive these ERP responses, and in turn to probe what features of language mediate the cognitive processes that underlie human language comprehension, and especially the integration of meaning across words 
Background
While a full discussion of each ERP component and the features of language thought to trigger each are beyond the scope of this document (for reviews see e g  BIBREF0 , BIBREF2 , BIBREF4 , BIBREF5 , and BIBREF6 ), we introduce some basic features of ERP components to help in the discussion later  ERP components are electrical potential responses measured with respect to a baseline that are triggered by an event (in our case the presentation of a new word to a participant in an experiment)  The name of each ERP component reflects whether the potential is positive or negative relative to the baseline  The N400 is so-named because it is Negative relative to a baseline (the baseline is typically recorded just before a word is presented at an electrode that is not affected by the ERP response) and because it peaks in magnitude at about 400ms after a word is presented to a participant in an experiment  The P600 is Positive relative to a baseline and peaks around 600ms after a word is presented to a participant (though its overall duration is much longer and less specific in time than the N400)  The post-N400 positivity is so-named because it is part of a biphasic response; it is a positivity that occurs after the negativity associated with the N400  The early post-N400 positivity (EPNP) is also part of a biphasic response, but the positivity has an eariler onset than the standard PNP  Finally, the LAN and ELAN are the left-anterior negativity and early left-anterior negativity respectively  These are named for their timing, spatial distribution on the scalp, and direction of difference from the baseline  It is important to note that ERP components can potentially cancel and mask each other, and that it is difficult to precisely localize the neural activity that causes the changes in electrical potential at the electrodes where those changes are measured 
Related Work
This work is most closely related to the paper from which we get the ERP data: BIBREF0   In that work, the authors relate the surprisal of a word, i e  the (negative log) probability of the word appearing in its context, to each of the ERP signals we consider here  The authors do not directly train a model to predict ERPs  Instead, models of the probability distribution of each word in context are used to compute a surprisal for each word, which is input into a mixed effects regression along with word frequency, word length, word position in the sentence, and sentence position in the experiment  The effect of the surprisal is assessed using a likelihood-ratio test  In BIBREF7 , the authors take an approach similar to BIBREF0   The authors compare the explanatory power of surprisal (as computed by an LSTM or a Recurrent Neural Network Grammar (RNNG) language model) to a measure of syntactic complexity they call “distance" that counts the number of parser actions in the RNNG language model  The authors find that surprisal (as predicted by the RNNG) and distance are both significant factors in a mixed effects regression which predicts the P600, while the surprisal as computed by an LSTM is not  Unlike BIBREF0 and BIBREF7 , we do not use a linking function (e g  surprisal) to relate a language model to ERPs

ground_truth: Unidirectional LSTM networks with 2, 6, 7, 8, and  9 layers.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What datasets are used?
Context 2:  We thus lose the interpretability provided by the linking function, but we are able to predict a significant proportion of the variance for all of the ERP components, where prior work could not  We interpret our results through characterization of the ERPs in terms of how they relate to each other and to eye-tracking data rather than through a linking function  The authors in BIBREF8 also use a recurrent neural network to predict neural activity directly  In that work the authors predict magnetoencephalography (MEG) activity, a close cousin to EEG, recorded while participants read a chapter of Harry Potter and the Sorcerer’s Stone BIBREF9   Their approach to characterization of processing at each MEG sensor location is to determine whether it is best predicted by the context vector of the recurrent network (prior to the current word being processed), the embedding of the current word, or the probability of the current word given the context  In future work we also intend to add these types of studies to the ERP predictions 
Discussion
In this work we find that all six of the ERP components from BIBREF0 can be predicted above chance by a model which has been pretrained using a language modeling objective and then directly trained to predict the components  This is in contrast to prior work which has successfully linked language models to the N400 BIBREF0 and P600 BIBREF7 but not the other ERP components  We also note that contrary to BIBREF7 , we find that an LSTM does contain information that can be used to predict EEG data, and in particular that it can predict the P600  We speculate that the analysis used in BIBREF7 did not find reliable effects because the language models were related to the EEG data through functions chosen a priori (the surprisal, and the `distance' metric)  These functions, though interpretable, might be interpretable at the cost of losing much of the information in the representations learned by the network 
In addition, we show through our multitask learning analysis that information is shared between ERP components, and between ERP components and behavioral data  Although these relationships must be viewed with caution until they can be verified across multiple datasets and with more variation in neural network architectures, here we consider some potential reasons for our findings  The broad point we wish to make is that by better understanding which ERP components share information with each other and with behavioral data through the type of analysis we present here (multitask learning) or other means, we can better understand what drives each ERP component and in turn the processes involved in human language comprehension 
Conclusion
We have shown that ERP components can be predicted from neural networks pretrained as language models and fine-tuned to directly predict those components  To the best of our knowledge, prior work has not successfully used statistical models to predict all of these components  Furthermore, we have shown that multitask learning benefits the prediction of ERP components and can suggest how components relate to each other  At present, these joint-training benefit relationships are only suggestive, but if these relationships ultimately lead to insights about what drives each ERP component, then the components become more useful tools for studying human language comprehension  By using multitask learning as a method of characterization, we have found some expected relationships (LAN+P600 and ELAN+P600) and several more surprising relationships  We believe that this is exactly the kind of finding that makes multitask learning an interesting exploratory technique in this area  Additionally, we have shown that information can be shared between heterogeneous types of data (eye-tracking, self-paced reading, and ERP components) in the domain of human language processing prediction, and in particular between behavioral and neural data  Given the small datasets associated with human language processing, using heterogeneous data is a potentially major advantage of a multitask approach  In future work, we will further explore what information is encoded into the model representations when neural and behavioral data are used to train neural networks, and how these representations differ from the representations in a model trained on language alone 
Acknowledgments
We thank our reviewers for their valuable feedback  This work is supported in part by National Institutes of Health grant number U01NS098969 
Appendix
Here we present a visualization (Figure FIGREF21 ) of the results presented in Table TABREF9 of the main paper, and a visualization (Figure FIGREF22 ) of a more complete set of results from which the information in Table TABREF16 of the main paper is drawn  We also show supplemental results for variants of our primary analysis on multitask learning with eye-tracking, self-paced reading time and ERP data  In the variants we modify the input representation to our decoder network to see whether the relationships between the behavioral data and neural activity appear to be consistent with different choices of encoder architectures  Additional (and more varied) choices or architectures are left to future work  The results in Table TABREF23 reflect using only the forward-encoder (rather than the bi-LSTM) in the encoder network, while the results in Table TABREF24 reflect using only the word embeddings (i e  bypassing the LSTM entirely)  While the results are clearly worse for each of these choices of architecture than for using a bi-LSTM encoder, the relationships between the behavioral data and the ERP signals is qualitatively similar  Finally, TABREF25 shows the Pearson correlation coefficient between different measures  We note that the patterns of correlation are different than the patterns of which measures benefit from joint training with each other 

ground_truth: Answer with content missing: (Whole Method and Results sections) The primary dataset we use is the ERP data collected and computed by Frank et al. (2015), and we also use behavioral data (eye-tracking data and self-paced reading times) from Frank et al. (2013) which were collected on the same set of 205 sentences.
Select:
- ERP data collected and computed by Frank et al. (2015)
- behavioral data (eye-tracking data and self-paced reading times) from Frank et al. (2013)

the ERP data: BIBREF0

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the GhostVLAD approach?
Context 3: INTRODUCTION
The idea of language identification is to classify a given audio signal into a particular class using a classification algorithm  Commonly language identification task was done using i-vector systems [1]  A very well known approach for language identification proposed by N  Dahek et al  [1] uses the GMM-UBM model to obtain utterance level features called i-vectors  Recent advances in deep learning [15,16] have helped to improve the language identification task using many different neural network architectures which can be trained efficiently using GPUs for large scale datasets  These neural networks can be configured in various ways to obtain better accuracy for language identification task  Early work on using Deep learning for language Identification was published by Pavel Matejka et al  [2], where they used stacked bottleneck features extracted from deep neural networks for language identification task and showed that the bottleneck features learned by Deep neural networks are better than simple MFCC or PLP features  Later the work by I  Lopez-Moreno et al  [3] from Google showed how to use Deep neural networks to directly map the sequence of MFCC frames into its language class so that we can apply language identification at the frame level  Speech signals will have both spatial and temporal information, but simple DNNs are not able to capture temporal information  Work done by J  Gonzalez-Dominguez et al  [4] by Google developed an LSTM based language identification model which improves the accuracy over the DNN based models  Work done by Alicia et al  [5] used CNNs to improve upon i-vector [1] and other previously developed systems  The work done by Daniel Garcia-Romero et al  [6] has used a combination of Acoustic model trained for speech recognition with Time-delay neural networks where they train the TDNN model by feeding the stacked bottleneck features from acoustic model to predict the language labels at the frame level  Recently X-vectors [7] is proposed for speaker identification task and are shown to outperform all the previous state of the art speaker identification algorithms and are also used for language identification by David Snyder et al  [8] 
In this paper, we explore multiple pooling strategies for language identification task  Mainly we propose Ghost-VLAD based pooling method for language identification  Inspired by the recent work by W  Xie et al  [9] and Y  Zhong et al  [10], we use Ghost-VLAD to improve the accuracy of language identification task for Indian languages  We explore multiple pooling strategies including NetVLAD pooling [11], Average pooling and Statistics pooling( as proposed in X-vectors [7]) and show that Ghost-VLAD pooling is the best pooling strategy for language identification  Our model obtains the best accuracy of 98 24%, and it outperforms all the other previously proposed pooling methods  We conduct all our experiments on 635hrs of audio data for 7 Indian languages collected from $\textbf {All India Radio}$ news channel  The paper is organized as follows  In section 2, we explain the proposed pooling method for language identification  In section 3, we explain our dataset  In section 4, we describe the experiments, and in section 5, we describe the results 
POOLING STRATEGIES
In any language identification model, we want to obtain utterance level representation which has very good language discriminative features  These representations should be compact and should be easily separable by a linear classifier  The idea of any pooling strategy is to pool the frame-level representations into a single utterance level representation  Previous works by [7] have used simple mean and standard deviation aggregation to pool the frame-level features from the top layer of the neural network to obtain the utterance level features  Recently [9] used VLAD based pooling strategy for speaker identification which is inspired from [10] proposed for face recognition  The NetVLAD [11] and Ghost-VLAD [10] methods are proposed for Place recognition and face recognition, respectively, and in both cases, they try to aggregate the local descriptors into global features  In our case, the local descriptors are features extracted from ResNet [15], and the global utterance level feature is obtained by using GhostVLAD pooling  In this section, we explain different pooling methods, including NetVLAD, Ghost-VLAD, Statistic pooling, and Average pooling 
POOLING STRATEGIES ::: NetVLAD pooling
The NetVLAD pooling strategy was initially developed for place recognition by R  Arandjelovic et al  [11]  The NetVLAD is an extension to VLAD [18] approach where they were able to replace the hard assignment based clustering with soft assignment based clustering so that it can be trained with neural network in an end to end fashion  In our case, we use the NetVLAD layer to map N local features of dimension D into a fixed dimensional vector, as shown in Figure 1 (Left side) 
The model takes spectrogram as an input and feeds into CNN based ResNet architecture  The ResNet is used to map the spectrogram into 3D feature map of dimension HxWxD  We convert this 3D feature map into 2D by unfolding H and W dimensions, creating a NxD dimensional feature map, where N=HxW  The NetVLAD layer is kept on top of the feature extraction layer of ResNet, as shown in Figure 1  The NetVLAD now takes N features vectors of dimension D and computes a matrix V of dimension KxD, where K is the number clusters in the NetVLAD layer, and D is the dimension of the feature vector  The matrix V is computed as follows 
Where $w_k$,$b_k$ and $c_k$ are trainable parameters for the cluster $k$ and V(j,k) represents a point in the V matrix for (j,k)th location  The matrix is constructed using the equation (1) where the first term corresponds to the soft assignment of the input $x_i$ to the cluster $c_k$, whereas the second term corresponds to the residual term which tells how far the input descriptor $x_i$ is from the cluster center $c_k$ 
POOLING STRATEGIES ::: GhostVLAD pooling
GhostVLAD is an extension of the NetVLAD approach, which we discussed in the previous section  The GhostVLAD model was proposed for face recognition by Y  Zhong [10]  GhostVLAD works exactly similar to NetVLAD except it adds Ghost clusters along with the NetVLAD clusters  So, now we will have a K+G number of clusters instead of K clusters  Where G is the number of ghost clusters, we want to add (typically 2-4)  The Ghost clusters are added to map any noisy or irrelevant content into ghost clusters and are not included during the feature aggregation stage, as shown in Figure 1 (Right side)  Which means that we compute the matrix V for both normal cluster K and ghost clusters G, but we will not include the vectors belongs to ghost cluster from V during concatenation of the features  Due to which, during feature aggregation stage the contribution of the noisy and unwanted features to normal VLAD clusters are assigned less weights while Ghost clusters absorb most of the weight  We illustrate this in Figure 1(Right Side), where the ghost clusters are shown in red color  We use Ghost clusters when we are computing the V matrix, but they are excluded during the concatenation stage  These concatenated features are fed into the projection layer, followed by softmax to predict the language label 
POOLING STRATEGIES ::: Statistic and average pooling
In statistic pooling, we compute the first and second order statistics of the local features from the top layer of the ResNet model  The 3-D feature map is unfolded to create N features of D dimensions, and then we compute the mean and standard deviation of all these N vectors and get two D dimensional vectors, one for mean and the other for standard deviation  We then concatenate these 2 features and feed it to the projection layer for predicting the language label 
In the Average pooling layer, we compute only the first-order statistics (mean) of the local features from the top layer of the CNN model  The feature map from the top layer of CNN is unfolded to create N features of D dimensions, and then we compute the mean of all these N vectors and get D dimensional representation  We then feed this feature to the projection layer followed by softmax for predicting the language label 
DATASET
In this section, we describe our dataset collection process  We collected and curated around 635Hrs of audio data for 7 Indian languages, namely Kannada, Hindi, Telugu, Malayalam, Bengali, and English  We collected the data from the All India Radio news channel where an actor will be reading news for about 5-10 mins  To cover many speakers for the dataset, we crawled data from 2010 to 2019  Since the audio is very long to train any deep neural network directly, we segment the audio clips into smaller chunks using Voice activity detector  Since the audio clips will have music embedded during the news, we use Inhouse music detection model to remove the music segments from the dataset to make the dataset clean and our dataset contains 635Hrs of clean audio which is divided into 520Hrs of training data containing 165K utterances and 115Hrs of testing data containing 35K utterances  The amount of audio data for training and testing for each of the language is shown in the table bellow 
EXPERIMENTS
In this section, we describe the feature extraction process and network architecture in detail  We use spectral features of 256 dimensions computed using 512 point FFT for every frame, and we add an energy feature for every frame giving us total 257 features for every frame  We use a window size of 25ms and frame shift of 10ms during feature computation  We crop random 5sec audio data from each utterance during training which results in a spectrogram of size 257x500 (features x number of features)  We use these spectrograms as input to our CNN model during training  During testing, we compute the prediction score irrespective of the audio length 
For the network architecture, we use ResNet-34 architecture, as described in [9]  The model uses convolution layers with Relu activations to map the spectrogram of size 257x500 input into 3D feature map of size 1x32x512  This feature cube is converted into 2D feature map of dimension 32x512 and fed into Ghost-VLAD/NetVLAD layer to generate a representation that has more language discrimination capacity  We use Adam optimizer with an initial learning rate of 0 01 and a final learning rate of 0 00001 for training  Each model is trained for 15 epochs with early stopping criteria 
For the baseline, we train an i-vector model using GMM-UBM  We fit a small classifier on top of the generated i-vectors to measure the accuracy  This model is referred as i-vector+svm   To compare our model with the previous state of the art system, we set up the x-vector language identification system [8]  The x-vector model used time-delay neural networks (TDNN) along with statistic-pooling  We use 7 layer TDNN architecture similar to [8] for training  We refer to this model as tdnn+stat-pool   Finally, we set up a Deep LSTM based language identification system similar to [4] but with little modification where we add statistics pooling for the last layers hidden activities before classification  We use 3 layer Bi-LSTM with 256 hidden units at each layer  We refer to this model as LSTM+stat-pool  We train our i-vector+svm and TDNN+stat-pool using Kaldi toolkit  We train our NetVLAD and GhostVLAD experiments using Keras by modifying the code given by [9] for language identification  We train the LSTM+stat-pool and the remaining experiments using Pytorch [14] toolkit, and we will opensource all the codes and data soon 
RESULTS
In this section, we compare the performance of our system with the recent state of the art language identification approaches  We also compare different pooling strategies and finally, compare the robustness of our system to the length of the input spectrogram during training  We visualize the embeddings generated by the GhostVLAD method and conclude that the GhostVLAD embeddings shows very good feature discrimination capabilities 
RESULTS ::: Comparison with different approaches
We compare our system performance with the previous state of the art language identification approaches, as shown in Table 2  The i-vector+svm system is trained using GMM-UBM models to generate i-vectors as proposed in [1]  Once the i-vectors are extracted, we fit SVM classifier to classify the audio  The TDNN+stat-pool system is trained with a statistics pooling layer and is called the x-vector system as proposed by David Snyder et al  [11] and is currently the state of the art language identification approach as far as our knowledge  Our methods outperform the state of the art x-vector system by absolute 1 88% improvement in F1-score, as shown in Table 2 
RESULTS ::: Comparison with different pooling techniques
We compare our approach with different pooling strategies in Table 3  We use ResNet as our base feature extraction network  We keep the base network the same and change only the pooling layers to see which pooling approach performs better for language identification task  Our experiments show that GhostVLAD pooling outperforms all the other pooling methods by achieving 98 43% F1-Score 
RESULTS ::: Duration analysis
To observe the performance of our method with different input durations, we conducted an experiment where we train our model on different input durations  Since our model uses ResNet as the base feature extractor, we need to feed fixed-length spectrogram  We conducted 4 different experiments where we trained the model using 2sec, 3sec, 4sec and 5sec spectrograms containing 200,300,400 and 500 frames respectively  We observed that the model trained with a 5sec spectrogram is the best model, as shown in Table 4 
RESULTS ::: Visualization of embeddings
We visualize the embeddings generated by our approach to see the effectiveness  We extracted 512-dimensional embeddings for our testing data and reduced the dimensionality using t-sne projection  The t-sne plot of the embeddings space is shown in Figure 3  The plot shows that the embeddings learned by our approach has very good discriminative properties
Conclusion
In this work, we use Ghost-VLAD pooling approach that was originally proposed for face recognition to improve language identification performance for Indian languages  We collected and curated 630 hrs audio data from news All India Radio news channel for 7 Indian languages  Our experimental results shows that our approach outperforms the previous state of the art methods by an absolute 1 88% F1-score  We have also conducted experiments with different pooling strategies proposed in the past, and the GhostVLAD pooling approach turns out to be the best approach for aggregating frame-level features into a single utterance level feature  Our experiments also prove that our approach works much better even if the input during training contains smaller durations  Finally, we see that the embeddings generated by our method has very good language discriminative features and helps to improve the performance of language identification 

ground_truth: extension of the NetVLAD, adds Ghost clusters along with the NetVLAD clusters

An extension of NetVLAD which replaces hard assignment-based clustering with soft assignment-based clustering with the additon o fusing Ghost clusters to deal with noisy content.