prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. which lstm models did they compare with?
Context 1: Introduction
Recently, deep neural network has been widely employed in various recognition tasks  Increasing the depth of neural network is a effective way to improve the performance, and convolutional neural network (CNN) has benefited from it in visual recognition task BIBREF0   Deeper long short-term memory (LSTM) recurrent neural networks (RNNs) are also applied in large vocabulary continuous speech recognition (LVCSR) task, because LSTM networks have shown better performance than Fully-connected feed-forward deep neural network BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4  
Training neural network becomes more challenge when it goes deep  A conceptual tool called linear classifier probe is introduced to better understand the dynamics inside a neural network BIBREF5   The discriminating features of linear classifier is the hidden units of a intermediate layer  For deep neural networks, it is observed that deeper layer's accuracy is lower than that of shallower layers  Therefore, the tool shows the difficulty of deep neural model training visually 
Layer-wise pre-training is a successful method to train very deep neural networks BIBREF6   The convergence becomes harder with increasing the number of layers, even though the model is initialized with Xavier or its variants BIBREF7 , BIBREF8   But the deeper network which is initialized with a shallower trained network could converge well 
The size of LVCSR training dataset goes larger and training with only one GPU becomes high time consumption inevitably  Therefore, parallel training with multi-GPUs is more suitable for LVCSR system  Mini-batch based stochastic gradient descent (SGD) is the most popular method in neural network training procedure  Asynchronous SGD is a successful effort for parallel training based on it BIBREF9 , BIBREF10   It can many times speed up the training time without decreasing the accuracy  Besides, synchronous SGD is another effective effort, where the parameter server waits for every works to finish their computation and sent their local models to it, and then it sends updated model back to all workers BIBREF11   Synchronous SGD converges well in parallel training with data parallelism, and is also easy to implement 
In order to further improve the performance of deep neural network with parallel training, several methods are proposed  Model averaging method achieves linear speedup, as the final model is averaged from all parameters of local models in different workers BIBREF12 , BIBREF13 , but the accuracy decreases compared with single GPU training  Moreover, blockwise model-updating filter (BMUF) provides another almost linear speedup approach with multi-GPUs on the basis of model averaging  It can achieve improvement or no-degradation of recognition performance compared with mini-batch SGD on single GPU BIBREF14  
Moving averaged (MA) approaches are also proposed for parallel training  It is demonstrated that the moving average of the parameters obtained by SGD performs as well as the parameters that minimize the empirical cost, and moving average parameters can be used as the estimator of them, if the size of training data is large enough BIBREF15   One pass learning is then proposed, which is the combination of learning rate schedule and averaged SGD using moving average BIBREF16   Exponential moving average (EMA) is proposed as a non-interference method BIBREF17   EMA model is not broadcasted to workers to update their local models, and it is applied as the final model of entire training process  EMA method is utilized with model averaging and BMUF to further decrease the character error rate (CER)  It is also easy to implement in existing parallel training systems 
Frame stacking can also speed up the training time BIBREF18   The super frame is stacked by several regular frames, and it contains the information of them  Thus, the network can see multiple frames at a time, as the super frame is new input  Frame stacking can also lead to faster decoding 
For streaming voice search service, it needs to display intermediate recognition results while users are still speaking  As a result, the system needs to fulfill high real-time requirement, and we prefer unidirectional LSTM network rather than bidirectional one  High real-time requirement means low real time factor (RTF), but the RTF of deep LSTM model is higher inevitably  The dilemma of recognition accuracy and real-time requirement is an obstacle to the employment of deep LSTM network  Deep model outperforms because it contains more knowledge, but it is also cumbersome  As a result, the knowledge of deep model can be distilled to a shallow model BIBREF19   It provided a effective way to employ the deep model to the real-time system 
In this paper, we explore a entire deep LSTM RNN training framework, and employ it to real-time application  The deep learning systems benefit highly from a large quantity of labeled training data  Our first and basic speech recognition system is trained on 17000 hours of Shenma voice search dataset  It is a generic dataset sampled from diverse aspects of search queries  The requirement of speech recognition system also addressed by specific scenario, such as map and navigation task  The labeled dataset is too expensive, and training a new model with new large dataset from the beginning costs lots of time  Thus, it is natural to think of transferring the knowledge from basic model to new scenario's model  Transfer learning expends less data and less training time than full training  In this paper, we also introduce a novel transfer learning strategy with segmental Minimum Bayes-Risk (sMBR)  As a result, transfer training with only 1000 hours data can match equivalent performance for full training with 7300 hours data 
Our deep LSTM training framework for LVCSR is presented in Section 2  Section 3 describes how the very deep models does apply in real world applications, and how to transfer the model to another task  The framework is analyzed and discussed in Section 4, and followed by the conclusion in Section 5 
Layer-wise Training with Soft Target and Hard Target
Gradient-based optimization of deep LSTM network with random initialization get stuck in poor solution easily  Xavier initialization can partially solve this problem BIBREF7 , so this method is the regular initialization method of all training procedure  However, it does not work well when it is utilized to initialize very deep model directly, because of vanishing or exploding gradients  Instead, layer-wise pre-training method is a effective way to train the weights of very deep architecture BIBREF6 , BIBREF20   In layer-wise pre-training procedure, a one-layer LSTM model is firstly trained with normalized initialization  Sequentially, two-layers LSTM model's first layer is initialized by trained one-layer model, and its second layer is regularly initialized  In this way, a deep architecture is layer-by-layer trained, and it can converge well 
In conventional layer-wise pre-training, only parameters of shallower network are transfered to deeper one, and the learning targets are still the alignments generated by HMM-GMM system  The targets are vectors that only one state's probability is one, and the others' are zeros  They are known as hard targets, and they carry limited knowledge as only one state is active  In contrast, the knowledge of shallower network should be also transfered to deeper one  It is obtained by the softmax layer of existing model typically, so each state has a probability rather than only zero or one, and called as soft target  As a result, the deeper network which is student network learns the parameters and knowledge from shallower one which is called teacher network  When training the student network from the teacher network, the final alignment is the combination of hard target and soft target in our layer-wise training phase  The final alignment provides various knowledge which transfered from teacher network and extracted from true labels  If only soft target is learned, student network perform no better than teacher network, but it could outperform teacher network as it also learns true labels 
The deeper network spends less time to getting the same level of original network than the network trained from the beginning, as a period of low performance is skipped  Therefore, training with hard and soft target is a time saving method  For large training dataset, training with the whole dataset still spends too much time  A network firstly trained with only a small part of dataset could go deeper as well, and so the training time reducing rapidly  When the network is deep enough, it then trained on the entire dataset to get further improvement  There is no gap of accuracy between these two approaches, but latter one saves much time 
Differential Saturation Check
The objects of conventional saturation check are gradients and the cell activations BIBREF4   Gradients are clipped to range [-5, 5], while the cell activations clipped to range [-50, 50]  Apart from them, the differentials of recurrent layers is also limited  If the differentials go beyond the range, corresponding back propagation is skipped, while if the gradients and cell activations go beyond the bound, values are set as the boundary values  The differentials which are too large or too small will lead to the gradients easily vanishing, and it demonstrates the failure of this propagation  As a result, the parameters are not updated, and next propagation  
Sequence Discriminative Training
Cross-entropy (CE) is widely used in speech recognition training system as a frame-wise discriminative training criterion  However, it is not well suited to speech recognition, because speech recognition training is a sequential learning problem  In contrast, sequence discriminative training criterion has shown to further improve performance of neural network first trained with cross-entropy BIBREF21 , BIBREF22 , BIBREF23   We choose state-level minimum bayes risk (sMBR) BIBREF21 among a number of sequence discriminative criterion is proposed, such as maximum mutual information (MMI) BIBREF24 and minimum phone error (MPE) BIBREF25   MPE and sMBR are designed to minimize the expected error of different granularity of labels, while CE aims to minimizes expected frame error, and MMI aims to minimizes expected sentence error  State-level information is focused on by sMBR 
a frame-level accurate model is firstly trained by CE loss function, and then sMBR loss function is utilized for further training to get sequence-level accuracy  Only a part of training dataset is needed in sMBR training phase on the basis of whole dataset CE training 
Parallel Training
It is demonstrated that training with larger dataset can improve recognition accuracy  However, larger dataset means more training samples and more model parameters  Therefore, parallel training with multiple GPUs is essential, and it makes use of data parallelism BIBREF9   The entire training data is partitioned into several split without overlapping and they are distributed to different GPUs  Each GPU trains with one split of training dataset locally  All GPUs synchronize their local models with model average method after a mini-batch optimization BIBREF12 , BIBREF13  
Model average method achieves linear speedup in training phase, but the recognition accuracy decreases compared with single GPU training  Block-wise model updating filter (BMUF) is another successful effort in parallel training with linear speedup as well  It can achieve no-degradation of recognition accuracy with multi-GPUs BIBREF14   In the model average method, aggregated model INLINEFORM0 is computed and broadcasted to GPUs  On the basis of it, BMUF proposed a novel model updating strategy: INLINEFORM1 INLINEFORM2
Where INLINEFORM0 denotes model update, and INLINEFORM1 is the global-model update  There are two parameters in BMUF, block momentum INLINEFORM2 , and block learning rate INLINEFORM3   Then, the global model is updated as INLINEFORM4
Consequently, INLINEFORM0 is broadcasted to all GPUs to initial their local models, instead of INLINEFORM1 in model average method 
Averaged SGD is proposed to further accelerate the convergence speed of SGD  Averaged SGD leverages the moving average (MA) INLINEFORM0 as the estimator of INLINEFORM1 BIBREF15 : INLINEFORM2
Where INLINEFORM0 is computed by model averaging or BMUF  It is shown that INLINEFORM1 can well converge to INLINEFORM2 , with the large enough training dataset in single GPU training  It can be considered as a non-interference strategy that INLINEFORM3 does not participate the main optimization process, and only takes effect after the end of entire optimization  However, for the parallel training implementation, each INLINEFORM4 is computed by model averaging and BMUF with multiple models, and moving average model INLINEFORM5 does not well converge, compared with single GPU training 
Model averaging based methods are employed in parallel training of large scale dataset, because of their faster convergence, and especially no-degradation implementation of BMUF  But combination of model averaged based methods and moving average does not match the expectation of further enhance performance and it is presented as INLINEFORM0
The weight of each INLINEFORM0 is equal in moving average method regardless the effect of temporal order  But INLINEFORM1 closer to the end of training achieve higher accuracy in the model averaging based approach, and thus it should be with more proportion in final INLINEFORM2   As a result, exponential moving average(EMA) is appropriate, which the weight for each older parameters decrease exponentially, and never reaching zero  After moving average based methods, the EMA parameters are updated recursively as INLINEFORM3
Here INLINEFORM0 represents the degree of weight decrease, and called exponential updating rate  EMA is also a non-interference training strategy that is implemented easily, as the updated model is not broadcasted  Therefore, there is no need to add extra learning rate updating approach, as it can be appended to existing training procedure directly 
Deployment
There is a high real time requirement in real world application, especially in online voice search system  Shenma voice search is one of the most popular mobile search engines in China, and it is a streaming service that intermediate recognition results displayed while users are still speaking  Unidirectional LSTM network is applied, rather than bidirectional one, because it is well suited to real-time streaming speech recognition 
Distillation
It is demonstrated that deep neural network architecture can achieve improvement in LVCSR  However, it also leads to much more computation and higher RTF, so that the recognition result can not be displayed in real time  It should be noted that deeper neural network contains more knowledge, but it is also cumbersome  the knowledge is key to improve the performance  If it can be transfered from cumbersome model to a small model, the recognition ability can also be transfered to the small model  Knowledge transferring to small model is called distillation BIBREF19   The small model can perform as well as cumbersome model, after distilling  It provide a way to utilize high-performance but high RTF model in real time system  The class probability produced by the cumbersome model is regarded as soft target, and the generalization ability of cumbersome model is transfered to small model with it  Distillation is model's knowledge transferring approach, so there is no need to use the hard target, which is different with the layer-wise training method 
9-layers unidirectional LSTM model achieves outstanding performance, but meanwhile it is too computationally expensive to allow deployment in real time recognition system  In order to ensure real-time of the system, the number of layers needs to be reduced  The shallower network can learn the knowledge of deeper network with distillation  It is found that RTF of 2-layers network is acceptable, so the knowledge is distilled from 9-layers well-trained model to 2-layers model  Table TABREF16 shows that distillation from 9-layers to 2-layers brings RTF decrease of relative 53%, while CER only increases 5%  The knowledge of deep network is almost transfered with distillation, Distillation brings promising RTF reduction, but only little knowledge of deep network is lost  Moreover, CER of 2-layers distilled LSTM decreases relative 14%, compared with 2-layers regular-trained LSTM 
Transfer Learning with sMBR
For a certain specific scenario, the model trained with the data recorded from it has better adaptation than the model trained with generic scenario  But it spends too much time training a model from the beginning, if there is a well-trained model for generic scenarios  Moreover, labeling a large quantity of training data in new scenario is both costly and time consuming  If a model transfer trained with smaller dataset can obtained the similar recognition accuracy compared with the model directly trained with larger dataset, it is no doubt that transfer learning is more practical  Since specific scenario is a subset of generic scenario, some knowledge can be shared between them  Besides, generic scenario consists of various conditions, so its model has greater robustness  As a result, not only shared knowledge but also robustness can be transfered from the model of generic scenario to the model of specific one 
As the model well trained from generic scenario achieves good performance in frame level classification, sequence discriminative training is required to adapt new model to specific scenario additionally  Moreover, it does not need alignment from HMM-GMM system, and it also saves amount of time to prepare alignment 
Training Data
A large quantity of labeled data is needed for training a more accurate acoustic model  We collect the 17000 hours labeled data from Shenma voice search, which is one of the most popular mobile search engines in China  The dataset is created from anonymous online users' search queries in Mandarin, and all audio file's sampling rate is 16kHz, recorded by mobile phones  This dataset consists of many different conditions, such as diverse noise even low signal-to-noise, babble, dialects, accents, hesitation and so on 
In the Amap, which is one of the most popular web mapping and navigation services in China, users can search locations and navigate to locations they want though voice search  To present the performance of transfer learning with sequence discriminative training, the model trained from Shenma voice search which is greneric scenario transfer its knowledge to the model of Amap voice search  7300 hours labeled data is collected in the similar way of Shenma voice search data collection 
Two dataset is divided into training set, validation set and test set separately, and the quantity of them is shown in Table TABREF10   The three sets are split according to speakers, in order to avoid utterances of same speaker appearing in three sets simultaneously  The test sets of Shenma and Amap voice search are called Shenma Test and Amap Test 
Experimental setup
LSTM RNNs outperform conventional RNNs for speech recognition system, especially deep LSTM RNNs, because of its long-range dependencies more accurately for temporal sequence conditions BIBREF26 , BIBREF23   Shenma and Amap voice search is a streaming service that intermediate recognition results displayed while users are still speaking  So as for online recognition in real time, we prefer unidirectional LSTM model rather than bidirectional one  Thus, the training system is unidirectional LSTM-based 
A 26-dimensional filter bank and 2-dimensional pitch feature is extracted for each frame, and is concatenated with first and second order difference as the final input of the network  The super frame are stacked by 3 frames without overlapping  The architecture we trained consists of two LSTM layers with sigmoid activation function, followed by a full-connection layer  The out layer is a softmax layer with 11088 hidden markov model (HMM) tied-states as output classes, the loss function is cross-entropy (CE)  The performance metric of the system in Mandarin is reported with character error rate (CER)  The alignment of frame-level ground truth is obtained by GMM-HMM system  Mini-batched SGD is utilized with momentum trick and the network is trained for a total of 4 epochs  The block learning rate and block momentum of BMUF are set as 1 and 0 9  5-gram language model is leveraged in decoder, and the vocabulary size is as large as 760000  Differentials of recurrent layers is limited to range [-10000,10000], while gradients are clipped to range [-5, 5] and cell activations clipped to range [-50, 50]  After training with CE loss, sMBR loss is employed to further improve the performance 
It has shown that BMUF outperforms traditional model averaging method, and it is utilized at the synchronization phase  After synchronizing with BMUF, EMA method further updates the model in non-interference way  The training system is deployed on the MPI-based HPC cluster where 8 GPUs  Each GPU processes non-overlap subset split from the entire large scale dataset in parallel 
Local models from distributed workers synchronize with each other in decentralized way  In the traditional model averaging and BMUF method, a parameter server waits for all workers to send their local models, aggregate them, and send the updated model to all workers  Computing resource of workers is wasted until aggregation of the parameter server done  Decentralized method makes full use of computing resource, and we employ the MPI-based Mesh AllReduce method  It is mesh topology as shown in Figure FIGREF12   There is no centralized parameter server, and peer to peer communication is used to transmit local models between workers  Local model INLINEFORM0 of INLINEFORM1 -th worker in INLINEFORM2 workers cluster is split to INLINEFORM3 pieces INLINEFORM4 , and send to corresponding worker  In the aggregation phase, INLINEFORM5 -th worker computed INLINEFORM6 splits of model INLINEFORM7 and send updated model INLINEFORM8 back to workers  As a result, all workers participate in aggregation and no computing resource is dissipated  It is significant to promote training efficiency, when the size of neural network model is too large  The EMA model is also updated additionally, but not broadcasting it 
Results
In order to evaluate our system, several sets of experiments are performed  The Shenma test set including about 9000 samples and Amap test set including about 7000 samples contain various real world conditions  It simulates the majority of user scenarios, and can well evaluates the performance of a trained model  Firstly, we show the results of models trained with EMA method  Secondly, for real world applications, very deep LSTM is distilled to a shallow one, so as for lower RTF  The model of Amap is also needed to train for map and navigation scenarios  The performance of transfer learning from Shenma voice search to Amap voice search is also presented 
Layer-wise Training
In layer-wise training, the deeper model learns both parameters and knowledge from the shallower model  The deeper model is initialized by the shallower one, and its alignment is the combination of hard target and soft target of shallower one  Two targets have the same weights in our framework  The teacher model is trained with CE  For each layer-wise trained CE model, corresponding sMBR model is also trained, as sMBR could achieve additional improvement  In our framework, 1000 hours data is randomly selected from the total dataset for sMBR training  There is no obvious performance enhancement when the size of sMBR training dataset increases 
For very deep unidirectional LSTM initialized with Xavier initialization algorithm, 6-layers model converges well, but there is no further improvement with increasing the number of layers  Therefore, the first 6 layers of 7-layers model is initialized by 6-layers model, and soft target is provided by 6-layers model  Consequently, deeper LSTM is also trained in the same way  It should be noticed that the teacher model of 9-layers model is the 8-layers model trained by sMBR, while the other teacher model is CE model  As shown in Table TABREF15 , the layer-wise trained models always performs better than the models with Xavier initialization, as the model is deep  Therefore, for the last layer training, we choose 8-layers sMBR model as the teacher model instead of CE model  A comparison between 6-layers and 9-layers sMBR models shows that 3 additional layers of layer-wise training brings relative 12 6% decreasing of CER  It is also significant that the averaged CER of sMBR models with different layers decreases absolute 0 73% approximately compared with CE models, so the improvement of sequence discriminative learning is promising 
Transfer Learning
2-layers distilled model of Shenma voice search has shown a impressive performance on Shenma Test, and we call it Shenma model  It is trained for generic search scenario, but it has less adaptation for specific scenario like Amap voice search  Training with very large dataset using CE loss is regarded as improvement of frame level recognition accuracy, and sMBR with less dataset further improves accuracy as sequence discriminative training  If robust model of generic scenario is trained, there is no need to train a model with very large dataset, and sequence discriminative training with less dataset is enough  Therefore, on the basis of Shenma model, it is sufficient to train a new Amap model with small dataset using sMBR  As shown in Table TABREF18 , Shenma model presents the worst performance among three methods, since it does not trained for Amap scenario  2-layers Shenma model further trained with sMBR achieves about 8 1% relative reduction, compared with 2-layers regular-trained Amap model  Both training sMBR datasets contain the same 1000 hours data  As a result, with the Shenma model, only about 14% data usage achieves lower CER, and it leads to great time and cost saving with less labeled data  Besides, transfer learning with sMBR does not use the alignment from the HMM-GMM system, so it also saves huge amount of time 
Conclusion
We have presented a whole deep unidirectional LSTM parallel training system for LVCSR  The recognition performance improves when the network goes deep  Distillation makes it possible that deep LSTM model transfer its knowledge to shallow model with little loss  The model could be distilled to 2-layers model with very low RTF, so that it can display the immediate recognition results  As a result, its CER decrease relatively 14%, compared with the 2-layers regular trained model  In addition, transfer learning with sMBR is also proposed  If a great model has well trained from generic scenario, only 14% of the size of training dataset is needed to train a more accuracy acoustic model for specific scenario  Our future work includes 1) finding more effective methods to reduce the CER by increasing the number of layers; 2) applying this training framework to Connectionist Temporal Classification (CTC) and attention-based neural networks 
Context 2: Introduction
End-to-end speech-to-text translation (ST) has attracted much attention recently BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 given its simplicity against cascading automatic speech recognition (ASR) and machine translation (MT) systems  The lack of labeled data, however, has become a major blocker for bridging the performance gaps between end-to-end models and cascading systems  Several corpora have been developed in recent years  post2013improved introduced a 38-hour Spanish-English ST corpus by augmenting the transcripts of the Fisher and Callhome corpora with English translations  di-gangi-etal-2019-must created the largest ST corpus to date from TED talks but the language pairs involved are out of English only  beilharz2019librivoxdeen created a 110-hour German-English ST corpus from LibriVox audiobooks  godard-etal-2018-low created a Moboshi-French ST corpus as part of a rare language documentation effort  woldeyohannis provided an Amharic-English ST corpus in the tourism domain  boito2019mass created a multilingual ST corpus involving 8 languages from a multilingual speech corpus based on Bible readings BIBREF7  Previous work either involves language pairs out of English, very specific domains, very low resource languages or a limited set of language pairs  This limits the scope of study, including the latest explorations on end-to-end multilingual ST BIBREF8, BIBREF9  Our work is mostly similar and concurrent to iranzosnchez2019europarlst who created a multilingual ST corpus from the European Parliament proceedings  The corpus we introduce has larger speech durations and more translation tokens  It is diversified with multiple speakers per transcript/translation  Finally, we provide additional out-of-domain test sets 
In this paper, we introduce CoVoST, a multilingual ST corpus based on Common Voice BIBREF10 for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  It includes a total 708 hours of French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh) speeches, with French and German ones having the largest durations among existing public corpora  We also collect an additional evaluation corpus from Tatoeba for French, German, Dutch, Russian and Spanish, resulting in a total of 9 3 hours of speech  Both corpora are created at the sentence level and do not require additional alignments or segmentation  Using the official Common Voice train-development-test split, we also provide baseline models, including, to our knowledge, the first end-to-end many-to-one multilingual ST models  CoVoST is released under CC0 license and free to use  The Tatoeba evaluation samples are also available under friendly CC licenses  All the data can be acquired at https://github com/facebookresearch/covost 
Data Collection and Processing ::: Common Voice (CoVo)
Common Voice BIBREF10 is a crowdsourcing speech recognition corpus with an open CC0 license  Contributors record voice clips by reading from a bank of donated sentences  Each voice clip was validated by at least two other users  Most of the sentences are covered by multiple speakers, with potentially different genders, age groups or accents 
Raw CoVo data contains samples that passed validation as well as those that did not  To build CoVoST, we only use the former one and reuse the official train-development-test partition of the validated data  As of January 2020, the latest CoVo 2019-06-12 release includes 29 languages  CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese 
Validated transcripts were sent to professional translators  Note that the translators had access to the transcripts but not the corresponding voice clips since clips would not carry additional information  Since transcripts were duplicated due to multiple speakers, we deduplicated the transcripts before sending them to translators  As a result, different voice clips of the same content (transcript) will have identical translations in CoVoST for train, development and test splits 
In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11  1) For German-English, French-English and Russian-English translations, we computed sentence-level BLEU BIBREF12 with the NLTK BIBREF13 implementation between the human translations and the automatic translations produced by a state-of-the-art system BIBREF14 (the French-English system was a Transformer big BIBREF15 separately trained on WMT14)  We applied this method to these three language pairs only as we are confident about the quality of the corresponding systems  Translations with a score that was too low were manually inspected and sent back to the translators when needed  2) We manually inspected examples where the source transcript was identical to the translation  3) We measured the perplexity of the translations using a language model trained on a large amount of clean monolingual data BIBREF14  We manually inspected examples where the translation had a high perplexity and sent them back to translators accordingly  4) We computed the ratio of English characters in the translations  We manually inspected examples with a low ratio and sent them back to translators accordingly  5) Finally, we used VizSeq BIBREF16 to calculate similarity scores between transcripts and translations based on LASER cross-lingual sentence embeddings BIBREF17  Samples with low scores were manually inspected and sent back for translation when needed 
We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint 
Data Collection and Processing ::: Tatoeba (TT)
Tatoeba (TT) is a community built language learning corpus having sentences aligned across multiple languages with the corresponding speech partially available  Its sentences are on average shorter than those in CoVoST (see also Table TABREF2) given the original purpose of language learning  Sentences in TT are licensed under CC BY 2 0 FR and part of the speeches are available under various CC licenses 
We construct an evaluation set from TT (for French, German, Dutch, Russian and Spanish) as a complement to CoVoST development and test sets  We collect (speech, transcript, English translation) triplets for the 5 languages and do not include those whose speech has a broken URL or is not CC licensed  We further filter these samples by sentence lengths (minimum 4 words including punctuations) to reduce the portion of short sentences  This makes the resulting evaluation set closer to real-world scenarios and more challenging 
We run the same quality checks for TT as for CoVoST but we do not find poor quality translations according to our criteria  Finally, we report the overlap between CoVo transcripts and TT sentences in Table TABREF5  We found a minimal overlap, which makes the TT evaluation set a suitable additional test set when training on CoVoST 
Data Analysis ::: Basic Statistics
Basic statistics for CoVoST and TT are listed in Table TABREF2 including (unique) sentence counts, speech durations, speaker demographics (partially available) as well as vocabulary and token statistics (based on Moses-tokenized sentences by sacreMoses) on both transcripts and translations  We see that CoVoST has over 327 hours of German speeches and over 171 hours of French speeches, which, to our knowledge, corresponds to the largest corpus among existing public ST corpora (the second largest is 110 hours BIBREF18 for German and 38 hours BIBREF19 for French)  Moreover, CoVoST has a total of 18 hours of Dutch speeches, to our knowledge, contributing the first public Dutch ST resource  CoVoST also has around 27-hour Russian speeches, 37-hour Italian speeches and 67-hour Persian speeches, which is 1 8 times, 2 5 times and 13 3 times of the previous largest public one BIBREF7  Most of the sentences (transcripts) in CoVoST are covered by multiple speakers with potentially different accents, resulting in a rich diversity in the speeches

ground_truth: Unidirectional LSTM networks with 2, 6, 7, 8, and  9 layers.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How big is Augmented LibriSpeech dataset?
Context 2:  For example, there are over 1,000 speakers and over 10 accents in the French and German development / test sets  This enables good coverage of speech variations in both model training and evaluation 
Data Analysis ::: Speaker Diversity
As we can see from Table TABREF2, CoVoST is diversified with a rich set of speakers and accents  We further inspect the speaker demographics in terms of sample distributions with respect to speaker counts, accent counts and age groups, which is shown in Figure FIGREF6, FIGREF7 and FIGREF8  We observe that for 8 of the 11 languages, at least 60% of the sentences (transcripts) are covered by multiple speakers  Over 80% of the French sentences have at least 3 speakers  And for German sentences, even over 90% of them have at least 5 speakers  Similarly, we see that a large portion of sentences are spoken in multiple accents for French, German, Dutch and Spanish  Speakers of each language also spread widely across different age groups (below 20, 20s, 30s, 40s, 50s, 60s and 70s) 
Baseline Results
We provide baselines using the official train-development-test split on the following tasks: automatic speech recognition (ASR), machine translation (MT) and speech translation (ST) 
Baseline Results ::: Experimental Settings ::: Data Preprocessing
We convert raw MP3 audio files from CoVo and TT into mono-channel waveforms, and downsample them to 16,000 Hz  For transcripts and translations, we normalize the punctuation, we tokenize the text with sacreMoses and lowercase it  For transcripts, we further remove all punctuation markers except for apostrophes  We use character vocabularies on all the tasks, with 100% coverage of all the characters  Preliminary experimentation showed that character vocabularies provided more stable training than BPE  For MT, the vocabulary is created jointly on both transcripts and translations  We extract 80-channel log-mel filterbank features, computed with a 25ms window size and 10ms window shift using torchaudio  The features are normalized to 0 mean and 1 0 standard deviation  We remove samples having more than 3,000 frames or more than 256 characters for GPU memory efficiency (less than 25 samples are removed for all languages) 
Baseline Results ::: Experimental Settings ::: Model Training
Our ASR and ST models follow the architecture in berard2018end, but have 3 decoder layers like that in pino2019harnessing  For MT, we use a Transformer base architecture BIBREF15, but with 3 encoder layers, 3 decoder layers and 0 3 dropout  We use a batch size of 10,000 frames for ASR and ST, and a batch size of 4,000 tokens for MT  We train all models using Fairseq BIBREF20 for up to 200,000 updates  We use SpecAugment BIBREF21 for ASR and ST to alleviate overfitting 
Baseline Results ::: Experimental Settings ::: Inference and Evaluation
We use a beam size of 5 for all models  We use the best checkpoint by validation loss for MT, and average the last 5 checkpoints for ASR and ST  For MT and ST, we report case-insensitive tokenized BLEU BIBREF22 using sacreBLEU BIBREF23  For ASR, we report word error rate (WER) and character error rate (CER) using VizSeq 
Baseline Results ::: Automatic Speech Recognition (ASR)
For simplicity, we use the same model architecture for ASR and ST, although we do not leverage ASR models to pretrain ST model encoders later  Table TABREF18 shows the word error rate (WER) and character error rate (CER) for ASR models  We see that French and German perform the best given they are the two highest resource languages in CoVoST  The other languages are relatively low resource (especially Turkish and Swedish) and the ASR models are having difficulties to learn from this data 
Baseline Results ::: Machine Translation (MT)
MT models take transcripts (without punctuation) as inputs and outputs translations (with punctuation)  For simplicity, we do not change the text preprocessing methods for MT to correct this mismatch  Moreover, this mismatch also exists in cascading ST systems, where MT model inputs are the outputs of an ASR model  Table TABREF20 shows the BLEU scores of MT models  We notice that the results are consistent with what we see from ASR models  For example thanks to abundant training data, French has a decent BLEU score of 29 8/25 4  German doesn't perform well, because of less richness of content (transcripts)  The other languages are low resource in CoVoST and it is difficult to train decent models without additional data or pre-training techniques 
Baseline Results ::: Speech Translation (ST)
CoVoST is a many-to-one multilingual ST corpus  While end-to-end one-to-many and many-to-many multilingual ST models have been explored very recently BIBREF8, BIBREF9, many-to-one multilingual models, to our knowledge, have not  We hence use CoVoST to examine this setting  Table TABREF22 and TABREF23 show the BLEU scores for both bilingual and multilingual end-to-end ST models trained on CoVoST  We observe that combining speeches from multiple languages is consistently bringing gains to low-resource languages (all besides French and German)  This includes combinations of distant languages, such as Ru+Fr, Tr+Fr and Zh+Fr  Moreover, some combinations do bring gains to high-resource language (French) as well: Es+Fr, Tr+Fr and Mn+Fr  We simply provide the most basic many-to-one multilingual baselines here, and leave the full exploration of the best configurations to future work  Finally, we note that for some language pairs, absolute BLEU numbers are relatively low as we restrict model training to the supervised data  We encourage the community to improve upon those baselines, for example by leveraging semi-supervised training 
Baseline Results ::: Multi-Speaker Evaluation
In CoVoST, large portion of transcripts are covered by multiple speakers with different genders, accents and age groups  Besides the standard corpus-level BLEU scores, we also want to evaluate model output variance on the same content (transcript) but different speakers  We hence propose to group samples (and their sentence BLEU scores) by transcript, and then calculate average per-group mean and average coefficient of variation defined as follows:
and
where $G$ is the set of sentence BLEU scores grouped by transcript and $G^{\prime } = \lbrace g | g\in G, |g|>1, \textrm {Mean}(g) > 0 \rbrace $ 
$\textrm {BLEU}_{MS}$ provides a normalized quality score as oppose to corpus-level BLEU or unnormalized average of sentence BLEU  And $\textrm {CoefVar}_{MS}$ is a standardized measure of model stability against different speakers (the lower the better)  Table TABREF24 shows the $\textrm {BLEU}_{MS}$ and $\textrm {CoefVar}_{MS}$ of our ST models on CoVoST test set  We see that German and Persian have the worst $\textrm {CoefVar}_{MS}$ (least stable) given their rich speaker diversity in the test set and relatively small train set (see also Figure FIGREF6 and Table TABREF2)  Dutch also has poor $\textrm {CoefVar}_{MS}$ because of the lack of training data  Multilingual models are consistantly more stable on low-resource languages  Ru+Fr, Tr+Fr, Fa+Fr and Zh+Fr even have better $\textrm {CoefVar}_{MS}$ than all individual languages 
Conclusion
We introduce a multilingual speech-to-text translation corpus, CoVoST, for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  We also provide baseline results, including, to our knowledge, the first end-to-end many-to-one multilingual model for spoken language translation  CoVoST is free to use with a CC0 license, and the additional Tatoeba evaluation samples are also CC-licensed 
Context 3: Introduction
The concept of message passing over graphs has been around for many years BIBREF0, BIBREF1, as well as that of graph neural networks (GNNs) BIBREF2, BIBREF3  However, GNNs have only recently started to be closely investigated, following the advent of deep learning  Some notable examples include BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12  These approaches are known as spectral  Their similarity with message passing (MP) was observed by BIBREF9 and formalized by BIBREF13 and BIBREF14 
The MP framework is based on the core idea of recursive neighborhood aggregation  That is, at every iteration, the representation of each vertex is updated based on messages received from its neighbors  All spectral GNNs can be described in terms of the MP framework 
GNNs have been applied with great success to bioinformatics and social network data, for node classification, link prediction, and graph classification  However, a few studies only have focused on the application of the MP framework to representation learning on text  This paper proposes one such application  More precisely, we represent documents as word co-occurrence networks, and develop an expressive MP GNN tailored to document understanding, the Message Passing Attention network for Document understanding (MPAD)  We also propose several hierarchical variants of MPAD  Evaluation on 10 document classification datasets show that our architectures learn representations that are competitive with the state-of-the-art  Furthermore, ablation experiments shed light on the impact of various architectural choices 
In what follows, we first provide some background about the MP framework (in sec  SECREF2), thoroughly describe and explain MPAD (sec  SECREF3), present our experimental framework (sec  SECREF4), report and interpret our results (sec  SECREF5), and provide a review of the relevant literature (sec  SECREF6) 
Message Passing Neural Networks
BIBREF13 proposed a MP framework under which many of the recently introduced GNNs can be reformulated  MP consists in an aggregation phase followed by a combination phase BIBREF14  More precisely, let $G(V,E)$ be a graph, and let us consider $v \in V$  At time $t+1$, a message vector $\mathbf {m}_v^{t+1}$ is computed from the representations of the neighbors $\mathcal {N}(v)$ of $v$:
The new representation $\mathbf {h}^{t+1}_v$ of $v$ is then computed by combining its current feature vector $\mathbf {h}^{t}_v$ with the message vector $\mathbf {m}_v^{t+1}$:
Messages are passed for $T$ time steps  Each step is implemented by a different layer of the MP network  Hence, iterations correspond to network depth  The final feature vector $\mathbf {h}_v^T$ of $v$ is based on messages propagated from all the nodes in the subtree of height $T$ rooted at $v$  It captures both the topology of the neighborhood of $v$ and the distribution of the vertex representations in it 
If a graph-level feature vector is needed, e g , for classification or regression, a READOUT pooling function, that must be invariant to permutations, is applied:
Next, we present the MP network we developed for document understanding 
Message Passing Attention network for Document understanding (MPAD) ::: Word co-occurrence networks
We represent a document as a statistical word co-occurrence network BIBREF18, BIBREF19 with a sliding window of size 2 overspanning sentences  Let us denote that graph $G(V,E)$  Each unique word in the preprocessed document is represented by a node in $G$, and an edge is added between two nodes if they are found together in at least one instantiation of the window  $G$ is directed and weighted: edge directions and weights respectively capture text flow and co-occurrence counts 
$G$ is a compact representation of its document  In $G$, immediate neighbors are consecutive words in the same sentence  That is, paths of length 2 correspond to bigrams  Paths of length more than 2 can correspond either to traditional $n$-grams or to relaxed $n$-grams, that is, words that never appear in the same sentence but co-occur with the same word(s)  Such nodes are linked through common neighbors 
Master node  Inspired by BIBREF3, our $G$ also includes a special document node, linked to all other nodes via unit weight bi-directional edges  In what follows, let us denote by $n$ the number of nodes in $G$, including the master node 
Message Passing Attention network for Document understanding (MPAD) ::: Message passing
We formulate our AGGREGATE function as:
where $\mathbf {H}^t \in \mathbb {R}^{n \times d}$ contains node features ($d$ is a hyperparameter), and $\mathbf {A} \in \mathbb {R}^{n \times n}$ is the adjacency matrix of $G$  Since $G$ is directed, $\mathbf {A}$ is asymmetric  Also, $\mathbf {A}$ has zero diagonal as we choose not to consider the feature of the node itself, only that of its incoming neighbors, when updating its representation  Since $G$ is weighted, the $i^{th}$ row of $A$ contains the weights of the edges incoming on node $v_i$  $\mathbf {D} \in \mathbb {R}^{n \times n}$ is the diagonal in-degree matrix of $G$  MLP denotes a multi-layer perceptron, and $\mathbf {M}^{t+1} \in \mathbb {R}^{n \times d}$ is the message matrix 
The use of a MLP was motivated by the observation that for graph classification, MP neural nets with 1-layer perceptrons are inferior to their MLP counterparts BIBREF14  Indeed, 1-layer perceptrons are not universal approximators of multiset functions  Note that like in BIBREF14, we use a different MLP at each layer 
Renormalization  The rows of $\mathbf {D}^{-1}\mathbf {A}$ sum to 1  This is equivalent to the renormalization trick of BIBREF9, but using only the in-degrees  That is, instead of computing a weighted sum of the incoming neighbors' feature vectors, we compute a weighted average of them  The coefficients are proportional to the strength of co-occurrence between words  One should note that by averaging, we lose the ability to distinguish between different neighborhood structures in some special cases, that is, we lose injectivity  Such cases include neighborhoods in which all nodes have the same representations, and neighborhoods of different sizes containing various representations in equal proportions BIBREF14  As suggested by the results of an ablation experiment, averaging is better than summing in our application (see subsection SECREF30)  Note that instead of simply summing/averaging, we also tried using GAT-like attention BIBREF11 in early experiments, without obtaining better results 
As far as our COMBINE function, we use the Gated Recurrent Unit BIBREF20, BIBREF21:
Omitting biases for readability, we have:
where the $\mathbf {W}$ and $\mathbf {U}$ matrices are trainable weight matrices not shared across time steps, $\sigma (\mathbf {x}) = 1/(1+\exp (-\mathbf {x}))$ is the sigmoid function, and $\mathbf {R}$ and $\mathbf {Z}$ are the parameters of the reset and update gates  The reset gate controls the amount of information from the previous time step (in $\mathbf {H}^t$) that should propagate to the candidate representations, $\tilde{\mathbf {H}}^{t+1}$  The new representations $\mathbf {H}^{t+1}$ are finally obtained by linearly interpolating between the previous and the candidate ones, using the coefficients returned by the update gate 
Interpretation  Updating node representations through a GRU should in principle allow nodes to encode a combination of local and global signals (low and high values of $t$, resp ), by allowing them to remember about past iterations  In addition, we also explicitly consider node representations at all iterations when reading out (see Eq  DISPLAY_FORM18) 
Message Passing Attention network for Document understanding (MPAD) ::: Readout
After passing messages and performing updates for $T$ iterations, we obtain a matrix $\mathbf {H}^T \in \mathbb {R}^{n \times d}$ containing the final vertex representations  Let $\hat{G}$ be graph $G$ without the special document node, and matrix $\mathbf {\hat{H}}^T \in \mathbb {R}^{(n-1) \times d}$ be the corresponding representation matrix (i e , $\mathbf {H}^T$ without the row of the document node) 
We use as our READOUT function the concatenation of self-attention applied to $\mathbf {\hat{H}}^T$ with the final document node representation  More precisely, we apply a global self-attention mechanism BIBREF22 to the rows of $\mathbf {\hat{H}}^T$  As shown in Eq  DISPLAY_FORM17, $\mathbf {\hat{H}}^T$ is first passed to a dense layer parameterized by matrix $\mathbf {W}_A^T \in \mathbb {R}^{d \times d}$  An alignment vector $\mathbf {a}$ is then derived by comparing, via dot products, the rows of the output of the dense layer $\mathbf {Y}^T \in \mathbb {R}^{(n-1) \times d}$ with a trainable vector $\mathbf {v}^T \in \mathbb {R}^d$ (initialized randomly) and normalizing with a softmax  The normalized alignment coefficients are finally used to compute the attentional vector $\mathbf {u}^T \in \mathbb {R}^d$ as a weighted sum of the final representations $\mathbf {\hat{H}}^T$ 
Note that we tried with multiple context vectors, i e , with a matrix $\mathbf {V}^T$ instead of a vector $\mathbf {v}^T$, like in BIBREF22, but results were not convincing, even when adding a regularization term to the loss to favor diversity among the rows of $\mathbf {V}^T$ 
Master node skip connection  $\mathbf {h}_G^T \in \mathbb {R}^{2d}$ is obtained by concatenating $\mathbf {u}^T$ and the final master node representation  That is, the master node vector bypasses the attention mechanism  This is equivalent to a skip or shortcut connection BIBREF23  The reason behind this choice is that we expect the special document node to learn a high-level summary about the document, such as its size, vocabulary, etc  (more details are given in subsection SECREF30)  Therefore, by making the master node bypass the attention layer, we directly inject global information about the document into its final representation 
Multi-readout  BIBREF14, inspired by Jumping Knowledge Networks BIBREF12, recommend to not only use the final representations when performing readout, but also that of the earlier steps  Indeed, as one iterates, node features capture more and more global information  However, retaining more local, intermediary information might be useful too  Thus, instead of applying the readout function only to $t=T$, we apply it to all time steps and concatenate the results, finally obtaining $\mathbf {h}_G \in \mathbb {R}^{T \times 2d}$ :
In effect, with this modification, we take into account features based on information aggregated from subtrees of different heights (from 1 to $T$), corresponding to local and global features 
Message Passing Attention network for Document understanding (MPAD) ::: Hierarchical variants of MPAD
Through the successive MP iterations, it could be argued that MPAD implicitly captures some soft notion of the hierarchical structure of documents (words $\rightarrow $ bigrams $\rightarrow $ compositions of bigrams, etc )  However, it might be beneficial to explicitly capture document hierarchy  Hierarchical architectures have brought significant improvements to many NLP tasks, such as language modeling and generation BIBREF24, BIBREF25, sentiment and topic classification BIBREF26, BIBREF27, and spoken language understanding BIBREF28, BIBREF29  Inspired by this line of research, we propose several hierarchical variants of MPAD, detailed in what follows  In all of them, we represent each sentence in the document as a word co-occurrence network, and obtain an embedding for it by applying MPAD as previously described 
MPAD-sentence-att  Here, the sentence embeddings are simply combined through self-attention 
MPAD-clique  In this variant, we build a complete graph where each node represents a sentence  We then feed that graph to MPAD, where the feature vectors of the nodes are initialized with the sentence embeddings previously obtained 
MPAD-path  This variant is similar to the clique one, except that instead of a complete graph, we build a path according to the natural flow of the text  That is, two nodes are linked by a directed edge if the two sentences they represent follow each other in the document 
Experiments ::: Datasets
We evaluate the quality of the document embeddings learned by MPAD on 10 document classification datasets, covering the topic identification, coarse and fine sentiment analysis and opinion mining, and subjectivity detection tasks  We briefly introduce the datasets next  Their statistics are reported in Table TABREF21 
(1) Reuters  This dataset contains stories collected from the Reuters news agency in 1987  Following common practice, we used the ModApte split and considered only the 10 classes with the highest number of positive training examples  We also removed documents belonging to more than one class and then classes left with no document (2 classes) 
(2) BBCSport BIBREF30 contains documents from the BBC Sport website corresponding to 2004-2005 sports news articles 
(3) Polarity BIBREF31 features positive and negative labeled snippets from Rotten Tomatoes 
(4) Subjectivity BIBREF32 contains movie review snippets from Rotten Tomatoes (subjective sentences), and Internet Movie Database plot summaries (objective sentences) 
(5) MPQA BIBREF33 is made of positive and negative phrases, annotated as part of the summer 2002 NRRC Workshop on Multi-Perspective Question Answering 
(6) IMDB BIBREF34 is a collection of highly polarized movie reviews from IMDB (positive and negative)  There are at most 30 reviews for each movie 
(7) TREC BIBREF35 consists of questions that are classified into 6 different categories 
(8) SST-1 BIBREF36 contains the same snippets as Polarity  The authors used the Stanford Parser to parse the snippets and split them into multiple sentences  They then used Amazon Mechanical Turk to annotate the resulting phrases according to their polarity (very negative, negative, neutral, positive, very positive) 
(9) SST-2 BIBREF36 is the same as SST-1 but with neutral reviews removed and snippets classified as positive or negative 
(10) Yelp2013 BIBREF26 features reviews obtained from the 2013 Yelp Dataset Challenge 
Experiments ::: Baselines
We evaluate MPAD against multiple state-of-the-art baseline models, including hierarchical ones, to enable fair comparison with the hierarchical MPAD variants 
doc2vec BIBREF37  Doc2vec (or paragraph vector) is an extension of word2vec that learns vectors for documents in a fully unsupervised manner  Document embeddings are then fed to a logistic regression classifier 
CNN BIBREF38  The convolutional neural network architecture, well-known in computer vision, is applied to text  There is one spatial dimension and the word embeddings are used as channels (depth dimensions) 
DAN BIBREF39  The Deep Averaging Network passes the unweighted average of the embeddings of the input words through multiple dense layers and a final softmax 
Tree-LSTM BIBREF40 is a generalization of the standard LSTM architecture to constituency and dependency parse trees 
DRNN BIBREF41  Recursive neural networks are stacked and applied to parse trees 
LSTMN BIBREF42 is an extension of the LSTM model where the memory cell is replaced by a memory network which stores word representations 
C-LSTM BIBREF43 combines convolutional and recurrent neural networks  The region embeddings provided by a CNN are fed to a LSTM 
SPGK BIBREF44 also models documents as word co-occurrence networks  It computes a graph kernel that compares shortest paths extracted from the word co-occurrence networks and then uses a SVM to categorize documents 
WMD BIBREF45 is an application of the well-known Earth Mover's Distance to text  A k-nearest neighbor classifier is used 
S-WMD BIBREF46 is a supervised extension of the Word Mover's Distance 
Semantic-CNN BIBREF47  Here, a CNN is applied to semantic units obtained by clustering words in the embedding space 
LSTM-GRNN BIBREF26 is a hierarchical model where sentence embeddings are obtained with a CNN and a GRU-RNN is fed the sentence representations to obtain a document vector 
HN-ATT BIBREF27 is another hierarchical model, where the same encoder architecture (a bidirectional GRU-RNN) is used for both sentences and documents, with different parameters  A self-attention mechanism is applied to the RNN annotations at each level 
Experiments ::: Model configuration and training
We preprocess all datasets using the code of BIBREF38  On Yelp2013, we also replace all tokens appearing strictly less than 6 times with a special UNK token, like in BIBREF27  We then build a directed word co-occurrence network from each document, with a window of size 2 
We use two MP iterations ($T$=2) for the basic MPAD, and two MP iterations at each level, for the hierarchical variants  We set $d$ to 64, except on IMDB and Yelp on which $d=128$, and use a two-layer MLP  The final graph representations are passed through a softmax for classification  We train MPAD in an end-to-end fashion by minimizing the cross-entropy loss function with the Adam optimizer BIBREF48 and an initial learning rate of 0 001 
To regulate potential differences in magnitude, we apply batch normalization after concatenating the feature vector of the master node with the self-attentional vector, that is, after the skip connection (see subsection SECREF16)  To prevent overfitting, we use dropout BIBREF49 with a rate of 0 5  We select the best epoch, capped at 200, based on the validation accuracy  When cross-validation is used (see 3rd column of Table TABREF21), we construct a validation set by randomly sampling 10% of the training set of each fold 
On all datasets except Yelp2013, we use the publicly available 300-dimensional pre-trained Google News vectors ($D$=300) BIBREF50 to initialize the node representations $\mathbf {H}^0$  On Yelp2013, we follow BIBREF27 and learn our own word vectors from the training and validation sets with the gensim implementation of word2vec BIBREF51 
MPAD was implemented in Python 3 6 using the PyTorch library BIBREF52  All experiments were run on a single machine consisting of a 3 4 GHz Intel Core i7 CPU with 16 GB of RAM and an NVidia GeForce Titan Xp GPU 
Results and ablations ::: Results
Experimental results are shown in Table TABREF28  For the baselines, the best scores reported in each original paper are shown  MPAD reaches best performance on 7 out of 10 datasets, and is close second elsewhere  Moreover, the 7 datasets on which MPAD ranks first widely differ in training set size, number of categories, and prediction task (topic, sentiment, subjectivity), which indicates that MPAD can perform well in different settings 
MPAD vs  hierarchical variants  On 9 datasets out of 10, one or more of the hierarchical variants outperform the vanilla MPAD architecture, highlighting the benefit of explicitly modeling the hierarchical nature of documents 
However, on Subjectivity, standard MPAD outperforms all hierarchical variants  On TREC, it reaches the same accuracy  We hypothesize that in some cases, using a different graph to separately encode each sentence might be worse than using one single graph to directly encode the document  Indeed, in the single document graph, some words that never appear in the same sentence can be connected through common neighbors, as was explained in subsection SECREF7  So, this way, some notion of cross-sentence context is captured while learning representations of words, bigrams, etc  at each MP iteration  This creates better informed representations, resulting in a better document embedding  With the hierarchical variants, on the other hand, each sentence vector is produced in isolation, without any contextual information about the other sentences in the document  Therefore, the final sentence embeddings might be of lower quality, and as a group might also contain redundant/repeated information  When the sentence vectors are finally combined into a document representation, it is too late to take context into account 
Results and ablations ::: Ablation studies
To understand the impact of some hyperparameters on performance, we conducted additional experiments on the Reuters, Polarity, and IMDB datasets, with the non-hierarchical version of MPAD  Results are shown in Table TABREF29 
Number of MP iterations  First, we varied the number of message passing iterations from 1 to 4  We can clearly see in Table TABREF29 that having more iterations improves performance  We attribute this to the fact that we are reading out at each iteration from 1 to $T$ (see Eq  DISPLAY_FORM18), which enables the final graph representation to encode a mixture of low-level and high-level features  Indeed, in initial experiments involving readout at $t$=$T$ only, setting $T\ge 2$ was always decreasing performance, despite the GRU-based updates (Eq  DISPLAY_FORM14)  These results were consistent with that of BIBREF53 and BIBREF9, who both are reading out only at $t$=$T$ too  We hypothesize that node features at $T\ge 2$ are too diffuse to be entirely relied upon during readout  More precisely, initially at $t$=0, node representations capture information about words, at $t$=1, about their 1-hop neighborhood (bigrams), at $t$=2, about compositions of bigrams, etc  Thus, pretty quickly, node features become general and diffuse  In such cases, considering also the lower-level, more precise features of the earlier iterations when reading out may be necessary 
Undirected edges  On Reuters, using an undirected graph leads to better performance, while on Polarity and IMDB, it is the opposite  This can be explained by the fact that Reuters is a topic classification task, for which the presence or absence of some patterns is important, but not necessarily the order in which they appear, while Polarity and IMDB are sentiment analysis tasks  To capture sentiment, modeling word order is crucial, e g , in detecting negation 
No master node  Removing the master node deteriorates performance across all datasets, clearly showing the value of having such a node  We hypothesize that since the special document node is connected to all other nodes, it is able to encode during message passing a summary of the document 
No renormalization  Here, we do not use the renormalization trick of BIBREF9 during MP (see subsection SECREF10)  That is, Eq  DISPLAY_FORM11 becomes $\mathbf {M}^{t+1} = \textsc {MLP}^{t+1}\big (\mathbf {A}\mathbf {H}^{t}\big )$  In other words, instead of computing a weighted average of the incoming neighbors' feature vectors, we compute a weighted sum of them  Unlike the mean, which captures distributions, the sum captures structural information BIBREF14  As shown in Table TABREF29, using sum instead of mean decreases performance everywhere, suggesting that in our application, capturing the distribution of neighbor representations is more important that capturing their structure  We hypothesize that this is the case because statistical word co-occurrence networks tend to have similar structural properties, regardless of the topic, polarity, sentiment, etc  of the corresponding documents 
Neighbors-only  In this experiment, we replaced the GRU combine function (see Eq  DISPLAY_FORM14) with the identity function  That is, we simply have $\mathbf {H}^{t+1}$=$\mathbf {M}^{t+1}$  Since $\mathbf {A}$ has zero diagonal, by doing so, we completely ignore the previous feature of the node itself when updating its representation  That is, the update is based entirely on its neighbors  Except on Reuters (almost no change), performance always suffers, stressing the need to take into account the root node during updates, not only its neighborhood 
Related work
In what follows, we offer a brief review of relevant studies, ranked by increasing order of similarity with our work 
BIBREF9, BIBREF54, BIBREF11, BIBREF10 conduct some node classification experiments on citation networks, where nodes are scientific papers, i e , textual data  However, text is only used to derive node feature vectors  The external graph structure, which plays a central role in determining node labels, is completely unrelated to text 
On the other hand, BIBREF55, BIBREF7 experiment on traditional document classification tasks  They both build $k$-nearest neighbor similarity graphs based on the Gaussian diffusion kernel  More precisely, BIBREF55 build one single graph where nodes are documents and distance is computed in the BoW space  Node features are then used for classification  Closer to our work, BIBREF7 represent each document as a graph  All document graphs are derived from the same underlying structure  Only node features, corresponding to the entries of the documents' BoW vectors, vary  The underlying, shared structure is that of a $k$-NN graph where nodes are vocabulary terms and similarity is the cosine of the word embedding vectors  BIBREF7 then perform graph classification  However they found performance to be lower than that of a naive Bayes classifier 
BIBREF56 use a GNN for hierarchical classification into a large taxonomy of topics  This task differs from traditional document classification  The authors represent documents as unweighted, undirected word co-occurrence networks with word embeddings as node features  They then use the spatial GNN of BIBREF15 to perform graph classification 
The work closest to ours is probably that of BIBREF53  The authors adopt the semi-supervised node classification approach of BIBREF9  They build one single undirected graph from the entire dataset, with both word and document nodes  Document-word edges are weighted by TF-IDF and word-word edges are weighted by pointwise mutual information derived from co-occurrence within a sliding window  There are no document-document edges  The GNN is trained based on the cross-entropy loss computed only for the labeled nodes, that is, the documents in the training set  When the final node representations are obtained, one can use that of the test documents to classify them and evaluate prediction performance 
There are significant differences between BIBREF53 and our work  First, our approach is inductive, not transductive  Indeed, while the node classification approach of BIBREF53 requires all test documents at training time, our graph classification model is able to perform inference on new, never-seen documents  The downside of representing documents as separate graphs, however, is that we lose the ability to capture corpus-level dependencies  Also, our directed graphs capture word ordering, which is ignored by BIBREF53  Finally, the approach of BIBREF53 requires computing the PMI for every word pair in the vocabulary, which may be prohibitive on datasets with very large vocabularies  On the other hand, the complexity of MPAD does not depend on vocabulary size 
Conclusion
We have proposed an application of the message passing framework to NLP, the Message Passing Attention network for Document understanding (MPAD)  Experiments conducted on 10 standard text classification datasets show that our architecture is competitive with the state-of-the-art  By processing weighted, directed word co-occurrence networks, MPAD is sensitive to word order and word-word relationship strength  To explicitly capture the hierarchical structure of documents, we also propose three hierarchical variants of MPAD, that we show bring improvements over the vanilla architecture 
Acknowledgments
We thank the NVidia corporation for the donation of a GPU as part of their GPU grant program 

ground_truth: Unanswerable

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which component is the least impactful?
Context 4:  FIGREF4 shows the distributions of seven question types grouped deterministically from the lexicons  Although these corpora have been independently developed, a general trend is found, where the what question type dominates, followed by how and who, followed by when and where, and so on 
Fig  FIGREF6 shows the distributions of answer categories automatically classified by our Convolutional Neural Network model trained on the data distributed by li:02a  Interestingly, each corpus focuses on different categories, Numeric for WikiQA and SelQA, Entity for SQuAD, and Person for InfoboxQA, which gives enough diversities for statistical learning to build robust models 
Answer Retrieval
This section describes another selection-based QA task, called answer retrieval, that finds the answer context from a larger dataset, the entire Wikipedia  SQuAD provides no mapping of the answer contexts to Wikipedia, whereas WikiQA and SelQA provide mappings; however, their data do not come from the same version of Wikipedia  We propose an automatic way of mapping the answer contexts from all corpora to the same version of Wikipeda so they can be coherently used for answer retrieval 
Each paragraph in Wikipedia is first indexed by Lucene using {1,2,3}-grams, where the paragraphs are separated by WikiExtractor and segmented by NLP4J (28 7M+ paragraphs are indexed)  Each answer sentence from the corpora in Table TABREF3 is then queried to Lucene, and the top-5 ranked paragraphs are retrieved  The cosine similarity between each sentence in these paragraphs and the answer sentence is measured for INLINEFORM0 -grams, say INLINEFORM1   A weight is assigned to each INLINEFORM2 -gram score, say INLINEFORM3 , and the weighted sum is measured: INLINEFORM4   The fixed weights of INLINEFORM5 are used for our experiments, which can be improved 
If there exists a sentence whose INLINEFORM0 , the paragraph consisting of that sentence is considered the silver-standard answer passage  Table TABREF3 shows how robust these silver-standard passages are based on human judgement ( INLINEFORM1 ) and how many passages are collected ( INLINEFORM2 ) for INLINEFORM3 , where the human judgement is performed on 50 random samples for each case  For answer retrieval, a dataset is created by INLINEFORM4 , which gives INLINEFORM5 accuracy and INLINEFORM6 coverage, respectively  Finally, each question is queried to Lucene and the top- INLINEFORM7 paragraphs are retrieved from the entire Wikipedia  If the answer sentence exists within those retrieved paragraphs according to the silver-standard, it is considered correct 
Finding a paragraph that includes the answer context out of the entire Wikipedia is an extremely difficult task (128 7M)  The last row of Table TABREF3 shows results from answer retrieval  Given INLINEFORM0 , SelQA and SQuAD show about 34% and 35% accuracy, which are reasonable  However, WikiQA shows a significantly lower accuracy of 12 47%; this is because the questions in WikiQA is about twice shorter than the questions in the other corpora such that not enough lexicons can be extracted from these questions for the Lucene search 
Answer Selection
Answer selection is evaluated by two metrics, mean average precision (MAP) and mean reciprocal rank (MRR)  The bigram CNN introduced by yu:14a is used to generate all the results in Table TABREF11 , where models are trained on either single or combined datasets  Clearly, the questions in WikiQA are the most challenging, and adding more training data from the other corpora hurts accuracy due to the uniqueness of query-based questions in this corpus  The best model is achieved by training on W+S+Q for SelQA; adding InfoboxQA hurts accuracy for SelQA although it gives a marginal gain for SQuAD  Just like WikiQA, InfoboxQA performs the best when it is trained on only itself  From our analysis, we suggest that to use models trained on WikiQA and InfoboxQA for short query-like questions, whereas to use ones trained on SelQA and SQuAD for long natural questions 
Answer Triggering
The results of INLINEFORM0 from the answer retrieval task in Section SECREF13 are used to create the datasets for answer triggering, where about 65% of the questions are not expected to find their answer contexts from the provided paragraphs for SelQA and SQuAD and 87 5% are not expected for WikiQA  Answer triggering is evaluated by the F1 scores as presented in Table TABREF11 , where three corpora are cross validated  The results on WikiQA are pretty low as expected from the poor accuracy on the answer retrieval task  Training on SelQA gives the best models for both WikiQA and SelQA  Training on SQuAD gives the best model for SQuAD although the model trained on SelQA is comparable  Since the answer triggering datasets are about 5 times larger than the answer selection datasets, it is computationally too expensive to combine all data for training  We plan to find a strong machine to perform this experiment in near future 
Related work
Lately, several deep learning approaches have been proposed for question answering  yu:14a presented a CNN model that recognizes the semantic similarity between two sentences  wang-nyberg:2015:ACL-IJCNLP presented a stacked bidirectional LSTM approach to read words in sequence, then outputs their similarity scores  feng:15a applied a general deep learning framework to non-factoid question answering  santos:16a introduced an attentive pooling mechanism that led to further improvements in selection-based QA 
Conclusion
We present a comprehensive comparison study of the existing corpora for selection-based question answering  Our intrinsic analysis provides a better understanding of the uniqueness or similarity between these corpora  Our extrinsic analysis shows the strength or weakness of combining these corpora together for statistical learning  Additionally, we create a silver-standard dataset for answer retrieval and triggering, which will be publicly available  In the future, we will explore different ways of improving the quality of our silver-standard datasets by fine-tuning the hyper-parameters 

ground_truth: Based on table results provided changing directed to undirected edges had least impact - max abs difference of 0.33 points on all three datasets.