prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. which languages are explored?
Context 1: Introduction
Part-of-speech tagging is now a classic task in natural language processing, for which many systems have been developed or adapted for a large variety of languages  Its aim is to associate each “word” with a morphosyntactic tag, whose granularity can range from a simple morphosyntactic category, or part-of-speech (hereafter PoS), to finer categories enriched with morphological features (gender, number, case, tense, mood, etc ) 
The use of machine learning algorithms trained on manually annotated corpora has long become the standard way to develop PoS taggers  A large variety of algorithms have been used, such as (in approximative chronological order) bigram and trigram hidden Markov models BIBREF0 , BIBREF1 , BIBREF2 , decision trees BIBREF3 , BIBREF4 , maximum entropy Markov models (MEMMs) BIBREF5 and Conditional Random Fields (CRFs) BIBREF6 , BIBREF7   With such machine learning algorithms, it is possible to build PoS taggers for any language, provided adequate training data is available 
As a complement to annotated corpora, it has previously been shown that external lexicons are valuable sources of information, in particular morphosyntactic lexicons, which provide a large inventory of (word, PoS) pairs  Such lexical information can be used in the form of constraints at tagging time BIBREF8 , BIBREF9 or during the training process as additional features combined with standard features extracted from the training corpus BIBREF10 , BIBREF11 , BIBREF12  
In recent years, a different approach to modelling lexical information and integrating it into natural language processing systems has emerged, namely the use of vector representations for words or word sequences BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18   Such representations, which are generally extracted from large amounts of raw text, have proved very useful for numerous tasks including PoS tagging, in particular when used in recurrent neural networks (RNNs) and more specifically in mono- or bi-directional, word-level and/or character-level long short-term memory networks (LSTMs) BIBREF19 , BIBREF16 , BIBREF17 , BIBREF20  
Both approaches to representing lexical properties and to integrating them into a PoS tagger improve tagging results  Yet they rely on resources of different natures  The main advantage of word vectors is that they are built in an unsupervised way, only requiring large amounts of raw textual data  They also encode finer-grained information than usual morphosyntactic lexicons, most of which do not include any quantitative data, not even simple frequency information  Conversely, lexical resources often provide information about scarcely attested words, for which corpus-based approaches such as word vector representations are of limited relevance  Moreover, morphological or morphosyntactic lexicons already exist for a number of languages, including less-resourced langauges for which it might be difficult to obtain the large amounts of raw data necessary to extract word vector representations 
Our main goal is therefore to compare the respective impact of external lexicons and word vector representations on the accuracy of PoS models  This question has already been investigated for 6 languages by BIBREF18 using the state-of-the-art CRF-based tagging system MarMoT  The authors found that their best-performing word-vector-based PoS tagging models outperform their models that rely on morphosyntactic resources (lexicons or morphological analysers)  In this paper, we report on larger comparison, carried out in a larger multilingual setting and comparing different tagging models  Using different 16 datasets, we compare the performances of two feature-based models enriched with external lexicons and of two LSTM-based models enriched with word vector representations  A secondary goal of our work is to compare the relative improvements linked to the use of external lexical information in the two feature-based models, which use different models (MEMM vs  CRF) and feature sets 
More specifically, our starting point is the MElt system BIBREF12 , an MEMM tagging system  We first briefly describe this system and the way we adapted it by integrating our own set of corpus-based and lexical features  We then introduce the tagging models we have trained for 16 different languages using our adapted version of MElt  These models are trained on the Universal Dependencies (v1 2) corpus set BIBREF21 , complemented by morphosyntactic lexicons  We compare the accuracy of our models with the scores obtained by the CRF-based system MarMoT BIBREF22 , BIBREF18 , retrained on the same corpora and the same external morphosyntactic lexicons  We also compare our results to those obtained by the best bidirectional LSTM models described by BIBREF20 , which both make use of Polyglot word vector representations published by BIBREF23   We will show that an optimised enrichment of feature-based models with morphosyntactic lexicon results in significant accuracy gains  The macro-averaged accuracy of our enriched MElt models is above that of enriched MarMoT models and virtually identical to that of LSTMs enriched with word vector representations  More precisely, per-language results indicate that lexicons provide more useful information for languages with a high lexical variability (such as morphologically rich languages), whereas word vectors are more informative for languages with a lower lexical variability (such as English) 
MElt
MElt BIBREF12 is a tagging system based on maximum entropy Markov models (MEMM) BIBREF5 , a class of discriminative models that are suitable for sequence labelling BIBREF5   The basic set of features used by MElt is given in BIBREF12   It is a superset of the feature sets used by BIBREF5 and BIBREF24 and includes both local standard features (for example the current word itself and its prefixes and suffixes of length 1 to 4) and contextual standard features (for example the tag just assigned to the preceding word)  In particular, with respect to Ratnaparkhi's feature set, MElt's basic feature set lifts the restriction that local standard features used to analyse the internal composition of the current word should only apply to rare words 
One of the advantages of feature-based models such as MEMMs and CRFs is that complementary information can be easily added in the form of additional features  This was investigated for instance by BIBREF25 , whose best-performing model for PoS tagging dialogues was obtained with a version of MElt extended with dialogue-specific features  Yet the motivation of MElt's developers was first and foremost to investigate the best way to integrate lexical information extracted from large-scale morphosyntactic lexical resources into their models, on top of the training data BIBREF12   They showed that performances are better when this external lexical information is integrated in the form of additional lexical features than when the external lexicon is used as constraints at tagging time  These lexical features can also be divided into local lexical features (for example the list of possible tags known to the external lexicon for the current word) and contextual lexical features (for example the list of possible tags known to the external lexicon for surrounding words)  In particular, lexical contextual features provide a means to model the right context of the current word, made of words that have not yet been tagged by the system but for which the lexicon often provides a list of possible tags  Moreover, tagging accuracy for out-of-vocabulary (OOV) words is improved, as a result of the fact that words unknown to the training corpus might be known to the external lexicon 
Despite a few experiments published with MElt on languages other than French BIBREF12 , BIBREF40 , BIBREF41 , the original feature set used by MElt (standard and lexical features) was designed and tested mostly on this language, by building and evaluating tagging models on a variant of the French TreeBank  Since our goal was to carry out experiments in a multilingual setting, we have decided to design our own set of features, using the standard MElt features as a starting point  With respect to the original MElt feature set, we have added new ones, such as prefixes and suffixes of the following word, as well as a hybrid contextual feature obtained by concatenating the tag predicted for the preceding word and the tag(s) provided by the external lexicon for the following word 
In order to select the best performing feature set, we carried out a series of experiments using the multilingual dataset provided during the SPMRL parsing shared task BIBREF42   This included discarding useless or harmful features and selecting the maximal length of the prefixes and suffixes to be used as features, both for the current word and for the following word 
We incorporated in MElt the best performing feature set, described in Table TABREF1   All models discussed in this paper are based on this feature set 
Corpora
We carried out our experiments on the Universal Dependencies v1 2 treebanks BIBREF21 , hereafter UD1 2, from which morphosyntactically annotated corpora can be trivially extracted  All UD1 2 corpora use a common tag set, the 17 universal PoS tags, which is an extension of the tagset proposed by BIBREF43  
As our goal is to study the impact of lexical information for PoS tagging, we have restricted our experiments to UD1 2 corpora that cover languages for which we have morphosyntactic lexicons at our disposal, and for which BIBREF20 provide results  We considered UD1 2 corpora for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish  Although this language list contains only one non-Indo-European (Indonesian), four major Indo-European sub-families are represented (Germanic, Romance, Slavic, Indo-Iranian)  Overall, the 16 languages considered in our experiments are typologically, morphologically and syntactically fairly diverse 
Lexicons
We generate our external lexicons using the set of source lexicons listed in Table TABREF3   Since external lexical information is exploited via features, there is no need for the external lexicons and the annotated corpora to use the same PoS inventory  Therefore, for each language, we simply extracted from the corresponding lexicon the PoS of each word based on its morphological tags, by removing all information provided except for its coarsest-level category  We also added entries for punctuations when the source lexicons did not contain any 
We also performed experiments in which we retained the full original tags provided by the lexicons, with all morphological features included  On average, results were slightly better than those presented in the paper, although not statistically significantly  Moreover, the granularity of tag inventories in the lexicons is diverse, which makes it difficult to draw general conclusions about results based on full tags  This is why we only report results based on (coarse) PoS extracted from the original lexicons 
Baseline models
In order to assess the respective contributions of external lexicons and word vector representations, we first compared the results of the three above-mentioned systems when trained without such additional lexical information  Table TABREF11 provides the results of MElt and MarMoT retrained on UD1 2 corpora, together with the results publised on the same corpora by BIBREF20 , using their best model not enhanced by external word vector representations —i e  the model they call INLINEFORM0 , which is a bidirectional LSTM that combines both word and character embeddings 
These results show that Plank et al 's (2016) bi-LSTM performs extremely well, surpassed by MarMoT on only 3 out of 16 datasets (Czech, French and Italian), and by MElt only once (Indonesian) 
Models enriched with external lexical information
Table TABREF13 provides the results of four systems enriched with lexical information  The feature-based systems MElt and MarMoT, respectively based on MEMMs and CRFs, are extended with the lexical information provided by our morphosyntactic lexicons  This extension takes the form of additional features, as described in Section SECREF2 for MElt  The results reported by BIBREF20 for their bidirectional LSTM when initialised with Polyglot embeddings trained on full wikipedias are also included, together with their new system FREQBIN, also initialised with Polyglot embeddings  FREQBIN trains bi-LSTMs to predict for each input word both a PoS and a label that represents its log frequency in the training data  As they word it, “the idea behind this model is to make the representation predictive for frequency, which encourages the model not to share representations between common and rare words, thus benefiting the handling of rare tokens ”
The results, which are also displayed in Figures FIGREF14 and FIGREF15 , show that all systems reach very similar results on average, although discrepancies can be observed from one dataset to another, on which we shall comment shortly  The best performing system in terms of macro-average is MElt (96 60%)  Both bi-LSTM systems reach the same score (96 58%), the difference with MElt's results being non significant, whereas MarMoT is only 0 14% behind (96 46%)  Given the better baseline scores of the neural approaches, these results show that the benefit of using external lexicons in the feature-based models MElt and MarMoT are much higher than those using Polyglot word vector representations as initialisations for bi-LSTMs 
Yet these very similar overall results reflect a different picture when focusing on OOV tagging accuracy  The best models for OOV tagging accuracy are, by far, FREQBIN models, which are beaten by MarMoT and by MElt only once each (on English and Danish respectively)  The comparison on OOV tagging between MElt and MarMoT shows that MElt performs better on average than MarMoT, despite the fact that MarMoT's baseline results were better than those reached by MElt  This shows that the information provided by external morphosyntactic lexicons is better exploited by MElt's lexical features than by those used by MarMoT  On the other hand, the comparison of both bi-LSTM-based approaches confirm that the FREQBIN models is better by over 10% absolute on OOV tagging accuracy (94 28% vs  83 59%), with 65% lower error rate 
One of the important differences between the lexical information provided by an external lexicon and word vectors built from raw corpora, apart from the very nature of the lexical information provided, is the coverage and accuracy of this lexical information on rare words  All words in a morphosyntactic lexicon are associated with information of a same granularity and quality, which is not the case with word representations such as provided by Polyglot  Models that take advantage of external lexicons should therefore perform comparatively better on datasets containing a higher proportion of rarer words, provided the lexicons' coverage is high  In order to confirm this intuition, we have used a lexical richness metric based on the type/token ratio  Since this ratio is well-known for being sensitive to corpus length, we normalised it by computing it over the 60,000 first tokens of each training set  When this normalised type/token ratio is plotted against the difference between the results of MElt and both bi-LSTM-based models, the expected correlation is clearly visible (see Figure FIGREF16 )  This explains why MElt obtains better results on the morphologically richer Slavic datasets (average normalised type/token ratio: 0 28, average accuracy difference: 0 32 compared to both bi-LSTM+Polyglot and FREQBIN+Polyglot) and, at the other end of the spectrum, significantly worse results on the English dataset (normalised type/token ratio: 0 15, average accuracy difference: -0 56 compared to bi-LSTM+Polyglot, -0 57 compared to FREQBIN+Polyglot) 
Conclusion
Two main conclusions can be drawn from our comparative results  First, feature-based tagging models adequately enriched with external morphosyntactic lexicons perform, on average, as well as bi-LSTMs enriched with word embeddings  Per-language results show that the best accuracy levels are reached by feature-based models, and in particular by our improved version of the MEMM-based system MElt, on datasets with high lexical variability (in short, for morphologically rich languages), whereas neural-based results perform better on datatsets with lower lexical variability (e g  for English) 
We have only compared the contribution of morphosyntactic lexicons to feature-based models (MEMMs, CRFs) and that of word vector representations to bi-LSTM-based models as reported by BIBREF20   As mentioned above, work on the contribution of word vector representations to feature-based approaches has been carried out by BIBREF18   However, the exploitation of existing morphosyntactic or morphological lexicons in neural models is a less studied question  Improvements over the state of the art might be achieved by integrating lexical information both from an external lexicon and from word vector representations into tagging models 
In that regard, further work will be required to understand which class of models perform the best  An option would be to integrate feature-based models such as a CRF with an LSTM-based layer, following recent proposals such as the one proposed by BIBREF45 for named entity recognition 
Context 2: Introduction
Speech-to-Text translation (ST) is essential for a wide range of scenarios: for example in emergency calls, where agents have to respond emergent requests in a foreign language BIBREF0; or in online courses, where audiences and speakers use different languages BIBREF1  To tackle this problem, existing approaches can be categorized into cascaded method BIBREF2, BIBREF3, where a machine translation (MT) model translates outputs of an automatic speech recognition (ASR) system into target language, and end-to-end method BIBREF4, BIBREF5, where a single model learns acoustic frames to target word sequence mappings in one step towards the final objective of interest  Although the cascaded model remains the dominant approach due to its better performance, the end-to-end method becomes more and more popular because it has lower latency by avoiding inferences with two models and rectifies the error propagation in theory 
Since it is hard to obtain a large-scale ST dataset, multi-task learning BIBREF5, BIBREF6 and pre-training techniques BIBREF7 have been applied to end-to-end ST model to leverage large-scale datasets of ASR and MT  A common practice is to pre-train two encoder-decoder models for ASR and MT respectively, and then initialize the ST model with the encoder of the ASR model and the decoder of the MT model  Subsequently, the ST model is optimized with the multi-task learning by weighing the losses of ASR, MT, and ST  This approach, however, causes a huge gap between pre-training and fine-tuning, which are summarized into three folds:
Subnet Waste: The ST system just reuses the ASR encoder and the MT decoder, while discards other pre-trained subnets, such as the MT encoder  Consequently, valuable semantic information captured by the MT encoder cannot be inherited by the final ST system 
Role Mismatch: The speech encoder plays different roles in pre-training and fine-tuning  The encoder is a pure acoustic model in pre-training, while it has to extract semantic and linguistic features additionally in fine-tuning, which significantly increases the learning difficulty 
Non-pre-trained Attention Module: Previous work BIBREF6 trains attention modules for ASR, MT and ST respectively, hence, the attention module of ST does not benefit from the pre-training 
To address these issues, we propose a Tandem Connectionist Encoding Network (TCEN), which is able to reuse all subnets in pre-training, keep the roles of subnets consistent, and pre-train the attention module  Concretely, the TCEN consists of three components, a speech encoder, a text encoder, and a target text decoder  Different from the previous work that pre-trains an encoder-decoder based ASR model, we only pre-train an ASR encoder by optimizing the Connectionist Temporal Classification (CTC) BIBREF8 objective function  In this way, the additional decoder of ASR is not required while keeping the ability to read acoustic features into the source language space by the speech encoder  Besides, the text encoder and decoder can be pre-trained on a large MT dataset  After that, we employ common used multi-task learning method to jointly learn ASR, MT and ST tasks 
Compared to prior works, the encoder of TCEN is a concatenation of an ASR encoder and an MT encoder and our model does not have an ASR decoder, so the subnet waste issue is solved  Furthermore, the two encoders work at tandem, disentangling acoustic feature extraction and linguistic feature extraction, ensuring the role consistency between pre-training and fine-tuning  Moreover, we reuse the pre-trained MT attention module in ST, so we can leverage the alignment information learned in pre-training 
Since the text encoder consumes word embeddings of plausible texts in MT task but uses speech encoder outputs in ST task, another question is how one guarantees the speech encoder outputs are consistent with the word embeddings  We further modify our model to achieve semantic consistency and length consistency  Specifically, (1) the projection matrix at the CTC classification layer for ASR is shared with the word embedding matrix, ensuring that they are mapped to the same latent space, and (2) the length of the speech encoder output is proportional to the length of the input frame, so it is much longer than a natural sentence  To bridge the length gap, source sentences in MT are lengthened by adding word repetitions and blank tokens to mimic the CTC output sequences 
We conduct comprehensive experiments on the IWSLT18 speech translation benchmark BIBREF1, demonstrating the effectiveness of each component  Our model is significantly better than previous methods by 3 6 and 2 2 BLEU scores for the subword-level decoding and character-level decoding strategies, respectively 
Our contributions are three-folds: 1) we shed light on why previous ST models cannot sufficiently utilize the knowledge learned from the pre-training process; 2) we propose a new ST model, which alleviates shortcomings in existing methods; and 3) we empirically evaluate the proposed model on a large-scale public dataset 
Background ::: Problem Formulation
End-to-end speech translation aims to translate a piece of audio into a target-language translation in one step  The raw speech signals are usually converted to sequences of acoustic features, e g  Mel filterbank features  Here, we define the speech feature sequence as $\mathbf {x} = (x_1, \cdots , x_{T_x})$ The transcription and translation sequences are denoted as $\mathbf {y^{s}} = (y_1^{s}, \cdots , y_{T_s}^{s})$, and $\mathbf {y^{t}} = (y_1^{t}, \cdots , y_{T_t}^{t})$ repectively  Each symbol in $\mathbf {y^{s}}$ or $\mathbf {y^{t}}$ is an integer index of the symbol in a vocabulary $V_{src}$ or $V_{trg}$ respectively (e g  $y^s_i=k, k\in [0, |V_{src}|-1]$)  In this work, we suppose that an ASR dataset, an MT dataset, and a ST dataset are available, denoted as $\mathcal {A} = \lbrace (\mathbf {x_i}, \mathbf {y^{s}_i})\rbrace _{i=0}^I$, $\mathcal {M} =\lbrace (\mathbf {y^{s}_j}, \mathbf {y^{t}_j})\rbrace _{j=0}^J$ and $ \mathcal {S} =\lbrace (\mathbf {x_l}, \mathbf {y^{t}_l})\rbrace _{l=0}^L$ respectively  Given a new piece of audio $\mathbf {x}$, our goal is to learn an end to end model to generate a translation sentence $\mathbf {y^{t}}$ without generating an intermediate result $\mathbf {y^{s}}$ 
Background ::: Multi-Task Learning and Pre-training for ST
To leverage large scale ASR and MT data, multi-task learning and pre-training techniques are widely employed to improve the ST system  As shown in Figure FIGREF4, there are three popular multi-task strategies for ST, including 1) one-to-many setting, in which a speech encoder is shared between ASR and ST tasks; 2) many-to-one setting in which a decoder is shared between MT and ST tasks; and 3) many-to-many setting where both the encoder and decoder are shared 
A many-to-many multi-task model contains two encoders as well as two decoders  It can be jointly trained on ASR, MT, and ST tasks  As the attention module is task-specific, three attentions are defined 
Usually, the size of $\mathcal {A}$ and $\mathcal {M}$ is much larger than $\mathcal {S}$  Therefore, the common training practice is to pre-train the model on ASR and MT tasks and then fine-tune it with a multi-task learning manner  However, as aforementioned, this method suffers from subnet waste, role mismatch and non-pre-trained attention issues, which severely limits the end-to-end ST performance 
Our method
In this section, we first introduce the architecture of TCEN, which consists of two encoders connected in tandem, and one decoder with an attention module  Then we give the pre-training and fine-tuning strategy for TCEN  Finally, we propose our solutions for semantic and length inconsistency problems, which are caused by multi-task learning 
Our method ::: TCEN Architecture
Figure FIGREF5 sketches the overall architecture of TCEN, including a speech encoder $enc_s$, a text encoder $enc_t$ and a decoder $dec$ with an attention module $att$  During training, the $enc_s$ acts like an acoustic model which reads the input $\mathbf {x}$ to word or subword representations $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into hidden representations $\mathbf {h^t}$  Finally, the $dec$ defines a distribution probability over target words  The advantage of our architecture is that two encoders disentangle acoustic feature extraction and linguistic feature extraction, making sure that valuable knowledge learned from ASR and MT tasks can be effectively leveraged for ST training  Besides, every module in pre-training can be utilized in fine-tuning, alleviating the subnet waste problem 
Follow BIBREF9 inaguma2018speech, we use CNN-BiLSTM architecture to build our model  Specifically, the input features $\mathbf {x}$ are organized as a sequence of feature vectors in length $T_x$  Then, $\mathbf {x}$ is passed into a stack of two convolutional layers followed by max-pooling:
where $\mathbf {v}^{(l-1)}$ is feature maps in last layer and $\mathbf {W}^{(l)}$ is the filter  The max-pooling layers downsample the sequence in length by a total factor of four  The down-sampled feature sequence is further fed into a stack of five bidirectional $d$-dimensional LSTM layers:
where $[;]$ denotes the vector concatenation  The final output representation from the speech encoder is denoted as $\mathbf {h^s}=(h^s_1, \cdots , h^s_{\frac{T_x}{4}})$, where $h_i^s \in \mathbb {R}^d$ 
The text encoder $enc_t$ consists of two bidirectional LSTM layers  In ST task, $enc_t$ accepts speech encoder output $\mathbf {h}^s$ as input  While in MT, $enc_t$ consumes the word embedding representation $\mathbf {e^s}$ derived from $\mathbf {y^s}$, where each element $e^s_i$ is computed by choosing the $y_i^s$-th vector from the source embedding matrix $W_{E^s}$  The goal of $enc_t$ is to extract high-level linguistic features like syntactic features or semantic features from lower level subword representations $\mathbf {h}^s$ or $\mathbf {e}^s$  Since $\mathbf {h}^s$ and $\mathbf {e}^s$ belong to different latent space and have different lengths, there remain semantic and length inconsistency problems  We will provide our solutions in Section SECREF21  The output sequence of $enc_t$ is denoted as $\mathbf {h}^t$ 
The decoder is defined as two unidirectional LSTM layers with an additive attention $att$  It predicts target sequence $\mathbf {y^{t}}$ by estimating conditional probability $P(\mathbf {y^{t}}|\mathbf {x})$:
Here, $z_k$ is the the hidden state of the deocder RNN at $k$ step and $c_k$ is a time-dependent context vector computed by the attention $att$ 
Our method ::: Training Procedure
Following previous work, we split the training procedure to pre-training and fine-tuning stages  In pre-training stage, the speech encoder $enc_s$ is trained towards CTC objective using dataset $\mathcal {A}$, while the text encoder $enc_t$ and the decoder $dec$ are trained on MT dataset $\mathcal {M}$  In fine-tuning stage, we jointly train the model on ASR, MT, and ST tasks 
Our method ::: Training Procedure ::: Pre-training
To sufficiently utilize the large dataset $\mathcal {A}$ and $\mathcal {M}$, the model is pre-trained on CTC-based ASR task and MT task in the pre-training stage 
For ASR task, in order to get rid of the requirement for decoder and enable the $enc_s$ to generate subword representation, we leverage connectionist temporal classification (CTC) BIBREF8 loss to train the speech encoder 
Given an input $\mathbf {x}$, $enc_s$ emits a sequence of hidden vectors $\mathbf {h^s}$, then a softmax classification layer predicts a CTC path $\mathbf {\pi }$, where $\pi _t \in V_{src} \cup $ {`-'} is the observing label at particular RNN step $t$, and `-' is the blank token representing no observed labels:
where $W_{ctc} \in \mathbb {R}^{d \times (|V_{src}|+1)}$ is the weight matrix in the classification layer and $T$ is the total length of encoder RNN 
A legal CTC path $\mathbf {\pi }$ is a variation of the source transcription $\mathbf {y}^s$ by allowing occurrences of blank tokens and repetitions, as shown in Table TABREF14  For each transcription $\mathbf {y}$, there exist many legal CTC paths in length $T$  The CTC objective trains the model to maximize the probability of observing the golden sequence $\mathbf {y}^s$, which is calculated by summing the probabilities of all possible legal paths:
where $\Phi _T(y)$ is the set of all legal CTC paths for sequence $\mathbf {y}$ with length $T$  The loss can be easily computed using forward-backward algorithm  More details about CTC are provided in supplementary material 
For MT task, we use the cross-entropy loss as the training objective  During training, $\mathbf {y^s}$ is converted to embedding vectors $\mathbf {e^s}$ through embedding layer $W_{E^s}$, then $enc_t$ consumes $\mathbf {e^s}$ and pass the output $\mathbf {h^t}$ to decoder  The objective function is defined as:
Our method ::: Training Procedure ::: Fine-tune
In fine-tune stage, we jointly update the model on ASR, MT, and ST tasks  The training for ASR and MT follows the same process as it was in pre-training stage 
For ST task, the $enc_s$ reads the input $\mathbf {x}$ and generates $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into $\mathbf {h^t}$  Finally, the $dec$ predicts the target sentence  The ST loss function is defined as:
Following the update strategy proposed by BIBREF11 luong2015multi, we allocate a different training ratio $\alpha _i$ for each task  When switching between tasks, we select randomly a new task $i$ with probability $\frac{\alpha _i}{\sum _{j}\alpha _{j}}$ 
Our method ::: Subnet-Consistency
Our model keeps role consistency between pre-training and fine-tuning by connecting two encoders for ST task  However, this leads to some new problems: 1) The text encoder consumes $\mathbf {e^s}$ during MT training, while it accepts $\mathbf {h^s}$ during ST training  However, $\mathbf {e^s}$ and $\mathbf {h^s}$ may not follow the same distribution, resulting in the semantic inconsistency  2) Besides, the length of $\mathbf {h^s}$ is not the same order of magnitude with the length of $\mathbf {e^s}$, resulting in the length inconsistency 
In response to the above two challenges, we propose two countermeasures: 1) We share weights between CTC classification layer and source-end word embedding layer during training of ASR and MT, encouraging $\mathbf {e^s}$ and $\mathbf {h^s}$ in the same space  2)We feed the text encoder source sentences in the format of CTC path, which are generated from a seq2seq model, making it more robust toward long inputs 
Our method ::: Subnet-Consistency ::: Semantic Consistency
As shown in Figure FIGREF5, during multi-task training, two different hidden features will be fed into the text encoder $enc_t$: the embedding representation $\mathbf {e}^s$ in MT task, and the $enc_s$ output $\mathbf {h^s}$ in ST task  Without any regularization, they may belong to different latent spaces  Due to the space gap, the $enc_t$ has to compromise between two tasks, limiting its performance on individual tasks 
To bridge the space gap, our idea is to pull $\mathbf {h^s}$ into the latent space where $\mathbf {e}^s$ belong  Specifically, we share the weight $W_{ctc}$ in CTC classification layer with the source embedding weights $W_{E^s}$, which means $W_{ctc} = W_{E^s}$  In this way, when predicting the CTC path $\mathbf {\pi }$, the probability of observing the particular label $w_i \in V_{src}\cup ${`-'} at time step $t$, $p(\pi _t=w_i|\mathbf {x})$, is computed by normalizing the product of hidden vector $h_t^s$ and the $i$-th vector in $W_{E^s}$:
The loss function closes the distance between $h^s_t$ and golden embedding vector, encouraging $\mathbf {h}^s$ have the same distribution with $\mathbf {e}^s$ 
Our method ::: Subnet-Consistency ::: Length Consistency
Another existing problem is length inconsistency  The length of the sequence $\mathbf {h^s}$ is proportional to the length of the input frame $\mathbf {x}$, which is much longer than the length of $\mathbf {e^s}$  To solve this problem, we train an RNN-based seq2seq model to transform normal source sentences to noisy sentences in CTC path format, and replace standard MT with denoising MT for multi-tasking 
Specifically, we first train a CTC ASR model based on dataset $\mathcal {A} = \lbrace (\mathbf {x}_i, \mathbf {y}^s_i)\rbrace _{i=0}^{I}$, and generate a CTC-path $\mathbf {\pi }_i$ for each audio $\mathbf {x}_i$ by greedy decoding  Then we define an operation $S(\cdot )$, which converts a CTC path $\mathbf {\pi }$ to a sequence of the unique tokens $\mathbf {u}$ and a sequence of repetition times for each token $\mathbf {l}$, denoted as $S(\mathbf {\pi }) = (\mathbf {u}, \mathbf {l})$  Notably, the operation is reversible, meaning that $S^{-1} (\mathbf {u}, \mathbf {l})=\mathbf {\pi }$  We use the example $\mathbf {\pi _1}$ in Table TABREF14 and show the corresponding $\mathbf {u}$ and $\mathbf {l}$ in Table TABREF24 
Then we build a dataset $\mathcal {P} = \lbrace (\mathbf {y^s}_i, \mathbf {u}_i, \mathbf {l}_i)\rbrace _{i=0}^{I}$ by decoding all the audio pieces in $\mathcal {A}$ and transform the resulting path by the operation $S(\cdot )$  After that, we train a seq2seq model, as shown in Figure FIGREF25, which takes $ \mathbf {y^s}_i$ as input and decodes $\mathbf {u}_i, \mathbf {l}_i$ as outputs  With the seq2seq model, a noisy MT dataset $\mathcal {M}^{\prime }=\lbrace (\mathbf {\pi }_l, \mathbf {y^t}_l)\rbrace _{l=0}^{L}$ is obtained by converting every source sentence $\mathbf {y^s}_i \in \mathcal {M}$ to $\mathbf {\pi _i}$, where $\mathbf {\pi }_i = S^{-1}(\mathbf {u}_i, \mathbf {l}_i)$  We did not use the standard seq2seq model which takes $\mathbf {y^s}$ as input and generates $\mathbf {\pi }$ directly, since there are too many blank tokens `-' in $\mathbf {\pi }$ and the model tends to generate a long sequence with only blank tokens  During MT training, we randomly sample text pairs from $\mathcal {M}^{\prime }$ and $\mathcal {M}$ according to a hyper-parameter $k$  After tuning on the validation set, about $30\%$ pairs are sampled from $\mathcal {M}^{\prime }$  In this way, the $enc_t$ is more robust toward the longer inputs given by the $enc_s$ 
Experiments
We conduct experiments on the IWSLT18 speech translation task BIBREF1
Context 3: Introduction
Long short term memory (LSTM) units BIBREF1 are popular for many sequence modeling tasks and are used extensively in language modeling  A key to their success is their articulated gating structure, which allows for more control over the information passed along the recurrence  However, despite the sophistication of the gating mechanisms employed in LSTMs and similar recurrent units, the input and context vectors are treated with simple linear transformations prior to gating  Non-linear transformations such as convolutions BIBREF2 have been used, but these have not achieved the performance of well regularized LSTMs for language modeling BIBREF3  
A natural way to improve the expressiveness of linear transformations is to increase the number of dimensions of the input and context vectors, but this comes with a significant increase in the number of parameters which may limit generalizability  An example is shown in Figure FIGREF1 , where LSTMs performance decreases with the increase in dimensions of the input and context vectors  Moreover, the semantics of the input and context vectors are different, suggesting that each may benefit from specialized treatment 
Guided by these insights, we introduce a new recurrent unit, the Pyramidal Recurrent Unit (PRU), which is based on the LSTM gating structure  Figure FIGREF2 provides an overview of the PRU  At the heart of the PRU is the pyramidal transformation (PT), which uses subsampling to effect multiple views of the input vector  The subsampled representations are combined in a pyramidal fusion structure, resulting in richer interactions between the individual dimensions of the input vector than is possible with a linear transformation  Context vectors, which have already undergone this transformation in the previous cell, are modified with a grouped linear transformation (GLT) which allows the network to learn latent representations in high dimensional space with fewer parameters and better generalizability (see Figure FIGREF1 ) 
We show that PRUs can better model contextual information and demonstrate performance gains on the task of language modeling  The PRU improves the perplexity of the current state-of-the-art language model BIBREF0 by up to 1 3 points, reaching perplexities of 56 56 and 64 53 on the Penn Treebank and WikiText2 datasets while learning 15-20% fewer parameters  Replacing an LSTM with a PRU results in improvements in perplexity across a variety of experimental settings  We provide detailed ablations which motivate the design of the PRU architecture, as well as detailed analysis of the effect of the PRU on other components of the language model 
Related work
Multiple methods, including a variety of gating structures and transformations, have been proposed to improve the performance of recurrent neural networks (RNNs)  We first describe these approaches and then provide an overview of recent work in language modeling 
Pyramidal Recurrent Units
We introduce Pyramidal Recurrent Units (PRUs), a new RNN architecture which improves modeling of context by allowing for higher dimensional vector representations while learning fewer parameters  Figure FIGREF2 provides an overview of PRU  We first elaborate on the details of the pyramidal transformation and the grouped linear transformation  We then describe our recurrent unit, PRU 
Pyramidal transformation for input
The basic transformation in many recurrent units is a linear transformation INLINEFORM0 defined as: DISPLAYFORM0
where INLINEFORM0 are learned weights that linearly map INLINEFORM1 to INLINEFORM2   To simplify notation, we omit the biases 
Motivated by successful applications of sub-sampling in computer vision (e g , BIBREF22 , BIBREF23 , BIBREF9 , BIBREF24 ), we subsample input vector INLINEFORM0 into INLINEFORM1 pyramidal levels to achieve representation of the input vector at multiple scales  This sub-sampling operation produces INLINEFORM2 vectors, represented as INLINEFORM3 , where INLINEFORM4 is the sampling rate and INLINEFORM5   We learn scale-specific transformations INLINEFORM6 for each INLINEFORM7   The transformed subsamples are concatenated to produce the pyramidal analog to INLINEFORM8 , here denoted as INLINEFORM9 : DISPLAYFORM0
where INLINEFORM0 indicates concatenation  We note that pyramidal transformation with INLINEFORM1 is the same as the linear transformation 
To improve gradient flow inside the recurrent unit, we combine the input and output using an element-wise sum (when dimension matches) to produce residual analog of pyramidal transformation, as shown in Figure FIGREF2 BIBREF25  
We sub-sample the input vector INLINEFORM0 into INLINEFORM1 pyramidal levels using the kernel-based approach BIBREF8 , BIBREF9   Let us assume that we have a kernel INLINEFORM2 with INLINEFORM3 elements  Then, the input vector INLINEFORM4 can be sub-sampled as: DISPLAYFORM0
where INLINEFORM0 represents the stride and INLINEFORM1  
The number of parameters learned by the linear transformation and the pyramidal transformation with INLINEFORM0 pyramidal levels to map INLINEFORM1 to INLINEFORM2 are INLINEFORM3 and INLINEFORM4 respectively  Thus, pyramidal transformation reduces the parameters of a linear transformation by a factor of INLINEFORM5   For example, the pyramidal transformation (with INLINEFORM6 and INLINEFORM7 ) learns INLINEFORM8 fewer parameters than the linear transformation 
Grouped linear transformation for context
Many RNN architectures apply linear transformations to both the input and context vector  However, this may not be ideal due to the differing semantics of each vector  In many NLP applications including language modeling, the input vector is a dense word embedding which is shared across all contexts for a given word in a dataset  In contrast, the context vector is highly contextualized by the current sequence  The differences between the input and context vector motivate their separate treatment in the PRU architecture 
The weights learned using the linear transformation (Eq  EQREF9 ) are reused over multiple time steps, which makes them prone to over-fitting BIBREF26   To combat over-fitting, various methods, such as variational dropout BIBREF26 and weight dropout BIBREF0 , have been proposed to regularize these recurrent connections  To further improve generalization abilities while simultaneously enabling the recurrent unit to learn representations at very high dimensional space, we propose to use grouped linear transformation (GLT) instead of standard linear transformation for recurrent connections BIBREF27   While pyramidal and linear transformations can be applied to transform context vectors, our experimental results in Section SECREF39 suggests that GLTs are more effective 
The linear transformation INLINEFORM0 maps INLINEFORM1 linearly to INLINEFORM2   Grouped linear transformations break the linear interactions by factoring the linear transformation into two steps  First, a GLT splits the input vector INLINEFORM3 into INLINEFORM4 smaller groups such that INLINEFORM5   Second, a linear transformation INLINEFORM6 is applied to map INLINEFORM7 linearly to INLINEFORM8 , for each INLINEFORM9   The INLINEFORM10 resultant output vectors INLINEFORM11 are concatenated to produce the final output vector INLINEFORM12 

ground_truth: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish

Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish , Swedish

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the baselines?
Context 2:  Since IWSLT participators use different data pre-processing methods, we reproduce several competitive baselines based on the ESPnet BIBREF12 for a fair comparison 
Experiments ::: Dataset ::: Speech translation data:
The organizer provides a speech translation corpus extracting from the TED talk (ST-TED), which consists of raw English wave files, English transcriptions, and aligned German translations  The corpus contains 272 hours of English speech with 171k segments  We split 2k segments from the corpus as dev set and tst2010, tst2013, tst2014, tst2015 are used as test sets 
Speech recognition data: Aside from ST-TED, TED-LIUM2 corpus BIBREF13 is provided as speech recognition data, which contains 207 hours of English speech and 93k transcript sentences 
Text translation data: We use transcription and translation pairs in the ST-TED corpus and WIT3 as in-domain MT data, which contains 130k and 200k sentence pairs respectively  WMT2018 is used as out-of-domain training data which consists of 41M sentence pairs 
Data preprocessing: For speech data, the utterances are segmented into multiple frames with a 25 ms window size and a 10 ms step size  Then we extract 80-channel log-Mel filter bank and 3-dimensional pitch features using Kaldi BIBREF14, resulting in 83-dimensional input features  We normalize them by the mean and the standard deviation on the whole training set  The utterances with more than 3000 frames are discarded  The transcripts in ST-TED are in true-case with punctuation while in TED-LIUM2, transcripts are in lower-case and unpunctuated  Thus, we lowercase all the sentences and remove the punctuation to keep consistent  To increase the amount of training data, we perform speed perturbation on the raw signals with speed factors 0 9 and 1 1  For the text translation data, sentences longer than 80 words or shorter than 10 words are removed  Besides, we discard pairs whose length ratio between source and target sentence is smaller than 0 5 or larger than 2 0  Word tokenization is performed using the Moses scripts and both English and German words are in lower-case 
We use two different sets of vocabulary for our experiments  For the subword experiments, both English and German vocabularies are generated using sentencepiece BIBREF15 with a fixed size of 5k tokens  BIBREF9 inaguma2018speech show that increasing the vocabulary size is not helpful for ST task  For the character experiments, both English and German sentences are represented in the character level 
For evaluation, we segment each audio with the LIUM SpkDiarization tool BIBREF16 and then perform MWER segmentation with RWTH toolkit BIBREF17  We use lowercase BLEU as evaluation metric 
Experiments ::: Baseline Models and Implementation
We compare our method with following baselines 
Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder  It is trained from scratch on the ST-TED corpus 
Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained  The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus  The MT model has a text encoder and decoder with the same architecture of which in TCEN  It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data 
Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting  In the first two settings, we train the model with $\alpha _{st}=0 75$ while $\alpha _{asr}=0 25$ or $\alpha _{mt}=0 25$  For many-to-many setting, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$   For MT task, we use only in-domain data 
Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models  Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation  Their model solves the subnet waste issue by concatenating an ST decoder to an ASR encoder-decoder model  Notably, their ST decoder can consume representations from the speech encoder as well as the ASR decoder  For a fair comparison, the speech encoder and the ASR decoder are initialized from the pre-trained ASR model  The Triangle model is fine-tuned under their multi-task manner 
All our baselines as well as TCEN are implemented based on ESPnet BIBREF12, the RNN size is set as $d=1024$ for all models  We use a dropout of 0 3 for embeddings and encoders, and train using Adadelta with initial learning rate of 1 0 for a maximum of 10 epochs 
For training of TCEN, we set $\alpha _{asr}=0 2$ and $\alpha _{mt}=0 8$ in the pre-training stage, since the MT dataset is much larger than ASR dataset  For fine-tune, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$, same as the `many-to-many' baseline 
For testing, we select the model with the best accuracy on speech translation task on dev set  At inference time, we use a beam size of 10, and the beam scores include length normalization with a weight of 0 2 
Experiments ::: Experimental Results
Table TABREF29 shows the results on four test sets as well as the average performance  Our method significantly outperforms the strong `many-to-many+pretrain' baseline by 3 6 and 2 2 BLEU scores respectively, indicating the proposed method is very effective that substantially improves the translation quality  Besides, both pre-training and multi-task learning can improve translation quality, and the pre-training settings (2nd-4th rows) are more effective compared to multi-task settings (5th-8th rows)  We observe a performance degradation in the `triangle+pretrain' baseline  Compared to our method, where the decoder receives higher-level syntactic and semantic linguistic knowledge extracted from text encoder, their ASR decoder can only provide lower word-level linguistic information  Besides, since their model lacks text encoder and the architecture of ST decoder is different from MT decoder, their model cannot utilize the large-scale MT data in all the training stages  Interestingly, we find that the char-level models outperform the subword-level models in all settings, especially in vanilla baseline  A similar phenomenon is observed by BIBREF6 berard2018end  A possible explanation is that learning the alignments between speech frames and subword units in another language is notoriously difficult  Our method can bring more gains in the subword setting since our model is good at learning the text-to-text alignment and the subword-level alignment is more helpful to the translation quality 
Experiments ::: Discussion ::: Ablation Study
To better understand the contribution of each component, we perform an ablation study on subword-level experiments  The results are shown in Table TABREF37  In `-MT noise' setting, we do not add noise to source sentences for MT  In `-weight sharing' setting, we use different parameters in CTC classification layer and source embedding layer  These two experiments prove that both weight sharing and using noisy MT input benefit to the final translation quality  Performance degrades more in `-weight sharing', indicating the semantic consistency contributes more to our model  In the `-pretrain' experiment, we remove the pre-training stage and directly update the model on three tasks, leading to a dramatic decrease on BLEU score, indicating the pre-training is an indispensable step for end-to-end ST 
Experiments ::: Discussion ::: Learning Curve
It is interesting to investigate why our method is superior to baselines  We find that TCEN achieves a higher final result owing to a better start-point in fine-tuning  Figure FIGREF39 provides learning curves of subword accuracy on validation set  The x-axis denotes the fine-tuning training steps  The vanilla model starts at a low accuracy, because its networks are not pre-trained on the ASR and MT data  The trends of our model and `many-to-many+pretrain' are similar, but our model outperforms it about five points in the whole fine-tuning process  It indicates that the gain comes from bridging the gap between pre-training and fine-tuning rather than a better fine-tuning process 
Experiments ::: Discussion ::: Compared with a Cascaded System
Table TABREF29 compares our model with end-to-end baselines  Here, we compare our model with cascaded systems  We build a cascaded system by combining the ASR model and MT model used in pre-training baseline  Word error rate (WER) of the ASR system and BLEU score of the MT system are reported in the supplementary material  In addition to a simple combination of the ASR and MT systems, we also re-segment the ASR outputs before feeding to the MT system, denoted as cascaded+re-seg  Specifically, we train a seq2seq model BIBREF19 on the MT dataset, where the source side is a no punctuation sentence and the target side is a natural sentence  After that, we use the seq2seq model to add sentence boundaries and punctuation on ASR outputs  Experimental results are shown in Table TABREF41  Our end-to-end model outperforms the simple cascaded model over 2 BLEU scores, and it achieves a comparable performance with the cascaded model combining with a sentence re-segment model 
Related Work
Early works conduct speech translation in a pipeline manner BIBREF2, BIBREF20, where the ASR output lattices are fed into an MT system to generate target sentences  HMM BIBREF21, DenseNet BIBREF22, TDNN BIBREF23 are commonly used ASR systems, while RNN with attention BIBREF19 and Transformer BIBREF10 are top choices for MT  To enhance the robustness of the NMT model towards ASR errors, BIBREF24 DBLP:conf/eacl/TsvetkovMD14 and BIBREF25 DBLP:conf/asru/ChenHHL17 propose to simulate the noise in training and inference 
To avoid error propagation and high latency issues, recent works propose translating the acoustic speech into text in target language without yielding the source transcription BIBREF4  Since ST data is scarce, pre-training BIBREF7, multi-task learning BIBREF4, BIBREF6, curriculum learning BIBREF26, attention-passing BIBREF27, and knowledge distillation BIBREF28, BIBREF29 strategies have been explored to utilize ASR data and MT data  Specifically, BIBREF5 DBLP:conf/interspeech/WeissCJWC17 show improvements of performance by training the ST model jointly with the ASR and the MT model  BIBREF6 berard2018end observe faster convergence and better results due to pre-training and multi-task learning on a larger dataset  BIBREF7 DBLP:conf/naacl/BansalKLLG19 show that pre-training a speech encoder on one language can improve ST quality on a different source language  All of them follow the traditional multi-task training strategies  BIBREF26 DBLP:journals/corr/abs-1802-06003 propose to use curriculum learning to improve ST performance on syntactically distant language pairs  To effectively leverage transcriptions in ST data, BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 augment the multi-task model where the target decoder receives information from the source decoder and they show improvements on low-resource speech translation  Their model just consumes ASR and ST data, in contrast, our work sufficiently utilizes the large-scale MT data to capture the rich semantic knowledge  BIBREF30 DBLP:conf/icassp/JiaJMWCCALW19 use pre-trained MT and text-to-speech (TTS) synthesis models to convert weakly supervised data into ST pairs and demonstrate that an end-to-end MT model can be trained using only synthesised data 
Conclusion
This paper has investigated the end-to-end method for ST  It has discussed why there is a huge gap between pre-training and fine-tuning in previous methods  To alleviate these issues, we have proposed a method, which is capable of reusing every sub-net and keeping the role of sub-net consistent between pre-training and fine-tuning  Empirical studies have demonstrated that our model significantly outperforms baselines 
Context 3:  DISPLAYFORM0
GLTs learn representations at low dimensionality  Therefore, a GLT requires INLINEFORM0 fewer parameters than the linear transformation  We note that GLTs are subset of linear transformations  In a linear transformation, each neuron receives an input from each element in the input vector while in a GLT, each neuron receives an input from a subset of the input vector  Therefore, GLT is the same as a linear transformation when INLINEFORM1  
Pyramidal Recurrent Unit
We extend the basic gating architecture of LSTM with the pyramidal and grouped linear transformations outlined above to produce the Pyramidal Recurrent Unit (PRU), whose improved sequence modeling capacity is evidenced in Section SECREF4  
At time INLINEFORM0 , the PRU combines the input vector INLINEFORM1 and the previous context vector (or previous hidden state vector) INLINEFORM2 using the following transformation function as: DISPLAYFORM0
where INLINEFORM0 indexes the various gates in the LSTM model, and INLINEFORM1 and INLINEFORM2 represents the pyramidal and grouped linear transformations defined in Eqns  EQREF10 and EQREF15 , respectively 
We will now incorporate INLINEFORM0 into LSTM gating architecture to produce PRU  At time INLINEFORM1 , a PRU cell takes INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 as inputs to produce forget INLINEFORM5 , input INLINEFORM6 , output INLINEFORM7 , and content INLINEFORM8 gate signals  The inputs are combined with these gate signals to produce context vector INLINEFORM9 and cell state INLINEFORM10   Mathematically, the PRU with the LSTM gating architecture can be defined as: DISPLAYFORM0
where INLINEFORM0 represents the element-wise multiplication operation, and INLINEFORM1 and INLINEFORM2 are the sigmoid and hyperbolic tangent activation functions  We note that LSTM is a special case of PRU when INLINEFORM3 = INLINEFORM4 =1 
Experiments
To showcase the effectiveness of the PRU, we evaluate the performance on two standard datasets for word-level language modeling and compare with state-of-the-art methods  Additionally, we provide a detailed examination of the PRU and its behavior on the language modeling tasks 
Set-up
Following recent works, we compare on two widely used datasets, the Penn Treebank (PTB) BIBREF28 as prepared by BIBREF29 and WikiText2 (WT-2) BIBREF20   For both datasets, we follow the same training, validation, and test splits as in BIBREF0  
We extend the language model, AWD-LSTM BIBREF0 , by replacing LSTM layers with PRU  Our model uses 3-layers of PRU with an embedding size of 400  The number of parameters learned by state-of-the-art methods vary from 18M to 66M with majority of the methods learning about 22M to 24M parameters on the PTB dataset  For a fair comparison with state-of-the-art methods, we fix the model size to 19M and vary the value of INLINEFORM0 and hidden layer sizes so that total number of learned parameters is similar across different configurations  We use 1000, 1200, and 1400 as hidden layer sizes for values of INLINEFORM1 =1,2, and 4, respectively  We use the same settings for the WT-2 dataset  We set the number of pyramidal levels INLINEFORM2 to two in our experiments and use average pooling for sub-sampling  These values are selected based on our ablation experiments on the validation set (Section SECREF39 )  We measure the performance of our models in terms of word-level perplexity  We follow the same training strategy as in BIBREF0  
To understand the effect of regularization methods on the performance of PRUs, we perform experiments under two different settings: (1) Standard dropout: We use a standard dropout BIBREF12 with probability of 0 5 after embedding layer, the output between LSTM layers, and the output of final LSTM layer  (2) Advanced dropout: We use the same dropout techniques with the same dropout values as in BIBREF0   We call this model as AWD-PRU 
Results
Table TABREF23 compares the performance of the PRU with state-of-the-art methods  We can see that the PRU achieves the best performance with fewer parameters 
PRUs achieve either the same or better performance than LSTMs  In particular, the performance of PRUs improves with the increasing value of INLINEFORM0   At INLINEFORM1 , PRUs outperform LSTMs by about 4 points on the PTB dataset and by about 3 points on the WT-2 dataset  This is explained in part by the regularization effect of the grouped linear transformation (Figure FIGREF1 )  With grouped linear and pyramidal transformations, PRUs learn rich representations at very high dimensional space while learning fewer parameters  On the other hand, LSTMs overfit to the training data at such high dimensions and learn INLINEFORM2 to INLINEFORM3 more parameters than PRUs 
With the advanced dropouts, the performance of PRUs improves by about 4 points on the PTB dataset and 7 points on the WT-2 dataset  This further improves with finetuning on the PTB (about 2 points) and WT-2 (about 1 point) datasets 
For similar number of parameters, the PRU with standard dropout outperforms most of the state-of-the-art methods by large margin on the PTB dataset (e g  RAN BIBREF7 by 16 points with 4M less parameters, QRNN BIBREF33 by 16 points with 1M more parameters, and NAS BIBREF31 by 1 58 points with 6M less parameters)  With advanced dropouts, the PRU delivers the best performance  On both datasets, the PRU improves the perplexity by about 1 point while learning 15-20% fewer parameters 
PRU is a drop-in replacement for LSTM, therefore, it can improve language models with modern inference techniques such as dynamic evaluation BIBREF21   When we evaluate PRU-based language models (only with standard dropout) with dynamic evaluation on the PTB test set, the perplexity of PRU ( INLINEFORM0 ) improves from 62 42 to 55 23 while the perplexity of an LSTM ( INLINEFORM1 ) with similar settings improves from 66 29 to 58 79; suggesting that modern inference techniques are equally applicable to PRU-based language models 
Analysis
It is shown above that the PRU can learn representations at higher dimensionality with more generalization power, resulting in performance gains for language modeling  A closer analysis of the impact of the PRU in a language modeling system reveals several factors that help explain how the PRU achieves these gains 
As exemplified in Table TABREF34 , the PRU tends toward more confident decisions, placing more of the probability mass on the top next-word prediction than the LSTM

ground_truth: Vanilla ST baseline, encoder pre-training, in which the ST encoder is initialized from an ASR model, decoder pre-training, in which the ST decoder is initialized from an MT model, encoder-decoder pre-training, where both the encoder and decoder are pre-trained, many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models, Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation

Vanilla ST baseline, Pre-training baselines, Multi-task baselines, Many-to-many+pre-training, Triangle+pre-train

Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder. It is trained from scratch on the ST-TED corpus.

Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained. The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus. The MT model has a text encoder and decoder with the same architecture of which in TCEN. It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data.

Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting. In the first two settings, we train the model with $\alpha _{st}=0.75$ while $\alpha _{asr}=0.25$ or $\alpha _{mt}=0.25$. For many-to-many setting, we use $\alpha _{st}=0.6, \alpha _{asr}=0.2$ and $\alpha _{mt}=0.2$.. For MT task, we use only in-domain data.

Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models. 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what previous RNN models do they compare with?
Context 3:  To quantify this effect, we calculate the entropy of the next-token distribution for both the PRU and the LSTM using 3687 contexts from the PTB validation set  Figure FIGREF32 shows a histogram of the entropies of the distribution, where bins of size 0 23 are used to effect categories  We see that the PRU more often produces lower entropy distributions corresponding to higher confidences for next-token choices  This is evidenced by the mass of the red PRU curve lying in the lower entropy ranges compared to the blue LSTM's curve  The PRU can produce confident decisions in part because more information is encoded in the higher dimensional context vectors 
The PRU has the ability to model individual words at different resolutions through the pyramidal transform; which provides multiple paths for the gradient to the embedding layer (similar to multi-task learning) and improves the flow of information  When considering the embeddings by part of speech, we find that the pyramid level 1 embeddings exhibit higher variance than the LSTM across all POS categories (Figure FIGREF33 ), and that pyramid level 2 embeddings show extremely low variance  We hypothesize that the LSTM must encode both coarse group similarities and individual word differences into the same vector space, reducing the space between individual words of the same category  The PRU can rely on the subsampled embeddings to account for coarse-grained group similarities, allowing for finer individual word distinctions in the embedding layer  This hypothesis is strengthened by the entropy results described above: a model which can make finer distinctions between individual words can more confidently assign probability mass  A model that cannot make these distinctions, such as the LSTM, must spread its probability mass across a larger class of similar words 
Saliency analysis using gradients help identify relevant words in a test sequence that contribute to the prediction BIBREF34 , BIBREF35 , BIBREF36   These approaches compute the relevance as the squared norm of the gradients obtained through back-propagation  Table TABREF34 visualizes the heatmaps for different sequences  PRUs, in general, give more relevance to contextual words than LSTMs, such as southeast (sample 1), cost (sample 2), face (sample 4), and introduced (sample 5), which help in making more confident decisions  Furthermore, when gradients during back-propagation are visualized BIBREF37 (Table TABREF34 ), we find that PRUs have better gradient coverage than LSTMs, suggesting PRUs use more features than LSTMs that contributes to the decision  This also suggests that PRUs update more parameters at each iteration which results in faster training  Language model in BIBREF0 takes 500 and 750 epochs to converge with PRU and LSTM as a recurrent unit, respectively 
Ablation studies
In this section, we provide a systematic analysis of our design choices  Our training methodology is the same as described in Section SECREF19 with the standard dropouts  For a thorough understanding of our design choices, we use a language model with a single layer of PRU and fix the size of embedding and hidden layers to 600  The word-level perplexities are reported on the validation sets of the PTB and the WT-2 datasets 
The two hyper-parameters that control the trade-off between performance and number of parameters in PRUs are the number of pyramidal levels INLINEFORM0 and groups INLINEFORM1   Figure FIGREF35 provides a trade-off between perplexity and recurrent unit (RU) parameters 
Variable INLINEFORM0 and fixed INLINEFORM1 : When we increase the number of pyramidal levels INLINEFORM2 at a fixed value of INLINEFORM3 , the performance of the PRU drops by about 1 to 4 points while reducing the total number of recurrent unit parameters by up to 15%  We note that the PRU with INLINEFORM4 at INLINEFORM5 delivers similar performance as the LSTM while learning about 15% fewer recurrent unit parameters 
Fixed INLINEFORM0 and variable INLINEFORM1 : When we vary the value of INLINEFORM2 at fixed number of pyramidal levels INLINEFORM3 , the total number of recurrent unit parameters decreases significantly with a minimal impact on the perplexity  For example, PRUs with INLINEFORM4 and INLINEFORM5 learns 77% fewer recurrent unit parameters while its perplexity (lower is better) increases by about 12% in comparison to LSTMs  Moreover, the decrease in number of parameters at higher value of INLINEFORM6 enables PRUs to learn the representations in high dimensional space with better generalizability (Table TABREF23 ) 
Table TABREF43 shows the impact of different transformations of the input vector INLINEFORM0 and the context vector INLINEFORM1   We make following observations: (1) Using the pyramidal transformation for the input vectors improves the perplexity by about 1 point on both the PTB and WT-2 datasets while reducing the number of recurrent unit parameters by about 14% (see R1 and R4)  We note that the performance of the PRU drops by up to 1 point when residual connections are not used (R4 and R6)  (2) Using the grouped linear transformation for context vectors reduces the total number of recurrent unit parameters by about 75% while the performance drops by about 11% (see R3 and R4)  When we use the pyramidal transformation instead of the linear transformation, the performance drops by up to 2% while there is no significant drop in the number of parameters (R4 and R5) 
We set sub-sampling kernel INLINEFORM0 (Eq  EQREF12 ) with stride INLINEFORM1 and size of 3 ( INLINEFORM2 ) in four different ways: (1) Skip: We skip every other element in the input vector  (2) Convolution: We initialize the elements of INLINEFORM3 randomly from normal distribution and learn them during training the model  We limit the output values between -1 and 1 using INLINEFORM4 activation function to make training stable  (3) Avg  pool: We initialize the elements of INLINEFORM5 to INLINEFORM6   (4) Max pool: We select the maximum value in the kernel window INLINEFORM7  
Table TABREF45 compares the performance of the PRU with different sampling methods  Average pooling performs the best while skipping give comparable performance  Both of these methods enable the network to learn richer word representations while representing the input vector in different forms, thus delivering higher performance  Surprisingly, a convolution-based sub-sampling method does not perform as well as the averaging method  The INLINEFORM0 function used after convolution limits the range of output values which are further limited by the LSTM gating structure, thereby impeding in the flow of information inside the cell  Max pooling forces the network to learn representations from high magnitude elements, thus distinguishing features between elements vanishes, resulting in poor performance 
Conclusion
We introduce the Pyramidal Recurrent Unit, which better model contextual information by admitting higher dimensional representations with good generalizability  When applied to the task of language modeling, PRUs improve perplexity across several settings, including recent state-of-the-art systems  Our analysis shows that the PRU improves the flow of gradient and expand the word embedding subspace, resulting in more confident decisions  Here we have shown improvements for language modeling  In future, we plan to study the performance of PRUs on different tasks, including machine translation and question answering  In addition, we will study the performance of the PRU on language modeling with more recent inference techniques, such as dynamic evaluation and mixture of softmax 
Acknowledgments
This research was supported by NSF (IIS 1616112, III 1703166), Allen Distinguished Investigator Award, and gifts from Allen Institute for AI, Google, Amazon, and Bloomberg  We are grateful to Aaron Jaech, Hannah Rashkin, Mandar Joshi, Aniruddha Kembhavi, and anonymous reviewers for their helpful comments 

ground_truth: Variational LSTM, CharCNN, Pointer Sentinel-LSTM, RHN, NAS Cell, SRU, QRNN, RAN, 4-layer skip-connection LSTM, AWD-LSTM, Quantized LSTM