prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what is the source of the data?
Context 1: Introduction
Nowadays deep learning techniques outperform the other conventional methods in most of the speech-related tasks  Training robust deep neural networks for each task depends on the availability of powerful processing GPUs, as well as standard and large scale datasets  In text-independent speaker verification, large-scale datasets are available, thanks to the NIST SRE evaluations and other data collection projects such as VoxCeleb BIBREF0 
In text-dependent speaker recognition, experiments with end-to-end architectures conducted on large proprietary databases have demonstrated their superiority over traditional approaches BIBREF1  Yet, contrary to text-independent speaker recognition, text-dependent speaker recognition lacks large-scale publicly available databases  The two most well-known datasets are probably RSR2015 BIBREF2 and RedDots BIBREF3  The former contains speech data collected from 300 individuals in a controlled manner, while the latter is used primarily for evaluation rather than training, due to its small number of speakers (only 64)  Motivated by this lack of large-scale dataset for text-dependent speaker verification, we chose to proceed with the collection of the DeepMine dataset, which we expect to become a standard benchmark for the task 
Apart from speaker recognition, large amounts of training data are required also for training automatic speech recognition (ASR) systems  Such datasets should not only be large in size, they should also be characterized by high variability with respect to speakers, age and dialects  While several datasets with these properties are available for languages like English, Mandarin, French, this is not the case for several other languages, such as Persian  To this end, we proceeded with collecting a large-scale dataset, suitable for building robust ASR models in Persian 
The main goal of the DeepMine project was to collect speech from at least a few thousand speakers, enabling research and development of deep learning methods  The project started at the beginning of 2017, and after designing the database and the developing Android and server applications, the data collection began in the middle of 2017  The project finished at the end of 2018 and the cleaned-up and final version of the database was released at the beginning of 2019  In BIBREF4, the running project and its data collection scenarios were described, alongside with some preliminary results and statistics  In this paper, we announce the final and cleaned-up version of the database, describe its different parts and provide various evaluation setups for each part  Finally, since the database was designed mainly for text-dependent speaker verification purposes, some baseline results are reported for this task on the official evaluation setups  Additional baseline results are also reported for Persian speech recognition  However, due to the space limitation in this paper, the baseline results are not reported for all the database parts and conditions  They will be defined and reported in the database technical documentation and in a future journal paper 
Data Collection
DeepMine is publicly available for everybody with a variety of licenses for different users  It was collected using crowdsourcing BIBREF4  The data collection was done using an Android application  Each respondent installed the application on his/her personal device and recorded several phrases in different sessions  The Android application did various checks on each utterance and if it passed all of them, the respondent was directed to the next phrase  For more information about data collection scenario, please refer to BIBREF4 
Data Collection ::: Post-Processing
In order to clean-up the database, the main post-processing step was to filter out problematic utterances  Possible problems include speaker word insertions (e g  repeating some part of a phrase), deletions, substitutions, and involuntary disfluencies  To detect these, we implemented an alignment stage, similar to the second alignment stage in the LibriSpeech project BIBREF5  In this method, a custom decoding graph was generated for each phrase  The decoding graph allows for word skipping and word insertion in the phrase 
For text-dependent and text-prompted parts of the database, such errors are not allowed  Hence, any utterances with errors were removed from the enrollment and test lists  For the speech recognition part, a sub-part of the utterance which is correctly aligned to the corresponding transcription is kept  After the cleaning step, around 190 thousand utterances with full transcription and 10 thousand with sub-part alignment have remained in the database 
Data Collection ::: Statistics
After processing the database and removing problematic respondents and utterances, 1969 respondents remained in the database, with 1149 of them being male and 820 female  297 of the respondents could not read English and have therefore read only the Persian prompts  About 13200 sessions were recorded by females and similarly, about 9500 sessions by males, i e  women are over-represented in terms of sessions, even though their number is 17% smaller than that of males  Other useful statistics related to the database are shown in Table TABREF4 
The last status of the database, as well as other related and useful information about its availability can be found on its website, together with a limited number of samples 
DeepMine Database Parts
The DeepMine database consists of three parts  The first one contains fixed common phrases to perform text-dependent speaker verification  The second part consists of random sequences of words useful for text-prompted speaker verification, and the last part includes phrases with word- and phoneme-level transcription, useful for text-independent speaker verification using a random phrase (similar to Part4 of RedDots)  This part can also serve for Persian ASR training  Each part is described in more details below  Table TABREF11 shows the number of unique phrases in each part of the database  For the English text-dependent part, the following phrases were selected from part1 of the RedDots database, hence the RedDots can be used as an additional training set for this part:
“My voice is my password ”
“OK Google ”
“Artificial intelligence is for real ”
“Actions speak louder than words ”
“There is no such thing as a free lunch ”
DeepMine Database Parts ::: Part1 - Text-dependent (TD)
This part contains a set of fixed phrases which are used to verify speakers in text-dependent mode  Each speaker utters 5 Persian phrases, and if the speaker can read English, 5 phrases selected from Part1 of the RedDots database are also recorded 
We have created three experimental setups with different numbers of speakers in the evaluation set  For each setup, speakers with more recording sessions are included in the evaluation set and the rest of the speakers are used for training in the background set (in the database, all background sets are basically training data)  The rows in Table TABREF13 corresponds to the different experimental setups and shows the numbers of speakers in each set  Note that, for English, we have filtered the (Persian native) speakers by the ability to read English  Therefore, there are fewer speakers in each set for English than for Persian  There is a small “dev” set in each setup which can be used for parameter tuning to prevent over-tuning on the evaluation set 
For each experimental setup, we have defined several official trial lists with different numbers of enrollment utterances per trial in order to investigate the effects of having different amounts of enrollment data  All trials in one trial list have the same number of enrollment utterances (3 to 6) and only one test utterance  All enrollment utterances in a trial are taken from different consecutive sessions and the test utterance is taken from yet another session  From all the setups and conditions, the 100-spk with 3-session enrollment (3-sess) is considered as the main evaluation condition  In Table TABREF14, the number of trials for Persian 3-sess are shown for the different types of trial in the text-dependent speaker verification (SV)  Note that for Imposter-Wrong (IW) trials (i e  imposter speaker pronouncing wrong phrase), we merely create one wrong trial for each Imposter-Correct (IC) trial to limit the huge number of possible trials for this case  So, the number of trials for IC and IW cases are the same 
DeepMine Database Parts ::: Part2 - Text-prompted (TP)
For this part, in each session, 3 random sequences of Persian month names are shown to the respondent in two modes: In the first mode, the sequence consists of all 12 months, which will be used for speaker enrollment  The second mode contains a sequence of 3 month names that will be used as a test utterance  In each 8 sessions received by a respondent from the server, there are 3 enrollment phrases of all 12 months (all in just one session), and $7 \times 3$ other test phrases, containing fewer words  For a respondent who can read English, 3 random sequences of English digits are also recorded in each session  In one of the sessions, these sequences contain all digits and the remaining ones contain only 4 digits 
Similar to the text-dependent case, three experimental setups with different number of speaker in the evaluation set are defined (corresponding to the rows in Table TABREF16)  However, different strategy is used for defining trials: Depending on the enrollment condition (1- to 3-sess), trials are enrolled on utterances of all words from 1 to 3 different sessions (i e  3 to 9 utterances)  Further, we consider two conditions for test utterances: seq test utterance with only 3 or 4 words and full test utterances with all words (i e  same words as in enrollment but in different order)  From all setups an all conditions, the 100-spk with 1-session enrolment (1-sess) is considered as the main evaluation condition for the text-prompted case  In Table TABREF16, the numbers of trials (sum for both seq and full conditions) for Persian 1-sess are shown for the different types of trials in the text-prompted SV  Again, we just create one IW trial for each IC trial 
DeepMine Database Parts ::: Part3 - Text-independent (TI)
In this part, 8 Persian phrases that have already been transcribed on the phone level are displayed to the respondent  These phrases are chosen mostly from news and Persian Wikipedia  If the respondent is unable to read English, instead of 5 fixed phrases and 3 random digit strings, 8 other Persian phrases are also prompted to the respondent to have exactly 24 phrases in each recording session 
This part can be useful at least for three potential applications  First, it can be used for text-independent speaker verification  The second application of this part (same as Part4 of RedDots) is text-prompted speaker verification using random text (instead of a random sequence of words)  Finally, the third application is large vocabulary speech recognition in Persian (explained in the next sub-section) 
Based on the recording sessions, we created two experimental setups for speaker verification  In the first one, respondents with at least 17 recording sessions are included to the evaluation set, respondents with 16 sessions to the development and the rest of respondents to the background set (can be used as training data)  In the second setup, respondents with at least 8 sessions are included to the evaluation set, respondents with 6 or 7 sessions to the development and the rest of respondents to the background set  Table TABREF18 shows numbers of speakers in each set of the database for text-independent SV case 
For text-independent SV, we have considered 4 scenarios for enrollment and 4 scenarios for test  The speaker can be enrolled using utterances from 1, 2 or 3 consecutive sessions (1sess to 3sess) or using 8 utterances from 8 different sessions  The test speech can be one utterance (1utt) for short duration scenario or all utterances in one session (1sess) for long duration case  In addition, test speech can be selected from 5 English phrases for cross-language testing (enrollment using Persian utterances and test using English utterances)  From all setups, 1sess-1utt and 1sess-1sess for 438-spk set are considered as the main evaluation setups for text-independent case  Table TABREF19 shows number of trials for these setups 
For text-prompted SV with random text, the same setup as text-independent case together with corresponding utterance transcriptions can be used 
DeepMine Database Parts ::: Part3 - Speech Recognition
As explained before, Part3 of the DeepMine database can be used for Persian read speech recognition  There are only a few databases for speech recognition in Persian BIBREF6, BIBREF7  Hence, this part can at least partly address this problem and enable robust speech recognition applications in Persian  Additionally, it can be used for speaker recognition applications, such as training deep neural networks (DNNs) for extracting bottleneck features BIBREF8, or for collecting sufficient statistics using DNNs for i-vector training 
We have randomly selected 50 speakers (25 for each gender) from the all speakers in the database which have net speech (without silence parts) between 25 minutes to 50 minutes as test speakers  For each speaker, the utterances in the first 5 sessions are included to (small) test-set and the other utterances of test speakers are considered as a large-test-set  The remaining utterances of the other speakers are included in the training set  The test-set, large-test-set and train-set contain 5 9, 28 5 and 450 hours of speech respectively 
There are about 8300 utterances in Part3 which contain only Persian full names (i e  first and family name pairs)  Each phrase consists of several full names and their phoneme transcriptions were extracted automatically using a trained Grapheme-to-Phoneme (G2P)  These utterances can be used to evaluate the performance of a systems for name recognition, which is usually more difficult than the normal speech recognition because of the lack of a reliable language model 
Experiments and Results
Due to the space limitation, we present results only for the Persian text-dependent speaker verification and speech recognition 
Experiments and Results ::: Speaker Verification Experiments
We conducted an experiment on text-dependent speaker verification part of the database, using the i-vector based method proposed in BIBREF9, BIBREF10 and applied it to the Persian portion of Part1  In this experiment, 20-dimensional MFCC features along with first and second derivatives are extracted from 16 kHz signals using HTK BIBREF11 with 25 ms Hamming windowed frames with 15 ms overlap 
The reported results are obtained with a 400-dimensional gender independent i-vector based system  The i-vectors are first length-normalized and are further normalized using phrase- and gender-dependent Regularized Within-Class Covariance Normalization (RWCCN) BIBREF10  Cosine distance is used to obtain speaker verification scores and phrase- and gender-dependent s-norm is used for normalizing the scores  For aligning speech frames to Gaussian components, monophone HMMs with 3 states and 8 Gaussian components in each state are used BIBREF10  We only model the phonemes which appear in the 5 Persian text-dependent phrases 
For speaker verification experiments, the results were reported in terms of Equal Error Rate (EER) and Normalized Detection Cost Function as defined for NIST SRE08 ($\mathrm {NDCF_{0 01}^{min}}$) and NIST SRE10 ($\mathrm {NDCF_{0 001}^{min}}$)  As shown in Table TABREF22, in text-dependent SV there are 4 types of trials: Target-Correct and Imposter-Correct refer to trials when the pass-phrase is uttered correctly by target and imposter speakers respectively, and in same manner, Target-Wrong and Imposter-Wrong refer to trials when speakers uttered a wrong pass-phrase  In this paper, only the correct trials (i e  Target-Correct as target trials vs Imposter-Correct as non-target trials) are considered for evaluating systems as it has been proved that these are the most challenging trials in text-dependent SV BIBREF8, BIBREF12 
Table TABREF23 shows the results of text-dependent experiments using Persian 100-spk and 3-sess setup  For filtering trials, the respondents' mobile brand and model were used in this experiment  In the table, the first two letters in the filter notation relate to the target trials and the second two letters (i e  right side of the colon) relate for non-target trials  For target trials, the first Y means the enrolment and test utterances were recorded using a device with the same brand by the target speaker  The second Y letter means both recordings were done using exactly the same device model  Similarly, the first Y for non-target trials means that the devices of target and imposter speakers are from the same brand (i e  manufacturer)  The second Y means that, in addition to the same brand, both devices have the same model  So, the most difficult target trials are “NN”, where the speaker has used different a device at the test time  In the same manner, the most difficult non-target trials which should be rejected by the system are “YY” where the imposter speaker has used the same device model as the target speaker (note that it does not mean physically the same device because each speaker participated in the project using a personal mobile device)  Hence, the similarity in the recording channel makes rejection more difficult 
The first row in Table TABREF23 shows the results for all trials  By comparing the results with the best published results on RSR2015 and RedDots BIBREF10, BIBREF8, BIBREF12, it is clear that the DeepMine database is more challenging than both RSR2015 and RedDots databases  For RSR2015, the same i-vector/HMM-based method with both RWCCN and s-norm has achieved EER less than 0 3% for both genders (Table VI in BIBREF10)  The conventional Relevance MAP adaptation with HMM alignment without applying any channel-compensation techniques (i e  without applying RWCCN and s-norm due to the lack of suitable training data) on RedDots Part1 for the male has achieved EER around 1 5% (Table XI in BIBREF10)  It is worth noting that EERs for DeepMine database without any channel-compensation techniques are 2 1 and 3 7% for males and females respectively 
One interesting advantage of the DeepMine database compared to both RSR2015 and RedDots is having several target speakers with more than one mobile device  This is allows us to analyse the effects of channel compensation methods  The second row in Table TABREF23 corresponds to the most difficult trials where the target trials come from mobile devices with different models while imposter trials come from the same device models  It is clear that severe degradation was caused by this kind of channel effects (i e  decreasing within-speaker similarities while increasing between-speaker similarities), especially for females 
The results in the third row show the condition when target speakers at the test time use exactly the same device that was used for enrollment  Comparing this row with the results in the first row proves how much improvement can be achieved when exactly the same device is used by the target speaker 
The results in the fourth row show the condition when imposter speakers also use the same device model at test time to fool the system  So, in this case, there is no device mismatch in all trials  By comparing the results with the third row, we can see how much degradation is caused if we only consider the non-target trials with the same device 
The fifth row shows similar results when the imposter speakers use device of the same brand as the target speaker but with a different model  Surprisingly, in this case, the degradation is negligible and it means that mobiles from a specific brand (manufacturer) have different recording channel properties 
The degraded female results in the sixth row as compared to the third row show the effect of using a different device model from the same brand for target trials  For males, the filters brings almost the same subsets of trials, which explains the very similar results in this case 
Looking at the first two and the last row of Table TABREF23, one can notice the significantly worse performance obtained for the female trials as compared to males  Note that these three rows include target trials where the devices used for enrollment do not necessarily match the devices used for recording test utterances  On the other hand, in rows 3 to 6, which exclude such mismatched trials, the performance for males and females is comparable  This suggest that the degraded results for females are caused by some problematic trials with device mismatch  The exact reason for this degradation is so far unclear and needs a further investigation 
In the last row of the table, the condition of the second row is relaxed: the target device should have different model possibly from the same brand and imposter device only needs to be from the same brand  In this case, as was expected, the performance degradation is smaller than in the second row 
Experiments and Results ::: Speech Recognition Experiments
In addition to speaker verification, we present several speech recognition experiments on Part3  The experiments were performed with the Kaldi toolkit BIBREF13  For training HMM-based MonoPhone model, only 20 thousands of shortest utterances are used and for other models the whole training data is used  The DNN based acoustic model is a time-delay DNN with low-rank factorized layers and skip connections without i-vector adaptation (a modified network from one of the best performing LibriSpeech recipes)  The network is shown in Table TABREF25: there are 16 F-TDNN layers, with dimension 1536 and linear bottleneck layers of dimension 256  The acoustic model is trained for 10 epochs using lattice-free maximum mutual information (LF-MMI) with cross-entropy regularization BIBREF14  Re-scoring is done using a pruned trigram language model and the size of the dictionary is around 90,000 words 
Table TABREF26 shows the results in terms of word error rate (WER) for different evaluated methods  As can be seen, the created database can be used to train well performing and practically usable Persian ASR models 
Conclusions
In this paper, we have described the final version of a large speech corpus, the DeepMine database  It has been collected using crowdsourcing and, according to the best of our knowledge, it is the largest public text-dependent and text-prompted speaker verification database in two languages: Persian and English  In addition, it is the largest text-independent speaker verification evaluation database, making it suitable to robustly evaluate state-of-the-art methods on different conditions  Alongside these appealing properties, it comes with phone-level transcription, making it suitable to train deep neural network models for Persian speech recognition 
We provided several evaluation protocols for each part of the database  The protocols allow researchers to investigate the performance of different methods in various scenarios and study the effects of channels, duration and phrase text on the performance  We also provide two test sets for speech recognition: One normal test set with a few minutes of speech for each speaker and one large test set with more (30 minutes on average) speech that can be used for any speaker adaptation method 
As baseline results, we reported the performance of an i-vector/HMM based method on Persian text-dependent part  Moreover, we conducted speech recognition experiments using conventional HMM-based methods, as well as state-of-the-art deep neural network based method using Kaldi toolkit with promising performance  Text-dependent results have shown that the DeepMine database is more challenging than RSR2015 and RedDots databases 
Acknowledgments
The data collection project was mainly supported by Sharif DeepMine company  The work on the paper was supported by Czech National Science Foundation (GACR) project "NEUREM3" No  19-26934X and the National Programme of Sustainability (NPU II) project "IT4Innovations excellence in science - LQ1602" 
Context 3: Introduction
Speech-to-Text translation (ST) is essential for a wide range of scenarios: for example in emergency calls, where agents have to respond emergent requests in a foreign language BIBREF0; or in online courses, where audiences and speakers use different languages BIBREF1  To tackle this problem, existing approaches can be categorized into cascaded method BIBREF2, BIBREF3, where a machine translation (MT) model translates outputs of an automatic speech recognition (ASR) system into target language, and end-to-end method BIBREF4, BIBREF5, where a single model learns acoustic frames to target word sequence mappings in one step towards the final objective of interest  Although the cascaded model remains the dominant approach due to its better performance, the end-to-end method becomes more and more popular because it has lower latency by avoiding inferences with two models and rectifies the error propagation in theory 
Since it is hard to obtain a large-scale ST dataset, multi-task learning BIBREF5, BIBREF6 and pre-training techniques BIBREF7 have been applied to end-to-end ST model to leverage large-scale datasets of ASR and MT  A common practice is to pre-train two encoder-decoder models for ASR and MT respectively, and then initialize the ST model with the encoder of the ASR model and the decoder of the MT model  Subsequently, the ST model is optimized with the multi-task learning by weighing the losses of ASR, MT, and ST  This approach, however, causes a huge gap between pre-training and fine-tuning, which are summarized into three folds:
Subnet Waste: The ST system just reuses the ASR encoder and the MT decoder, while discards other pre-trained subnets, such as the MT encoder  Consequently, valuable semantic information captured by the MT encoder cannot be inherited by the final ST system 
Role Mismatch: The speech encoder plays different roles in pre-training and fine-tuning  The encoder is a pure acoustic model in pre-training, while it has to extract semantic and linguistic features additionally in fine-tuning, which significantly increases the learning difficulty 
Non-pre-trained Attention Module: Previous work BIBREF6 trains attention modules for ASR, MT and ST respectively, hence, the attention module of ST does not benefit from the pre-training 
To address these issues, we propose a Tandem Connectionist Encoding Network (TCEN), which is able to reuse all subnets in pre-training, keep the roles of subnets consistent, and pre-train the attention module  Concretely, the TCEN consists of three components, a speech encoder, a text encoder, and a target text decoder  Different from the previous work that pre-trains an encoder-decoder based ASR model, we only pre-train an ASR encoder by optimizing the Connectionist Temporal Classification (CTC) BIBREF8 objective function  In this way, the additional decoder of ASR is not required while keeping the ability to read acoustic features into the source language space by the speech encoder  Besides, the text encoder and decoder can be pre-trained on a large MT dataset  After that, we employ common used multi-task learning method to jointly learn ASR, MT and ST tasks 
Compared to prior works, the encoder of TCEN is a concatenation of an ASR encoder and an MT encoder and our model does not have an ASR decoder, so the subnet waste issue is solved  Furthermore, the two encoders work at tandem, disentangling acoustic feature extraction and linguistic feature extraction, ensuring the role consistency between pre-training and fine-tuning  Moreover, we reuse the pre-trained MT attention module in ST, so we can leverage the alignment information learned in pre-training 
Since the text encoder consumes word embeddings of plausible texts in MT task but uses speech encoder outputs in ST task, another question is how one guarantees the speech encoder outputs are consistent with the word embeddings  We further modify our model to achieve semantic consistency and length consistency  Specifically, (1) the projection matrix at the CTC classification layer for ASR is shared with the word embedding matrix, ensuring that they are mapped to the same latent space, and (2) the length of the speech encoder output is proportional to the length of the input frame, so it is much longer than a natural sentence  To bridge the length gap, source sentences in MT are lengthened by adding word repetitions and blank tokens to mimic the CTC output sequences 
We conduct comprehensive experiments on the IWSLT18 speech translation benchmark BIBREF1, demonstrating the effectiveness of each component  Our model is significantly better than previous methods by 3 6 and 2 2 BLEU scores for the subword-level decoding and character-level decoding strategies, respectively 
Our contributions are three-folds: 1) we shed light on why previous ST models cannot sufficiently utilize the knowledge learned from the pre-training process; 2) we propose a new ST model, which alleviates shortcomings in existing methods; and 3) we empirically evaluate the proposed model on a large-scale public dataset 
Background ::: Problem Formulation
End-to-end speech translation aims to translate a piece of audio into a target-language translation in one step  The raw speech signals are usually converted to sequences of acoustic features, e g  Mel filterbank features  Here, we define the speech feature sequence as $\mathbf {x} = (x_1, \cdots , x_{T_x})$ The transcription and translation sequences are denoted as $\mathbf {y^{s}} = (y_1^{s}, \cdots , y_{T_s}^{s})$, and $\mathbf {y^{t}} = (y_1^{t}, \cdots , y_{T_t}^{t})$ repectively  Each symbol in $\mathbf {y^{s}}$ or $\mathbf {y^{t}}$ is an integer index of the symbol in a vocabulary $V_{src}$ or $V_{trg}$ respectively (e g  $y^s_i=k, k\in [0, |V_{src}|-1]$)  In this work, we suppose that an ASR dataset, an MT dataset, and a ST dataset are available, denoted as $\mathcal {A} = \lbrace (\mathbf {x_i}, \mathbf {y^{s}_i})\rbrace _{i=0}^I$, $\mathcal {M} =\lbrace (\mathbf {y^{s}_j}, \mathbf {y^{t}_j})\rbrace _{j=0}^J$ and $ \mathcal {S} =\lbrace (\mathbf {x_l}, \mathbf {y^{t}_l})\rbrace _{l=0}^L$ respectively  Given a new piece of audio $\mathbf {x}$, our goal is to learn an end to end model to generate a translation sentence $\mathbf {y^{t}}$ without generating an intermediate result $\mathbf {y^{s}}$ 
Background ::: Multi-Task Learning and Pre-training for ST
To leverage large scale ASR and MT data, multi-task learning and pre-training techniques are widely employed to improve the ST system  As shown in Figure FIGREF4, there are three popular multi-task strategies for ST, including 1) one-to-many setting, in which a speech encoder is shared between ASR and ST tasks; 2) many-to-one setting in which a decoder is shared between MT and ST tasks; and 3) many-to-many setting where both the encoder and decoder are shared 
A many-to-many multi-task model contains two encoders as well as two decoders  It can be jointly trained on ASR, MT, and ST tasks  As the attention module is task-specific, three attentions are defined 
Usually, the size of $\mathcal {A}$ and $\mathcal {M}$ is much larger than $\mathcal {S}$  Therefore, the common training practice is to pre-train the model on ASR and MT tasks and then fine-tune it with a multi-task learning manner  However, as aforementioned, this method suffers from subnet waste, role mismatch and non-pre-trained attention issues, which severely limits the end-to-end ST performance 
Our method
In this section, we first introduce the architecture of TCEN, which consists of two encoders connected in tandem, and one decoder with an attention module  Then we give the pre-training and fine-tuning strategy for TCEN  Finally, we propose our solutions for semantic and length inconsistency problems, which are caused by multi-task learning 
Our method ::: TCEN Architecture
Figure FIGREF5 sketches the overall architecture of TCEN, including a speech encoder $enc_s$, a text encoder $enc_t$ and a decoder $dec$ with an attention module $att$  During training, the $enc_s$ acts like an acoustic model which reads the input $\mathbf {x}$ to word or subword representations $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into hidden representations $\mathbf {h^t}$  Finally, the $dec$ defines a distribution probability over target words  The advantage of our architecture is that two encoders disentangle acoustic feature extraction and linguistic feature extraction, making sure that valuable knowledge learned from ASR and MT tasks can be effectively leveraged for ST training  Besides, every module in pre-training can be utilized in fine-tuning, alleviating the subnet waste problem 
Follow BIBREF9 inaguma2018speech, we use CNN-BiLSTM architecture to build our model  Specifically, the input features $\mathbf {x}$ are organized as a sequence of feature vectors in length $T_x$  Then, $\mathbf {x}$ is passed into a stack of two convolutional layers followed by max-pooling:
where $\mathbf {v}^{(l-1)}$ is feature maps in last layer and $\mathbf {W}^{(l)}$ is the filter  The max-pooling layers downsample the sequence in length by a total factor of four  The down-sampled feature sequence is further fed into a stack of five bidirectional $d$-dimensional LSTM layers:
where $[;]$ denotes the vector concatenation  The final output representation from the speech encoder is denoted as $\mathbf {h^s}=(h^s_1, \cdots , h^s_{\frac{T_x}{4}})$, where $h_i^s \in \mathbb {R}^d$ 
The text encoder $enc_t$ consists of two bidirectional LSTM layers  In ST task, $enc_t$ accepts speech encoder output $\mathbf {h}^s$ as input  While in MT, $enc_t$ consumes the word embedding representation $\mathbf {e^s}$ derived from $\mathbf {y^s}$, where each element $e^s_i$ is computed by choosing the $y_i^s$-th vector from the source embedding matrix $W_{E^s}$  The goal of $enc_t$ is to extract high-level linguistic features like syntactic features or semantic features from lower level subword representations $\mathbf {h}^s$ or $\mathbf {e}^s$  Since $\mathbf {h}^s$ and $\mathbf {e}^s$ belong to different latent space and have different lengths, there remain semantic and length inconsistency problems  We will provide our solutions in Section SECREF21  The output sequence of $enc_t$ is denoted as $\mathbf {h}^t$ 
The decoder is defined as two unidirectional LSTM layers with an additive attention $att$  It predicts target sequence $\mathbf {y^{t}}$ by estimating conditional probability $P(\mathbf {y^{t}}|\mathbf {x})$:
Here, $z_k$ is the the hidden state of the deocder RNN at $k$ step and $c_k$ is a time-dependent context vector computed by the attention $att$ 
Our method ::: Training Procedure
Following previous work, we split the training procedure to pre-training and fine-tuning stages  In pre-training stage, the speech encoder $enc_s$ is trained towards CTC objective using dataset $\mathcal {A}$, while the text encoder $enc_t$ and the decoder $dec$ are trained on MT dataset $\mathcal {M}$  In fine-tuning stage, we jointly train the model on ASR, MT, and ST tasks 
Our method ::: Training Procedure ::: Pre-training
To sufficiently utilize the large dataset $\mathcal {A}$ and $\mathcal {M}$, the model is pre-trained on CTC-based ASR task and MT task in the pre-training stage 
For ASR task, in order to get rid of the requirement for decoder and enable the $enc_s$ to generate subword representation, we leverage connectionist temporal classification (CTC) BIBREF8 loss to train the speech encoder 
Given an input $\mathbf {x}$, $enc_s$ emits a sequence of hidden vectors $\mathbf {h^s}$, then a softmax classification layer predicts a CTC path $\mathbf {\pi }$, where $\pi _t \in V_{src} \cup $ {`-'} is the observing label at particular RNN step $t$, and `-' is the blank token representing no observed labels:
where $W_{ctc} \in \mathbb {R}^{d \times (|V_{src}|+1)}$ is the weight matrix in the classification layer and $T$ is the total length of encoder RNN 
A legal CTC path $\mathbf {\pi }$ is a variation of the source transcription $\mathbf {y}^s$ by allowing occurrences of blank tokens and repetitions, as shown in Table TABREF14  For each transcription $\mathbf {y}$, there exist many legal CTC paths in length $T$  The CTC objective trains the model to maximize the probability of observing the golden sequence $\mathbf {y}^s$, which is calculated by summing the probabilities of all possible legal paths:
where $\Phi _T(y)$ is the set of all legal CTC paths for sequence $\mathbf {y}$ with length $T$  The loss can be easily computed using forward-backward algorithm  More details about CTC are provided in supplementary material 
For MT task, we use the cross-entropy loss as the training objective  During training, $\mathbf {y^s}$ is converted to embedding vectors $\mathbf {e^s}$ through embedding layer $W_{E^s}$, then $enc_t$ consumes $\mathbf {e^s}$ and pass the output $\mathbf {h^t}$ to decoder  The objective function is defined as:
Our method ::: Training Procedure ::: Fine-tune
In fine-tune stage, we jointly update the model on ASR, MT, and ST tasks  The training for ASR and MT follows the same process as it was in pre-training stage 
For ST task, the $enc_s$ reads the input $\mathbf {x}$ and generates $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into $\mathbf {h^t}$  Finally, the $dec$ predicts the target sentence  The ST loss function is defined as:
Following the update strategy proposed by BIBREF11 luong2015multi, we allocate a different training ratio $\alpha _i$ for each task  When switching between tasks, we select randomly a new task $i$ with probability $\frac{\alpha _i}{\sum _{j}\alpha _{j}}$ 
Our method ::: Subnet-Consistency
Our model keeps role consistency between pre-training and fine-tuning by connecting two encoders for ST task  However, this leads to some new problems: 1) The text encoder consumes $\mathbf {e^s}$ during MT training, while it accepts $\mathbf {h^s}$ during ST training  However, $\mathbf {e^s}$ and $\mathbf {h^s}$ may not follow the same distribution, resulting in the semantic inconsistency  2) Besides, the length of $\mathbf {h^s}$ is not the same order of magnitude with the length of $\mathbf {e^s}$, resulting in the length inconsistency 
In response to the above two challenges, we propose two countermeasures: 1) We share weights between CTC classification layer and source-end word embedding layer during training of ASR and MT, encouraging $\mathbf {e^s}$ and $\mathbf {h^s}$ in the same space  2)We feed the text encoder source sentences in the format of CTC path, which are generated from a seq2seq model, making it more robust toward long inputs 
Our method ::: Subnet-Consistency ::: Semantic Consistency
As shown in Figure FIGREF5, during multi-task training, two different hidden features will be fed into the text encoder $enc_t$: the embedding representation $\mathbf {e}^s$ in MT task, and the $enc_s$ output $\mathbf {h^s}$ in ST task  Without any regularization, they may belong to different latent spaces  Due to the space gap, the $enc_t$ has to compromise between two tasks, limiting its performance on individual tasks 
To bridge the space gap, our idea is to pull $\mathbf {h^s}$ into the latent space where $\mathbf {e}^s$ belong  Specifically, we share the weight $W_{ctc}$ in CTC classification layer with the source embedding weights $W_{E^s}$, which means $W_{ctc} = W_{E^s}$  In this way, when predicting the CTC path $\mathbf {\pi }$, the probability of observing the particular label $w_i \in V_{src}\cup ${`-'} at time step $t$, $p(\pi _t=w_i|\mathbf {x})$, is computed by normalizing the product of hidden vector $h_t^s$ and the $i$-th vector in $W_{E^s}$:
The loss function closes the distance between $h^s_t$ and golden embedding vector, encouraging $\mathbf {h}^s$ have the same distribution with $\mathbf {e}^s$ 
Our method ::: Subnet-Consistency ::: Length Consistency
Another existing problem is length inconsistency  The length of the sequence $\mathbf {h^s}$ is proportional to the length of the input frame $\mathbf {x}$, which is much longer than the length of $\mathbf {e^s}$  To solve this problem, we train an RNN-based seq2seq model to transform normal source sentences to noisy sentences in CTC path format, and replace standard MT with denoising MT for multi-tasking 
Specifically, we first train a CTC ASR model based on dataset $\mathcal {A} = \lbrace (\mathbf {x}_i, \mathbf {y}^s_i)\rbrace _{i=0}^{I}$, and generate a CTC-path $\mathbf {\pi }_i$ for each audio $\mathbf {x}_i$ by greedy decoding  Then we define an operation $S(\cdot )$, which converts a CTC path $\mathbf {\pi }$ to a sequence of the unique tokens $\mathbf {u}$ and a sequence of repetition times for each token $\mathbf {l}$, denoted as $S(\mathbf {\pi }) = (\mathbf {u}, \mathbf {l})$  Notably, the operation is reversible, meaning that $S^{-1} (\mathbf {u}, \mathbf {l})=\mathbf {\pi }$  We use the example $\mathbf {\pi _1}$ in Table TABREF14 and show the corresponding $\mathbf {u}$ and $\mathbf {l}$ in Table TABREF24 
Then we build a dataset $\mathcal {P} = \lbrace (\mathbf {y^s}_i, \mathbf {u}_i, \mathbf {l}_i)\rbrace _{i=0}^{I}$ by decoding all the audio pieces in $\mathcal {A}$ and transform the resulting path by the operation $S(\cdot )$  After that, we train a seq2seq model, as shown in Figure FIGREF25, which takes $ \mathbf {y^s}_i$ as input and decodes $\mathbf {u}_i, \mathbf {l}_i$ as outputs  With the seq2seq model, a noisy MT dataset $\mathcal {M}^{\prime }=\lbrace (\mathbf {\pi }_l, \mathbf {y^t}_l)\rbrace _{l=0}^{L}$ is obtained by converting every source sentence $\mathbf {y^s}_i \in \mathcal {M}$ to $\mathbf {\pi _i}$, where $\mathbf {\pi }_i = S^{-1}(\mathbf {u}_i, \mathbf {l}_i)$  We did not use the standard seq2seq model which takes $\mathbf {y^s}$ as input and generates $\mathbf {\pi }$ directly, since there are too many blank tokens `-' in $\mathbf {\pi }$ and the model tends to generate a long sequence with only blank tokens  During MT training, we randomly sample text pairs from $\mathcal {M}^{\prime }$ and $\mathcal {M}$ according to a hyper-parameter $k$  After tuning on the validation set, about $30\%$ pairs are sampled from $\mathcal {M}^{\prime }$  In this way, the $enc_t$ is more robust toward the longer inputs given by the $enc_s$ 
Experiments
We conduct experiments on the IWSLT18 speech translation task BIBREF1

ground_truth: Android application

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What datasets did they use for evaluation?
Context 2: Introduction
Bidirectional Encoder Representations from Transformers (BERT) is a novel Transformer BIBREF0 model, which recently achieved state-of-the-art performance in several language understanding tasks, such as question answering, natural language inference, semantic similarity, sentiment analysis, and others BIBREF1  While well-suited to dealing with relatively short sequences, Transformers suffer from a major issue that hinders their applicability in classification of long sequences, i e  they are able to consume only a limited context of symbols as their input BIBREF2 
There are several natural language (NLP) processing tasks that involve such long sequences  Of particular interest are topic identification of spoken conversations BIBREF3, BIBREF4, BIBREF5 and call center customer satisfaction prediction BIBREF6, BIBREF7, BIBREF8, BIBREF9  Call center conversations, while usually quite short and to the point, often involve agents trying to solve very complex issues that the customers experience, resulting in some calls taking even an hour or more  For speech analytics purposes, these calls are typically transcribed using an automatic speech recognition (ASR) system, and processed in textual representations further down the NLP pipeline  These transcripts sometimes exceed the length of 5000 words  Furthermore, temporal information might play an important role in tasks like CSAT  For example, a customer may be angry at the beginning of the call, but after her issue is resolved, she would be very satisfied with the way it was handled  Therefore, simple bag of words models, or any model that does not include temporal dependencies between the inputs, may not be well-suited to handle this category of tasks  This motivates us to employ model such as BERT in this task 
In this paper, we propose a method that builds upon BERT's architecture  We split the input text sequence into shorter segments in order to obtain a representation for each of them using BERT  Then, we use either a recurrent LSTM BIBREF10 network, or another Transformer, to perform the actual classification  We call these techniques Recurrence over BERT (RoBERT) and Transformer over BERT (ToBERT)  Given that these models introduce a hierarchy of representations (segment-wise and document-wise), we refer to them as Hierarchical Transformers  To the best of our knowledge, no attempt has been done before to use the Transformer architecture for classification of such long sequences 
Our novel contributions are:
Two extensions - RoBERT and ToBERT - to the BERT model, which enable its application in classification of long texts by performing segmentation and using another layer on top of the segment representations 
State-of-the-art results on the Fisher topic classification task 
Significant improvement on the CSAT prediction task over the MS-CNN model 
Related work
Several dimensionality reduction algorithms such as RBM, autoencoders, subspace multinomial models (SMM) are used to obtain a low dimensional representation of documents from a simple BOW representation and then classify it using a simple linear classifiers BIBREF11, BIBREF12, BIBREF13, BIBREF4  In BIBREF14 hierarchical attention networks are used for document classification  They evaluate their model on several datasets with average number of words around 150  Character-level CNN are explored in BIBREF15 but it is prohibitive for very long documents  In BIBREF16, dataset collected from arXiv papers is used for classification  For classification, they sample random blocks of words and use them together for classification instead of using full document which may work well as arXiv papers are usually coherent and well written on a well defined topic  Their method may not work well on spoken conversations as random block of words usually do not represent topic of full conversation 
Several researchers addressed the problem of predicting customer satisfaction BIBREF6, BIBREF7, BIBREF8, BIBREF9  In most of these works, logistic regression, SVM, CNN are applied on different kinds of representations 
In BIBREF17, authors use BERT for document classification but the average document length is less than BERT maximum length 512  TransformerXL BIBREF2 is an extension to the Transformer architecture that allows it to better deal with long inputs for the language modelling task  It relies on the auto-regressive property of the model, which is not the case in our tasks 
Method ::: BERT
Because our work builds heavily upon BERT, we provide a brief summary of its features  BERT is built upon the Transformer architecture BIBREF0, which uses self-attention, feed-forward layers, residual connections and layer normalization as the main building blocks  It has two pre-training objectives:
Masked language modelling - some of the words in a sentence are being masked and the model has to predict them based on the context (note the difference from the typical autoregressive language model training objective);
Next sentence prediction - given two input sequences, decide whether the second one is the next sentence or not 
BERT has been shown to beat the state-of-the-art performance on 11 tasks with no modifications to the model architecture, besides adding a task-specific output layer BIBREF1  We follow same procedure suggested in BIBREF1 for our tasks  Fig  FIGREF8 shows the BERT model for classification  We obtain two kinds of representation from BERT: pooled output from last transformer block, denoted by H, and posterior probabilities, denoted by P  There are two variants of BERT - BERT-Base and BERT-Large  In this work we are using BERT-Base for faster training and experimentation, however, our methods are applicable to BERT-Large as well  BERT-Base and BERT-Large are different in model parameters such as number of transformer blocks, number of self-attention heads  Total number of parameters in BERT-Base are 110M and 340M in BERT-Large 
BERT suffers from major limitations in terms of handling long sequences  Firstly, the self-attention layer has a quadratic complexity $O(n^2)$ in terms of the sequence length $n$ BIBREF0  Secondly, BERT uses a learned positional embeddings scheme BIBREF1, which means that it won't likely be able to generalize to positions beyond those seen in the training data 
To investigate the effect of fine-tuning BERT on task performance, we use either the pre-trained BERT weights, or the weights from a BERT fine-tuned on the task-specific dataset on a segment-level (i e  we preserve the original label but fine-tune on each segment separately instead of on the whole text sequence)  We compare these results to using the fine-tuned segment-level BERT predictions directly as inputs to the next layer 
Method ::: Recurrence over BERT
Given that BERT is limited to a particular input length, we split the input sequence into segments of a fixed size with overlap  For each of these segments, we obtain H or P from BERT model  We then stack these segment-level representations into a sequence, which serves as input to a small (100-dimensional) LSTM layer  Its output serves as a document embedding  Finally, we use two fully connected layers with ReLU (30-dimensional) and softmax (the same dimensionality as the number of classes) activations to obtain the final predictions 
With this approach, we overcome BERT's computational complexity, reducing it to $O(n/k * k^2) = O(nk)$ for RoBERT, with $k$ denoting the segment size (the LSTM component has negligible linear complexity $O(k)$)  The positional embeddings are also no longer an issue 
Method ::: Transformer over BERT
Given that Transformers' edge over recurrent networks is their ability to effectively capture long distance relationships between words in a sequence BIBREF0, we experiment with replacing the LSTM recurrent layer in favor of a small Transformer model (2 layers of transformer building block containing self-attention, fully connected, etc )  To investigate if preserving the information about the input sequence order is important, we also build a variant of ToBERT which learns positional embeddings at the segment-level representations (but is limited to sequences of length seen during the training) 
ToBERT's computational complexity $O(\frac{n^2}{k^2})$ is asymptotically inferior to RoBERT, as the top-level Transformer model again suffers from quadratic complexity in the number of segments  However, in practice this number is much smaller than the input sequence length (${\frac{n}{k}} << n$), so we haven't observed performance or memory issues with our datasets 
Experiments
We evaluated our models on 3 different datasets:
CSAT dataset for CSAT prediction, consisting of spoken transcripts (automatic via ASR) 
20 newsgroups for topic identification task, consisting of written text;
Fisher Phase 1 corpus for topic identification task, consisting of spoken transcripts (manual);
Experiments ::: CSAT
CSAT dataset consists of US English telephone speech from call centers  For each call in this dataset, customers participated in that call gave a rating on his experience with agent  Originally, this dataset has labels rated on a scale 1-9 with 9 being extremely satisfied and 1 being extremely dissatisfied  Fig  FIGREF16 shows the histogram of ratings for our dataset  As the distribution is skewed towards extremes, we choose to do binary classification with ratings above 4 5 as satisfied and below 4 5 as dissatisfied  Quantization of ratings also helped us to create a balanced dataset  This dataset contains 4331 calls and we split them into 3 sets for our experiments: 2866 calls for training, 362 calls for validation and, finally, 1103 calls for testing 
We obtained the transcripts by employing an ASR system  The ASR system uses TDNN-LSTM acoustic model trained on Fisher and Switchboard datasets with lattice-free maximum mutual information criterion BIBREF18  The word error rates using four-gram language models were 9 2% and 17 3% respectively on Switchboard and CallHome portions of Eval2000 dataset 
Experiments ::: 20 newsgroups
20 newsgroups data set is one of the frequently used datasets in the text processing community for text classification and text clustering  This data set contains approximately 20,000 English documents from 20 topics to be identified, with 11314 documents for training and 7532 for testing  In this work, we used only 90% of documents for training and the remaining 10% for validation  For fair comparison with other publications, we used 53160 words vocabulary set available in the datasets website 
Experiments ::: Fisher
Fisher Phase 1 US English corpus is often used for automatic speech recognition in speech community  In this work, we used it for topic identification as in BIBREF3  The documents are 10-minute long telephone conversations between two people discussing a given topic  We used same training and test splits as BIBREF3 in which 1374 and 1372 documents are used for training and testing respectively  For validation of our model, we used 10% of training dataset and the remaining 90% was used for actual model training  The number of topics in this data set is 40 
Experiments ::: Dataset Statistics
Table TABREF22 shows statistics of our datasets  It can be observed that average length of Fisher is much higher than 20 newsgroups and CSAT  Cumulative distribution of document lengths for each dataset is shown in Fig  FIGREF21  It can be observed that almost all of the documents in Fisher dataset have length more than 1000 words  For CSAT, more than 50% of the documents have length greater than 500 and for 20newsgroups only 10% of the documents have length greater than 500  Note that, for CSAT and 20newsgroups, there are few documents with length more than 5000 
Experiments ::: Architecture and Training Details
In this work, we split document into segments of 200 tokens with a shift of 50 tokens to extract features from BERT model  For RoBERT, LSTM model is trained to minimize cross-entropy loss with Adam optimizer BIBREF19  The initial learning rate is set to $0 001$ and is reduced by a factor of $0 95$ if validation loss does not decrease for 3-epochs  For ToBERT, the Transformer is trained with the default BERT version of Adam optimizer BIBREF1 with an initial learning rate of $5e$-5  We report accuracy in all of our experiments  We chose a model with the best validation accuracy to calculate accuracy on the test set  To accomodate for non-determinism of some TensorFlow GPU operations, we report accuracy averaged over 5 runs 
Results
Table TABREF25 presents results using pre-trained BERT features  We extracted features from the pooled output of final transformer block as these were shown to be working well for most of the tasks BIBREF1  The features extracted from a pre-trained BERT model without any fine-tuning lead to a sub-par performance  However, We also notice that ToBERT model exploited the pre-trained BERT features better than RoBERT  It also converged faster than RoBERT  Table TABREF26 shows results using features extracted after fine-tuning BERT model with our datasets  Significant improvements can be observed compared to using pre-trained BERT features  Also, it can be noticed that ToBERT outperforms RoBERT on Fisher and 20newsgroups dataset by 13 63% and 0 81% respectively  On CSAT, ToBERT performs slightly worse than RoBERT but it is not statistically significant as this dataset is small 
Table TABREF27 presents results using fine-tuned BERT predictions instead of the pooled output from final transformer block  For each document, having obtained segment-wise predictions we can obtain final prediction for the whole document in three ways:
Compute the average of all segment-wise predictions and find the most probable class;
Find the most frequently predicted class;
Train a classification model 
It can be observed from Table TABREF27 that a simple averaging operation or taking most frequent predicted class works competitively for CSAT and 20newsgroups but not for the Fisher dataset  We believe the improvements from using RoBERT or ToBERT, compared to simple averaging or most frequent operations, are proportional to the fraction of long documents in the dataset  CSAT and 20newsgroups have (on average) significantly shorter documents than Fisher, as seen in Fig  FIGREF21  Also, significant improvements for Fisher could be because of less confident predictions from BERT model as this dataset has 40 classes  Fig  FIGREF31 presents the comparison of average voting and ToBERT for various document length ranges for Fisher dataset  We used fine-tuned BERT segment-level predictions (P) for this analysis  It can be observed that ToBERT outperforms average voting in every interval  To the best of our knowledge, this is a state-of-the-art result reported on the Fisher dataset 
Table TABREF32 presents the effect of position embeddings on the model performance  It can be observed that position embeddings did not significantly affect the model performance for Fisher and 20newsgroups, but they helped slightly in CSAT prediction (an absolute improvement of 0 64% F1-score)  We think that this is explained by the fact that Fisher and 20newsgroups are topic identification tasks, and the topic does not change much throughout these documents  However, CSAT may vary during the call, and in some cases a naive assumption that the sequential nature of the transcripts is irrelevant may lead to wrong conclusions 
Table TABREF33 compares our results with previous works  It can be seen that our model ToBERT outperforms CNN based experiments by significant margin on CSAT and Fisher datasets  For CSAT dataset, we used multi-scale CNN (MS-CNN) as the baseline, given its strong results on Fisher and 20newsgroups  The setup was replicated from BIBREF5 for comparison  We also see that our result on 20 newsgroups is 0 6% worse than the state-of-the-art 
Conclusions
In this paper, we presented two methods for long documents using BERT model: RoBERT and ToBERT  We evaluated our experiments on two classification tasks - customer satisfaction prediction and topic identification - using 3 datasets: CSAT, 20newsgroups and Fisher  We observed that ToBERT outperforms RoBERT on pre-trained BERT features and fine-tuned BERT features for all our tasks  Also, we noticed that fine-tuned BERT performs better than pre-trained BERT  We have shown that both RoBERT and ToBERT improved the simple baselines of taking an average (or the most frequent) of segment-wise predictions for long documents to obtain final prediction  Position embeddings did not significantly affect our models performance, but slightly improved the accuracy on the CSAT task  We obtained the best results on Fisher dataset and good improvements for CSAT task compared to the CNN baseline  It is interesting to note that the longer the average input in a given task, the bigger improvement we observe w r t  the baseline for that task  Our results confirm that both RoBERT and ToBERT can be used for long sequences with competitive performance and quick fine-tuning procedure  For future work, we shall focus on training models on long documents directly (i e  in an end-to-end manner) 
Context 3:  Since IWSLT participators use different data pre-processing methods, we reproduce several competitive baselines based on the ESPnet BIBREF12 for a fair comparison 
Experiments ::: Dataset ::: Speech translation data:
The organizer provides a speech translation corpus extracting from the TED talk (ST-TED), which consists of raw English wave files, English transcriptions, and aligned German translations  The corpus contains 272 hours of English speech with 171k segments  We split 2k segments from the corpus as dev set and tst2010, tst2013, tst2014, tst2015 are used as test sets 
Speech recognition data: Aside from ST-TED, TED-LIUM2 corpus BIBREF13 is provided as speech recognition data, which contains 207 hours of English speech and 93k transcript sentences 
Text translation data: We use transcription and translation pairs in the ST-TED corpus and WIT3 as in-domain MT data, which contains 130k and 200k sentence pairs respectively  WMT2018 is used as out-of-domain training data which consists of 41M sentence pairs 
Data preprocessing: For speech data, the utterances are segmented into multiple frames with a 25 ms window size and a 10 ms step size  Then we extract 80-channel log-Mel filter bank and 3-dimensional pitch features using Kaldi BIBREF14, resulting in 83-dimensional input features  We normalize them by the mean and the standard deviation on the whole training set  The utterances with more than 3000 frames are discarded  The transcripts in ST-TED are in true-case with punctuation while in TED-LIUM2, transcripts are in lower-case and unpunctuated  Thus, we lowercase all the sentences and remove the punctuation to keep consistent  To increase the amount of training data, we perform speed perturbation on the raw signals with speed factors 0 9 and 1 1  For the text translation data, sentences longer than 80 words or shorter than 10 words are removed  Besides, we discard pairs whose length ratio between source and target sentence is smaller than 0 5 or larger than 2 0  Word tokenization is performed using the Moses scripts and both English and German words are in lower-case 
We use two different sets of vocabulary for our experiments  For the subword experiments, both English and German vocabularies are generated using sentencepiece BIBREF15 with a fixed size of 5k tokens  BIBREF9 inaguma2018speech show that increasing the vocabulary size is not helpful for ST task  For the character experiments, both English and German sentences are represented in the character level 
For evaluation, we segment each audio with the LIUM SpkDiarization tool BIBREF16 and then perform MWER segmentation with RWTH toolkit BIBREF17  We use lowercase BLEU as evaluation metric 
Experiments ::: Baseline Models and Implementation
We compare our method with following baselines 
Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder  It is trained from scratch on the ST-TED corpus 
Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained  The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus  The MT model has a text encoder and decoder with the same architecture of which in TCEN  It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data 
Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting  In the first two settings, we train the model with $\alpha _{st}=0 75$ while $\alpha _{asr}=0 25$ or $\alpha _{mt}=0 25$  For many-to-many setting, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$   For MT task, we use only in-domain data 
Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models  Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation  Their model solves the subnet waste issue by concatenating an ST decoder to an ASR encoder-decoder model  Notably, their ST decoder can consume representations from the speech encoder as well as the ASR decoder  For a fair comparison, the speech encoder and the ASR decoder are initialized from the pre-trained ASR model  The Triangle model is fine-tuned under their multi-task manner 
All our baselines as well as TCEN are implemented based on ESPnet BIBREF12, the RNN size is set as $d=1024$ for all models  We use a dropout of 0 3 for embeddings and encoders, and train using Adadelta with initial learning rate of 1 0 for a maximum of 10 epochs 
For training of TCEN, we set $\alpha _{asr}=0 2$ and $\alpha _{mt}=0 8$ in the pre-training stage, since the MT dataset is much larger than ASR dataset  For fine-tune, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$, same as the `many-to-many' baseline 
For testing, we select the model with the best accuracy on speech translation task on dev set  At inference time, we use a beam size of 10, and the beam scores include length normalization with a weight of 0 2 
Experiments ::: Experimental Results
Table TABREF29 shows the results on four test sets as well as the average performance  Our method significantly outperforms the strong `many-to-many+pretrain' baseline by 3 6 and 2 2 BLEU scores respectively, indicating the proposed method is very effective that substantially improves the translation quality  Besides, both pre-training and multi-task learning can improve translation quality, and the pre-training settings (2nd-4th rows) are more effective compared to multi-task settings (5th-8th rows)  We observe a performance degradation in the `triangle+pretrain' baseline  Compared to our method, where the decoder receives higher-level syntactic and semantic linguistic knowledge extracted from text encoder, their ASR decoder can only provide lower word-level linguistic information  Besides, since their model lacks text encoder and the architecture of ST decoder is different from MT decoder, their model cannot utilize the large-scale MT data in all the training stages  Interestingly, we find that the char-level models outperform the subword-level models in all settings, especially in vanilla baseline  A similar phenomenon is observed by BIBREF6 berard2018end  A possible explanation is that learning the alignments between speech frames and subword units in another language is notoriously difficult  Our method can bring more gains in the subword setting since our model is good at learning the text-to-text alignment and the subword-level alignment is more helpful to the translation quality 
Experiments ::: Discussion ::: Ablation Study
To better understand the contribution of each component, we perform an ablation study on subword-level experiments  The results are shown in Table TABREF37  In `-MT noise' setting, we do not add noise to source sentences for MT  In `-weight sharing' setting, we use different parameters in CTC classification layer and source embedding layer  These two experiments prove that both weight sharing and using noisy MT input benefit to the final translation quality  Performance degrades more in `-weight sharing', indicating the semantic consistency contributes more to our model  In the `-pretrain' experiment, we remove the pre-training stage and directly update the model on three tasks, leading to a dramatic decrease on BLEU score, indicating the pre-training is an indispensable step for end-to-end ST 
Experiments ::: Discussion ::: Learning Curve
It is interesting to investigate why our method is superior to baselines  We find that TCEN achieves a higher final result owing to a better start-point in fine-tuning  Figure FIGREF39 provides learning curves of subword accuracy on validation set  The x-axis denotes the fine-tuning training steps  The vanilla model starts at a low accuracy, because its networks are not pre-trained on the ASR and MT data  The trends of our model and `many-to-many+pretrain' are similar, but our model outperforms it about five points in the whole fine-tuning process  It indicates that the gain comes from bridging the gap between pre-training and fine-tuning rather than a better fine-tuning process 
Experiments ::: Discussion ::: Compared with a Cascaded System
Table TABREF29 compares our model with end-to-end baselines  Here, we compare our model with cascaded systems  We build a cascaded system by combining the ASR model and MT model used in pre-training baseline  Word error rate (WER) of the ASR system and BLEU score of the MT system are reported in the supplementary material  In addition to a simple combination of the ASR and MT systems, we also re-segment the ASR outputs before feeding to the MT system, denoted as cascaded+re-seg  Specifically, we train a seq2seq model BIBREF19 on the MT dataset, where the source side is a no punctuation sentence and the target side is a natural sentence  After that, we use the seq2seq model to add sentence boundaries and punctuation on ASR outputs  Experimental results are shown in Table TABREF41  Our end-to-end model outperforms the simple cascaded model over 2 BLEU scores, and it achieves a comparable performance with the cascaded model combining with a sentence re-segment model 
Related Work
Early works conduct speech translation in a pipeline manner BIBREF2, BIBREF20, where the ASR output lattices are fed into an MT system to generate target sentences  HMM BIBREF21, DenseNet BIBREF22, TDNN BIBREF23 are commonly used ASR systems, while RNN with attention BIBREF19 and Transformer BIBREF10 are top choices for MT  To enhance the robustness of the NMT model towards ASR errors, BIBREF24 DBLP:conf/eacl/TsvetkovMD14 and BIBREF25 DBLP:conf/asru/ChenHHL17 propose to simulate the noise in training and inference 
To avoid error propagation and high latency issues, recent works propose translating the acoustic speech into text in target language without yielding the source transcription BIBREF4  Since ST data is scarce, pre-training BIBREF7, multi-task learning BIBREF4, BIBREF6, curriculum learning BIBREF26, attention-passing BIBREF27, and knowledge distillation BIBREF28, BIBREF29 strategies have been explored to utilize ASR data and MT data  Specifically, BIBREF5 DBLP:conf/interspeech/WeissCJWC17 show improvements of performance by training the ST model jointly with the ASR and the MT model  BIBREF6 berard2018end observe faster convergence and better results due to pre-training and multi-task learning on a larger dataset  BIBREF7 DBLP:conf/naacl/BansalKLLG19 show that pre-training a speech encoder on one language can improve ST quality on a different source language  All of them follow the traditional multi-task training strategies  BIBREF26 DBLP:journals/corr/abs-1802-06003 propose to use curriculum learning to improve ST performance on syntactically distant language pairs  To effectively leverage transcriptions in ST data, BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 augment the multi-task model where the target decoder receives information from the source decoder and they show improvements on low-resource speech translation  Their model just consumes ASR and ST data, in contrast, our work sufficiently utilizes the large-scale MT data to capture the rich semantic knowledge  BIBREF30 DBLP:conf/icassp/JiaJMWCCALW19 use pre-trained MT and text-to-speech (TTS) synthesis models to convert weakly supervised data into ST pairs and demonstrate that an end-to-end MT model can be trained using only synthesised data 
Conclusion
This paper has investigated the end-to-end method for ST  It has discussed why there is a huge gap between pre-training and fine-tuning in previous methods  To alleviate these issues, we have proposed a method, which is capable of reusing every sub-net and keeping the role of sub-net consistent between pre-training and fine-tuning  Empirical studies have demonstrated that our model significantly outperforms baselines 

ground_truth: CSAT dataset, 20 newsgroups, Fisher Phase 1 corpus

CSAT dataset , 20 newsgroups, Fisher Phase 1 corpus

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the baselines?
Context 4:  chair $\models $ wood  Now, minimizing the KL divergence necessitates maximizing $\log {\sum _j p_{v,j} \exp ({-KL(f_{w,i} (\operatorname{\mathbf {x}})||f_{v,j}(\operatorname{\mathbf {x}}))})}$ which in turn minimizes $KL(f_{w,i}(\operatorname{\mathbf {x}})||f_{v,j}(\operatorname{\mathbf {x}}))$  This will result in the support of the $i^{th}$ component of $w$ to be within the $j^{th}$ component of $v$, and holds for all component pairs leading to the entailment of $w$ within $v$  Consequently, we can see that bench $\models $ trees, bench $\models $ furniture, sling $\models $ trees, and sling $\models $ furniture  Thus, it introduces lexical relationship between the senses of child word and that of the parent word  Minimizing the KL also necessitates maximizing $\log {\sum _j {p_{v,j}} EL_{ij}(w,v)}$ term for all component pairs among $w$ and $v$  This is similar to maximizing expected likelihood kernel, which brings the means of $f_{w,i}(\operatorname{\mathbf {x}})$ and $f_{v,j}(\operatorname{\mathbf {x}})$ closer (weighted by their co-variances) as discussed in BIBREF10  Hence, the proposed approach captures the best of both worlds, thereby catering to both word similarity and entailment 
We also note that minimizing the KL divergence necessitates minimizing $\log {\sum _k p_{w,k} \exp ({-KL(f_{w,i}||f_{w,k})})}$ which in turn maximizes $KL(f_{w,i}||f_{w,k})$  This prevents the different mixture components of a word converging to single Gaussian and encourages capturing different possible senses of the word  The same is also achieved by minimizing $\sum _k {p_{w,k}} EL_{ik}(w,w)$ term and act as a regularization term which promotes diversity in learning senses of a word 
Experimentation and Results
We train our proposed model GM$\_$KL (Gaussian Mixture using KL Divergence) on the Text8 dataset BIBREF14 which is a pre-processed data of $17M$ words from wikipedia  Of which, 71290 unique and frequent words are chosen using the subsampling trick in BIBREF15  We compare GM$\_$KL with the previous approaches w2g BIBREF9 ( single Gaussian model) and w2gm BIBREF10 (mixture of Gaussian model with expected likelihood kernel)  For all the models used for experimentation, the embedding size ($D$) was set to 50, number of mixtures to 2, context window length to 10, batch size to 128  The word embeddings were initialized using a uniform distribution in the range of $[-\sqrt{\frac{3}{D}}$, $\sqrt{\frac{3}{D}}]$ such that the expectation of variance is 1 and mean 0 BIBREF16  One could also consider initializing the word embeddings using other contextual representations such as BERT BIBREF17 and ELMo BIBREF18 in the proposed approach  In order to purely analyze the performance of $\emph {GM\_KL}$ over the other models, we have chosen initialization using uniform distribution for experiments  For computational benefits, diagonal covariance is used similar to BIBREF10  Each mixture probability is constrained in the range $[0,1]$, summing to 1 by optimizing over unconstrained scores in the range $(-\infty ,\infty )$ and converting scores to probability using softmax function  The mixture scores are initialized to 0 to ensure fairness among all the components  The threshold for negative sampling was set to $10^{-5}$, as recommended in BIBREF3  Mini-batch gradient descent with Adagrad optimizer BIBREF19 was used with initial learning rate set to $0 05$ 
Table TABREF9 shows the qualitative results of GM$\_$KL  Given a query word and component id, the set of nearest neighbours along with their respective component ids are listed  For eg , the word `plane' in its 0th component captures the `geometry' sense and so are its neighbours, and its 1st component captures `vehicle' sense and so are its corresponding neighbours  Other words such as `rock' captures both the `metal' and `music' senses, `star' captures `celebrity' and `astronomical' senses, and `phone' captures `telephony' and `internet' senses 
We quantitatively compare the performance of the GM$\_$KL, w2g, and w2gm approaches on the SCWS dataset BIBREF6  The dataset consists of 2003 word pairs of polysemous and homonymous words with labels obtained by an average of 10 human scores  The Spearman correlation between the human scores and the model scores are computed  To obtain the model score, the following metrics are used:
MaxCos: Maximum cosine similarity among all component pairs of words $w$ and $v$:
AvgCos: Average component-wise cosine similarity between the words $w$ and $v$ 
KL$\_$approx: Formulated as shown in (DISPLAY_FORM8) between the words $w$ and $v$ 
KL$\_$comp: Maximum component-wise negative KL between words $w$ and $v$:
Table TABREF17 compares the performance of the approaches on the SCWS dataset  It is evident from Table TABREF17 that GM$\_$KL achieves better correlation than existing approaches for various metrics on SCWS dataset 
Table TABREF18 shows the Spearman correlation values of GM$\_$KL model evaluated on the benchmark word similarity datasets: SL BIBREF20, WS, WS-R, WS-S BIBREF21, MEN BIBREF22, MC BIBREF23, RG BIBREF24, YP BIBREF25, MTurk-287 and MTurk-771 BIBREF26, BIBREF27, and RW BIBREF28  The metric used for comparison is 'AvgCos'  It can be seen that for most of the datasets, GM$\_$KL achieves significantly better correlation score than w2g and w2gm approaches  Other datasets such as MC and RW consist of only a single sense, and hence w2g model performs better and GM$\_$KL achieves next better performance  The YP dataset have multiple senses but does not contain entailed data and hence could not make use of entailment benefits of GM$\_$KL 
Table TABREF19 shows the evaluation results of GM$\_$KL model on the entailment datasets such as entailment pairs dataset BIBREF29 created from WordNet with both positive and negative labels, a crowdsourced dataset BIBREF30 of 79 semantic relations labelled as entailed or not and annotated distributionally similar nouns dataset BIBREF31  The 'MaxCos' similarity metric is used for evaluation and the best precision and best F1-score is shown, by picking the optimal threshold  Overall, GM$\_$KL performs better than both w2g and w2gm approaches 
Conclusion
We proposed a KL divergence based energy function for learning multi-sense word embedding distributions modelled as Gaussian mixtures  Due to the intractability of the Gaussian mixtures for the KL divergence measure, we use an approximate KL divergence function  We also demonstrated that the proposed GM$\_$KL approaches performed better than other approaches on the benchmark word similarity and entailment datasets 
tocsectionAppendices
Approximation for KL divergence between mixtures of gaussians
KL between gaussian mixtures $f_{w}(\operatorname{\mathbf {x}})$ and $f_{v}(\operatorname{\mathbf {x}})$ can be decomposed as:
BIBREF12 presents KL approximation between gaussian mixtures using
product of gaussian approximation method where KL is approximated using product of component gaussians and
variational approximation method where KL is approximated by introducing some variational parameters 
The product of component gaussian approximation method using Jensen's inequality provides upper bounds as shown in equations DISPLAY_FORM23 and  
The variational approximation method provides lower bounds as shown in equations DISPLAY_FORM24 and DISPLAY_FORM25 
where $H$ represents the entropy term and the entropy of $i^{th}$ component of word $w$ with dimension $D$ is given as
In BIBREF13, the authors combine the lower and upper bounds from approximation methods of BIBREF12 to formulate a stricter bound on KL between gaussian mixtures 
From equations DISPLAY_FORM23 and DISPLAY_FORM25, a stricter lower bound for KL between gaussian mixtures is obtained as shown in equation DISPLAY_FORM26
From equations and DISPLAY_FORM24, a stricter upper bound for KL between gaussian mixtures is obtained as shown in equation DISPLAY_FORM27
Finally, the KL between gaussian mixtures is taken as the mean of KL upper and lower bounds as shown in equation DISPLAY_FORM28 

ground_truth: Vanilla ST baseline, encoder pre-training, in which the ST encoder is initialized from an ASR model, decoder pre-training, in which the ST decoder is initialized from an MT model, encoder-decoder pre-training, where both the encoder and decoder are pre-trained, many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models, Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation

Vanilla ST baseline, Pre-training baselines, Multi-task baselines, Many-to-many+pre-training, Triangle+pre-train

Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder. It is trained from scratch on the ST-TED corpus.

Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained. The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus. The MT model has a text encoder and decoder with the same architecture of which in TCEN. It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data.

Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting. In the first two settings, we train the model with $\alpha _{st}=0.75$ while $\alpha _{asr}=0.25$ or $\alpha _{mt}=0.25$. For many-to-many setting, we use $\alpha _{st}=0.6, \alpha _{asr}=0.2$ and $\alpha _{mt}=0.2$.. For MT task, we use only in-domain data.

Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models.