prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what is the source of the data?
Context 1: Introduction
Nowadays deep learning techniques outperform the other conventional methods in most of the speech-related tasks  Training robust deep neural networks for each task depends on the availability of powerful processing GPUs, as well as standard and large scale datasets  In text-independent speaker verification, large-scale datasets are available, thanks to the NIST SRE evaluations and other data collection projects such as VoxCeleb BIBREF0 
In text-dependent speaker recognition, experiments with end-to-end architectures conducted on large proprietary databases have demonstrated their superiority over traditional approaches BIBREF1  Yet, contrary to text-independent speaker recognition, text-dependent speaker recognition lacks large-scale publicly available databases  The two most well-known datasets are probably RSR2015 BIBREF2 and RedDots BIBREF3  The former contains speech data collected from 300 individuals in a controlled manner, while the latter is used primarily for evaluation rather than training, due to its small number of speakers (only 64)  Motivated by this lack of large-scale dataset for text-dependent speaker verification, we chose to proceed with the collection of the DeepMine dataset, which we expect to become a standard benchmark for the task 
Apart from speaker recognition, large amounts of training data are required also for training automatic speech recognition (ASR) systems  Such datasets should not only be large in size, they should also be characterized by high variability with respect to speakers, age and dialects  While several datasets with these properties are available for languages like English, Mandarin, French, this is not the case for several other languages, such as Persian  To this end, we proceeded with collecting a large-scale dataset, suitable for building robust ASR models in Persian 
The main goal of the DeepMine project was to collect speech from at least a few thousand speakers, enabling research and development of deep learning methods  The project started at the beginning of 2017, and after designing the database and the developing Android and server applications, the data collection began in the middle of 2017  The project finished at the end of 2018 and the cleaned-up and final version of the database was released at the beginning of 2019  In BIBREF4, the running project and its data collection scenarios were described, alongside with some preliminary results and statistics  In this paper, we announce the final and cleaned-up version of the database, describe its different parts and provide various evaluation setups for each part  Finally, since the database was designed mainly for text-dependent speaker verification purposes, some baseline results are reported for this task on the official evaluation setups  Additional baseline results are also reported for Persian speech recognition  However, due to the space limitation in this paper, the baseline results are not reported for all the database parts and conditions  They will be defined and reported in the database technical documentation and in a future journal paper 
Data Collection
DeepMine is publicly available for everybody with a variety of licenses for different users  It was collected using crowdsourcing BIBREF4  The data collection was done using an Android application  Each respondent installed the application on his/her personal device and recorded several phrases in different sessions  The Android application did various checks on each utterance and if it passed all of them, the respondent was directed to the next phrase  For more information about data collection scenario, please refer to BIBREF4 
Data Collection ::: Post-Processing
In order to clean-up the database, the main post-processing step was to filter out problematic utterances  Possible problems include speaker word insertions (e g  repeating some part of a phrase), deletions, substitutions, and involuntary disfluencies  To detect these, we implemented an alignment stage, similar to the second alignment stage in the LibriSpeech project BIBREF5  In this method, a custom decoding graph was generated for each phrase  The decoding graph allows for word skipping and word insertion in the phrase 
For text-dependent and text-prompted parts of the database, such errors are not allowed  Hence, any utterances with errors were removed from the enrollment and test lists  For the speech recognition part, a sub-part of the utterance which is correctly aligned to the corresponding transcription is kept  After the cleaning step, around 190 thousand utterances with full transcription and 10 thousand with sub-part alignment have remained in the database 
Data Collection ::: Statistics
After processing the database and removing problematic respondents and utterances, 1969 respondents remained in the database, with 1149 of them being male and 820 female  297 of the respondents could not read English and have therefore read only the Persian prompts  About 13200 sessions were recorded by females and similarly, about 9500 sessions by males, i e  women are over-represented in terms of sessions, even though their number is 17% smaller than that of males  Other useful statistics related to the database are shown in Table TABREF4 
The last status of the database, as well as other related and useful information about its availability can be found on its website, together with a limited number of samples 
DeepMine Database Parts
The DeepMine database consists of three parts  The first one contains fixed common phrases to perform text-dependent speaker verification  The second part consists of random sequences of words useful for text-prompted speaker verification, and the last part includes phrases with word- and phoneme-level transcription, useful for text-independent speaker verification using a random phrase (similar to Part4 of RedDots)  This part can also serve for Persian ASR training  Each part is described in more details below  Table TABREF11 shows the number of unique phrases in each part of the database  For the English text-dependent part, the following phrases were selected from part1 of the RedDots database, hence the RedDots can be used as an additional training set for this part:
“My voice is my password ”
“OK Google ”
“Artificial intelligence is for real ”
“Actions speak louder than words ”
“There is no such thing as a free lunch ”
DeepMine Database Parts ::: Part1 - Text-dependent (TD)
This part contains a set of fixed phrases which are used to verify speakers in text-dependent mode  Each speaker utters 5 Persian phrases, and if the speaker can read English, 5 phrases selected from Part1 of the RedDots database are also recorded 
We have created three experimental setups with different numbers of speakers in the evaluation set  For each setup, speakers with more recording sessions are included in the evaluation set and the rest of the speakers are used for training in the background set (in the database, all background sets are basically training data)  The rows in Table TABREF13 corresponds to the different experimental setups and shows the numbers of speakers in each set  Note that, for English, we have filtered the (Persian native) speakers by the ability to read English  Therefore, there are fewer speakers in each set for English than for Persian  There is a small “dev” set in each setup which can be used for parameter tuning to prevent over-tuning on the evaluation set 
For each experimental setup, we have defined several official trial lists with different numbers of enrollment utterances per trial in order to investigate the effects of having different amounts of enrollment data  All trials in one trial list have the same number of enrollment utterances (3 to 6) and only one test utterance  All enrollment utterances in a trial are taken from different consecutive sessions and the test utterance is taken from yet another session  From all the setups and conditions, the 100-spk with 3-session enrollment (3-sess) is considered as the main evaluation condition  In Table TABREF14, the number of trials for Persian 3-sess are shown for the different types of trial in the text-dependent speaker verification (SV)  Note that for Imposter-Wrong (IW) trials (i e  imposter speaker pronouncing wrong phrase), we merely create one wrong trial for each Imposter-Correct (IC) trial to limit the huge number of possible trials for this case  So, the number of trials for IC and IW cases are the same 
DeepMine Database Parts ::: Part2 - Text-prompted (TP)
For this part, in each session, 3 random sequences of Persian month names are shown to the respondent in two modes: In the first mode, the sequence consists of all 12 months, which will be used for speaker enrollment  The second mode contains a sequence of 3 month names that will be used as a test utterance  In each 8 sessions received by a respondent from the server, there are 3 enrollment phrases of all 12 months (all in just one session), and $7 \times 3$ other test phrases, containing fewer words  For a respondent who can read English, 3 random sequences of English digits are also recorded in each session  In one of the sessions, these sequences contain all digits and the remaining ones contain only 4 digits 
Similar to the text-dependent case, three experimental setups with different number of speaker in the evaluation set are defined (corresponding to the rows in Table TABREF16)  However, different strategy is used for defining trials: Depending on the enrollment condition (1- to 3-sess), trials are enrolled on utterances of all words from 1 to 3 different sessions (i e  3 to 9 utterances)  Further, we consider two conditions for test utterances: seq test utterance with only 3 or 4 words and full test utterances with all words (i e  same words as in enrollment but in different order)  From all setups an all conditions, the 100-spk with 1-session enrolment (1-sess) is considered as the main evaluation condition for the text-prompted case  In Table TABREF16, the numbers of trials (sum for both seq and full conditions) for Persian 1-sess are shown for the different types of trials in the text-prompted SV  Again, we just create one IW trial for each IC trial 
DeepMine Database Parts ::: Part3 - Text-independent (TI)
In this part, 8 Persian phrases that have already been transcribed on the phone level are displayed to the respondent  These phrases are chosen mostly from news and Persian Wikipedia  If the respondent is unable to read English, instead of 5 fixed phrases and 3 random digit strings, 8 other Persian phrases are also prompted to the respondent to have exactly 24 phrases in each recording session 
This part can be useful at least for three potential applications  First, it can be used for text-independent speaker verification  The second application of this part (same as Part4 of RedDots) is text-prompted speaker verification using random text (instead of a random sequence of words)  Finally, the third application is large vocabulary speech recognition in Persian (explained in the next sub-section) 
Based on the recording sessions, we created two experimental setups for speaker verification  In the first one, respondents with at least 17 recording sessions are included to the evaluation set, respondents with 16 sessions to the development and the rest of respondents to the background set (can be used as training data)  In the second setup, respondents with at least 8 sessions are included to the evaluation set, respondents with 6 or 7 sessions to the development and the rest of respondents to the background set  Table TABREF18 shows numbers of speakers in each set of the database for text-independent SV case 
For text-independent SV, we have considered 4 scenarios for enrollment and 4 scenarios for test  The speaker can be enrolled using utterances from 1, 2 or 3 consecutive sessions (1sess to 3sess) or using 8 utterances from 8 different sessions  The test speech can be one utterance (1utt) for short duration scenario or all utterances in one session (1sess) for long duration case  In addition, test speech can be selected from 5 English phrases for cross-language testing (enrollment using Persian utterances and test using English utterances)  From all setups, 1sess-1utt and 1sess-1sess for 438-spk set are considered as the main evaluation setups for text-independent case  Table TABREF19 shows number of trials for these setups 
For text-prompted SV with random text, the same setup as text-independent case together with corresponding utterance transcriptions can be used 
DeepMine Database Parts ::: Part3 - Speech Recognition
As explained before, Part3 of the DeepMine database can be used for Persian read speech recognition  There are only a few databases for speech recognition in Persian BIBREF6, BIBREF7  Hence, this part can at least partly address this problem and enable robust speech recognition applications in Persian  Additionally, it can be used for speaker recognition applications, such as training deep neural networks (DNNs) for extracting bottleneck features BIBREF8, or for collecting sufficient statistics using DNNs for i-vector training 
We have randomly selected 50 speakers (25 for each gender) from the all speakers in the database which have net speech (without silence parts) between 25 minutes to 50 minutes as test speakers  For each speaker, the utterances in the first 5 sessions are included to (small) test-set and the other utterances of test speakers are considered as a large-test-set  The remaining utterances of the other speakers are included in the training set  The test-set, large-test-set and train-set contain 5 9, 28 5 and 450 hours of speech respectively 
There are about 8300 utterances in Part3 which contain only Persian full names (i e  first and family name pairs)  Each phrase consists of several full names and their phoneme transcriptions were extracted automatically using a trained Grapheme-to-Phoneme (G2P)  These utterances can be used to evaluate the performance of a systems for name recognition, which is usually more difficult than the normal speech recognition because of the lack of a reliable language model 
Experiments and Results
Due to the space limitation, we present results only for the Persian text-dependent speaker verification and speech recognition 
Experiments and Results ::: Speaker Verification Experiments
We conducted an experiment on text-dependent speaker verification part of the database, using the i-vector based method proposed in BIBREF9, BIBREF10 and applied it to the Persian portion of Part1  In this experiment, 20-dimensional MFCC features along with first and second derivatives are extracted from 16 kHz signals using HTK BIBREF11 with 25 ms Hamming windowed frames with 15 ms overlap 
The reported results are obtained with a 400-dimensional gender independent i-vector based system  The i-vectors are first length-normalized and are further normalized using phrase- and gender-dependent Regularized Within-Class Covariance Normalization (RWCCN) BIBREF10  Cosine distance is used to obtain speaker verification scores and phrase- and gender-dependent s-norm is used for normalizing the scores  For aligning speech frames to Gaussian components, monophone HMMs with 3 states and 8 Gaussian components in each state are used BIBREF10  We only model the phonemes which appear in the 5 Persian text-dependent phrases 
For speaker verification experiments, the results were reported in terms of Equal Error Rate (EER) and Normalized Detection Cost Function as defined for NIST SRE08 ($\mathrm {NDCF_{0 01}^{min}}$) and NIST SRE10 ($\mathrm {NDCF_{0 001}^{min}}$)  As shown in Table TABREF22, in text-dependent SV there are 4 types of trials: Target-Correct and Imposter-Correct refer to trials when the pass-phrase is uttered correctly by target and imposter speakers respectively, and in same manner, Target-Wrong and Imposter-Wrong refer to trials when speakers uttered a wrong pass-phrase  In this paper, only the correct trials (i e  Target-Correct as target trials vs Imposter-Correct as non-target trials) are considered for evaluating systems as it has been proved that these are the most challenging trials in text-dependent SV BIBREF8, BIBREF12 
Table TABREF23 shows the results of text-dependent experiments using Persian 100-spk and 3-sess setup  For filtering trials, the respondents' mobile brand and model were used in this experiment  In the table, the first two letters in the filter notation relate to the target trials and the second two letters (i e  right side of the colon) relate for non-target trials  For target trials, the first Y means the enrolment and test utterances were recorded using a device with the same brand by the target speaker  The second Y letter means both recordings were done using exactly the same device model  Similarly, the first Y for non-target trials means that the devices of target and imposter speakers are from the same brand (i e  manufacturer)  The second Y means that, in addition to the same brand, both devices have the same model  So, the most difficult target trials are “NN”, where the speaker has used different a device at the test time  In the same manner, the most difficult non-target trials which should be rejected by the system are “YY” where the imposter speaker has used the same device model as the target speaker (note that it does not mean physically the same device because each speaker participated in the project using a personal mobile device)  Hence, the similarity in the recording channel makes rejection more difficult 
The first row in Table TABREF23 shows the results for all trials  By comparing the results with the best published results on RSR2015 and RedDots BIBREF10, BIBREF8, BIBREF12, it is clear that the DeepMine database is more challenging than both RSR2015 and RedDots databases  For RSR2015, the same i-vector/HMM-based method with both RWCCN and s-norm has achieved EER less than 0 3% for both genders (Table VI in BIBREF10)  The conventional Relevance MAP adaptation with HMM alignment without applying any channel-compensation techniques (i e  without applying RWCCN and s-norm due to the lack of suitable training data) on RedDots Part1 for the male has achieved EER around 1 5% (Table XI in BIBREF10)  It is worth noting that EERs for DeepMine database without any channel-compensation techniques are 2 1 and 3 7% for males and females respectively 
One interesting advantage of the DeepMine database compared to both RSR2015 and RedDots is having several target speakers with more than one mobile device  This is allows us to analyse the effects of channel compensation methods  The second row in Table TABREF23 corresponds to the most difficult trials where the target trials come from mobile devices with different models while imposter trials come from the same device models  It is clear that severe degradation was caused by this kind of channel effects (i e  decreasing within-speaker similarities while increasing between-speaker similarities), especially for females 
The results in the third row show the condition when target speakers at the test time use exactly the same device that was used for enrollment  Comparing this row with the results in the first row proves how much improvement can be achieved when exactly the same device is used by the target speaker 
The results in the fourth row show the condition when imposter speakers also use the same device model at test time to fool the system  So, in this case, there is no device mismatch in all trials  By comparing the results with the third row, we can see how much degradation is caused if we only consider the non-target trials with the same device 
The fifth row shows similar results when the imposter speakers use device of the same brand as the target speaker but with a different model  Surprisingly, in this case, the degradation is negligible and it means that mobiles from a specific brand (manufacturer) have different recording channel properties 
The degraded female results in the sixth row as compared to the third row show the effect of using a different device model from the same brand for target trials  For males, the filters brings almost the same subsets of trials, which explains the very similar results in this case 
Looking at the first two and the last row of Table TABREF23, one can notice the significantly worse performance obtained for the female trials as compared to males  Note that these three rows include target trials where the devices used for enrollment do not necessarily match the devices used for recording test utterances  On the other hand, in rows 3 to 6, which exclude such mismatched trials, the performance for males and females is comparable  This suggest that the degraded results for females are caused by some problematic trials with device mismatch  The exact reason for this degradation is so far unclear and needs a further investigation 
In the last row of the table, the condition of the second row is relaxed: the target device should have different model possibly from the same brand and imposter device only needs to be from the same brand  In this case, as was expected, the performance degradation is smaller than in the second row 
Experiments and Results ::: Speech Recognition Experiments
In addition to speaker verification, we present several speech recognition experiments on Part3  The experiments were performed with the Kaldi toolkit BIBREF13  For training HMM-based MonoPhone model, only 20 thousands of shortest utterances are used and for other models the whole training data is used  The DNN based acoustic model is a time-delay DNN with low-rank factorized layers and skip connections without i-vector adaptation (a modified network from one of the best performing LibriSpeech recipes)  The network is shown in Table TABREF25: there are 16 F-TDNN layers, with dimension 1536 and linear bottleneck layers of dimension 256  The acoustic model is trained for 10 epochs using lattice-free maximum mutual information (LF-MMI) with cross-entropy regularization BIBREF14  Re-scoring is done using a pruned trigram language model and the size of the dictionary is around 90,000 words 
Table TABREF26 shows the results in terms of word error rate (WER) for different evaluated methods  As can be seen, the created database can be used to train well performing and practically usable Persian ASR models 
Conclusions
In this paper, we have described the final version of a large speech corpus, the DeepMine database  It has been collected using crowdsourcing and, according to the best of our knowledge, it is the largest public text-dependent and text-prompted speaker verification database in two languages: Persian and English  In addition, it is the largest text-independent speaker verification evaluation database, making it suitable to robustly evaluate state-of-the-art methods on different conditions  Alongside these appealing properties, it comes with phone-level transcription, making it suitable to train deep neural network models for Persian speech recognition 
We provided several evaluation protocols for each part of the database  The protocols allow researchers to investigate the performance of different methods in various scenarios and study the effects of channels, duration and phrase text on the performance  We also provide two test sets for speech recognition: One normal test set with a few minutes of speech for each speaker and one large test set with more (30 minutes on average) speech that can be used for any speaker adaptation method 
As baseline results, we reported the performance of an i-vector/HMM based method on Persian text-dependent part  Moreover, we conducted speech recognition experiments using conventional HMM-based methods, as well as state-of-the-art deep neural network based method using Kaldi toolkit with promising performance  Text-dependent results have shown that the DeepMine database is more challenging than RSR2015 and RedDots databases 
Acknowledgments
The data collection project was mainly supported by Sharif DeepMine company  The work on the paper was supported by Czech National Science Foundation (GACR) project "NEUREM3" No  19-26934X and the National Programme of Sustainability (NPU II) project "IT4Innovations excellence in science - LQ1602" 
Context 2: Introduction
Recurrent neural networks (RNNs), including gated variants such as the long short-term memory (LSTM) BIBREF0 have become the standard model architecture for deep learning approaches to sequence modeling tasks  RNNs repeatedly apply a function with trainable parameters to a hidden state  Recurrent layers can also be stacked, increasing network depth, representational power and often accuracy  RNN applications in the natural language domain range from sentence classification BIBREF1 to word- and character-level language modeling BIBREF2   RNNs are also commonly the basic building block for more complex models for tasks such as machine translation BIBREF3 , BIBREF4 , BIBREF5 or question answering BIBREF6 , BIBREF7   Unfortunately standard RNNs, including LSTMs, are limited in their capability to handle tasks involving very long sequences, such as document classification or character-level machine translation, as the computation of features or states for different parts of the document cannot occur in parallel 
Convolutional neural networks (CNNs) BIBREF8 , though more popular on tasks involving image data, have also been applied to sequence encoding tasks BIBREF9   Such models apply time-invariant filter functions in parallel to windows along the input sequence  CNNs possess several advantages over recurrent models, including increased parallelism and better scaling to long sequences such as those often seen with character-level language data  Convolutional models for sequence processing have been more successful when combined with RNN layers in a hybrid architecture BIBREF10 , because traditional max- and average-pooling approaches to combining convolutional features across timesteps assume time invariance and hence cannot make full use of large-scale sequence order information 
We present quasi-recurrent neural networks for neural sequence modeling  QRNNs address both drawbacks of standard models: like CNNs, QRNNs allow for parallel computation across both timestep and minibatch dimensions, enabling high throughput and good scaling to long sequences  Like RNNs, QRNNs allow the output to depend on the overall order of elements in the sequence  We describe QRNN variants tailored to several natural language tasks, including document-level sentiment classification, language modeling, and character-level machine translation  These models outperform strong LSTM baselines on all three tasks while dramatically reducing computation time 
Model
Each layer of a quasi-recurrent neural network consists of two kinds of subcomponents, analogous to convolution and pooling layers in CNNs  The convolutional component, like convolutional layers in CNNs, allows fully parallel computation across both minibatches and spatial dimensions, in this case the sequence dimension  The pooling component, like pooling layers in CNNs, lacks trainable parameters and allows fully parallel computation across minibatch and feature dimensions 
Given an input sequence INLINEFORM0 of INLINEFORM1 INLINEFORM2 -dimensional vectors INLINEFORM3 , the convolutional subcomponent of a QRNN performs convolutions in the timestep dimension with a bank of INLINEFORM4 filters, producing a sequence INLINEFORM5 of INLINEFORM6 -dimensional candidate vectors INLINEFORM7   In order to be useful for tasks that include prediction of the next token, the filters must not allow the computation for any given timestep to access information from future timesteps  That is, with filters of width INLINEFORM8 , each INLINEFORM9 depends only on INLINEFORM10 through INLINEFORM11   This concept, known as a masked convolution BIBREF11 , is implemented by padding the input to the left by the convolution's filter size minus one 
We apply additional convolutions with separate filter banks to obtain sequences of vectors for the elementwise gates that are needed for the pooling function  While the candidate vectors are passed through a INLINEFORM0 nonlinearity, the gates use an elementwise sigmoid  If the pooling function requires a forget gate INLINEFORM1 and an output gate INLINEFORM2 at each timestep, the full set of computations in the convolutional component is then: DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , each in INLINEFORM3 , are the convolutional filter banks and INLINEFORM4 denotes a masked convolution along the timestep dimension  Note that if the filter width is 2, these equations reduce to the LSTM-like DISPLAYFORM0
Convolution filters of larger width effectively compute higher INLINEFORM0 -gram features at each timestep; thus larger widths are especially important for character-level tasks 
Suitable functions for the pooling subcomponent can be constructed from the familiar elementwise gates of the traditional LSTM cell  We seek a function controlled by gates that can mix states across timesteps, but which acts independently on each channel of the state vector  The simplest option, which BIBREF12 term “dynamic average pooling”, uses only a forget gate: DISPLAYFORM0
We term these three options f-pooling, fo-pooling, and ifo-pooling respectively; in each case we initialize INLINEFORM0 or INLINEFORM1 to zero  Although the recurrent parts of these functions must be calculated for each timestep in sequence, their simplicity and parallelism along feature dimensions means that, in practice, evaluating them over even long sequences requires a negligible amount of computation time 
A single QRNN layer thus performs an input-dependent pooling, followed by a gated linear combination of convolutional features  As with convolutional neural networks, two or more QRNN layers should be stacked to create a model with the capacity to approximate more complex functions 
Variants
Motivated by several common natural language tasks, and the long history of work on related architectures, we introduce several extensions to the stacked QRNN described above  Notably, many extensions to both recurrent and convolutional models can be applied directly to the QRNN as it combines elements of both model types 
Regularization An important extension to the stacked QRNN is a robust regularization scheme inspired by recent work in regularizing LSTMs 
The need for an effective regularization method for LSTMs, and dropout's relative lack of efficacy when applied to recurrent connections, led to the development of recurrent dropout schemes, including variational inference–based dropout BIBREF13 and zoneout BIBREF14   These schemes extend dropout to the recurrent setting by taking advantage of the repeating structure of recurrent networks, providing more powerful and less destructive regularization 
Variational inference–based dropout locks the dropout mask used for the recurrent connections across timesteps, so a single RNN pass uses a single stochastic subset of the recurrent weights  Zoneout stochastically chooses a new subset of channels to “zone out” at each timestep; for these channels the network copies states from one timestep to the next without modification 
As QRNNs lack recurrent weights, the variational inference approach does not apply  Thus we extended zoneout to the QRNN architecture by modifying the pooling function to keep the previous pooling state for a stochastic subset of channels  Conveniently, this is equivalent to stochastically setting a subset of the QRNN's INLINEFORM0 gate channels to 1, or applying dropout on INLINEFORM1 : DISPLAYFORM0
Thus the pooling function itself need not be modified at all  We note that when using an off-the-shelf dropout layer in this context, it is important to remove automatic rescaling functionality from the implementation if it is present  In many experiments, we also apply ordinary dropout between layers, including between word embeddings and the first QRNN layer 
Densely-Connected Layers We can also extend the QRNN architecture using techniques introduced for convolutional networks  For sequence classification tasks, we found it helpful to use skip-connections between every QRNN layer, a technique termed “dense convolution” by BIBREF15   Where traditional feed-forward or convolutional networks have connections only between subsequent layers, a “DenseNet” with INLINEFORM0 layers has feed-forward or convolutional connections between every pair of layers, for a total of INLINEFORM1   This can improve gradient flow and convergence properties, especially in deeper networks, although it requires a parameter count that is quadratic in the number of layers 
When applying this technique to the QRNN, we include connections between the input embeddings and every QRNN layer and between every pair of QRNN layers  This is equivalent to concatenating each QRNN layer's input to its output along the channel dimension before feeding the state into the next layer  The output of the last layer alone is then used as the overall encoding result 
Encoder–Decoder Models To demonstrate the generality of QRNNs, we extend the model architecture to sequence-to-sequence tasks, such as machine translation, by using a QRNN as encoder and a modified QRNN, enhanced with attention, as decoder  The motivation for modifying the decoder is that simply feeding the last encoder hidden state (the output of the encoder's pooling layer) into the decoder's recurrent pooling layer, analogously to conventional recurrent encoder–decoder architectures, would not allow the encoder state to affect the gate or update values that are provided to the decoder's pooling layer  This would substantially limit the representational power of the decoder 
Instead, the output of each decoder QRNN layer's convolution functions is supplemented at every timestep with the final encoder hidden state  This is accomplished by adding the result of the convolution for layer INLINEFORM0 (e g , INLINEFORM1 , in INLINEFORM2 ) with broadcasting to a linearly projected copy of layer INLINEFORM3 's last encoder state (e g , INLINEFORM4 , in INLINEFORM5 ): DISPLAYFORM0
where the tilde denotes that INLINEFORM0 is an encoder variable  Encoder–decoder models which operate on long sequences are made significantly more powerful with the addition of soft attention BIBREF3 , which removes the need for the entire input representation to fit into a fixed-length encoding vector  In our experiments, we computed an attentional sum of the encoder's last layer's hidden states  We used the dot products of these encoder hidden states with the decoder's last layer's un-gated hidden states, applying a INLINEFORM1 along the encoder timesteps, to weight the encoder states into an attentional sum INLINEFORM2 for each decoder timestep  This context, and the decoder state, are then fed into a linear layer followed by the output gate: DISPLAYFORM0
where INLINEFORM0 is the last layer 
While the first step of this attention procedure is quadratic in the sequence length, in practice it takes significantly less computation time than the model's linear and convolutional layers due to the simple and highly parallel dot-product scoring function 
Experiments
We evaluate the performance of the QRNN on three different natural language tasks: document-level sentiment classification, language modeling, and character-based neural machine translation  Our QRNN models outperform LSTM-based models of equal hidden size on all three tasks while dramatically improving computation speed  Experiments were implemented in Chainer BIBREF16  
Sentiment Classification
We evaluate the QRNN architecture on a popular document-level sentiment classification benchmark, the IMDb movie review dataset BIBREF17   The dataset consists of a balanced sample of 25,000 positive and 25,000 negative reviews, divided into equal-size train and test sets, with an average document length of 231 words BIBREF18   We compare only to other results that do not make use of additional unlabeled data (thus excluding e g , BIBREF19 ) 
Our best performance on a held-out development set was achieved using a four-layer densely-connected QRNN with 256 units per layer and word vectors initialized using 300-dimensional cased GloVe embeddings BIBREF20   Dropout of 0 3 was applied between layers, and we used INLINEFORM0 regularization of INLINEFORM1   Optimization was performed on minibatches of 24 examples using RMSprop BIBREF21 with learning rate of INLINEFORM2 , INLINEFORM3 , and INLINEFORM4  
Small batch sizes and long sequence lengths provide an ideal situation for demonstrating the QRNN's performance advantages over traditional recurrent architectures
Context 3: Introduction
Accurate grapheme-to-phoneme conversion (g2p) is important for any application that depends on the sometimes inconsistent relationship between spoken and written language  Most prominently, this includes text-to-speech and automatic speech recognition  Most work on g2p has focused on a few languages for which extensive pronunciation data is available BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   Most languages lack these resources  However, a low resource language's writing system is likely to be similar to the writing systems of languages that do have sufficient pronunciation data  Therefore g2p may be possible for low resource languages if this high resource data can be properly utilized 
We attempt to leverage high resource data by treating g2p as a multisource neural machine translation (NMT) problem  The source sequences for our system are words in the standard orthography in any language  The target sequences are the corresponding representation in the International Phonetic Alphabet (IPA)  Our results show that the parameters learned by the shared encoder–decoder are able to exploit the orthographic and phonemic similarities between the various languages in our data 
Low Resource g2p
Our approach is similar in goal to deri2016grapheme's model for adapting high resource g2p models for low resource languages  They trained weighted finite state transducer (wFST) models on a variety of high resource languages, then transferred those models to low resource languages, using a language distance metric to choose which high resource models to use and a phoneme distance metric to map the high resource language's phonemes to the low resource language's phoneme inventory  These distance metrics are computed based on data from Phoible BIBREF4 and URIEL BIBREF5  
Other low resource g2p systems have used a strategy of combining multiple models  schlippe2014combining trained several data-driven g2p systems on varying quantities of monolingual data and combined their outputs with a phoneme-level voting scheme  This led to improvements over the best-performing single system for small quantities of data in some languages  jyothilow trained recurrent neural networks for small data sets and found that a version of their system that combined the neural network output with the output of the wFST-based Phonetisaurus system BIBREF1 did better than either system alone 
A different approach came from kim2012universal, who used supervised learning with an undirected graphical model to induce the grapheme–phoneme mappings for languages written in the Latin alphabet  Given a short text in a language, the model predicts the language's orthographic rules  To create phonemic context features from the short text, the model naïvely maps graphemes to IPA symbols written with the same character, and uses the features of these symbols to learn an approximation of the phonotactic constraints of the language  In their experiments, these phonotactic features proved to be more valuable than geographical and genetic features drawn from WALS BIBREF6  
Multilingual Neural NLP
In recent years, neural networks have emerged as a common way to use data from several languages in a single system  Google's zero-shot neural machine translation system BIBREF7 shares an encoder and decoder across all language pairs  In order to facilitate this multi-way translation, they prepend an artificial token to the beginning of each source sentence at both training and translation time  The token identifies what language the sentence should be translated to  This approach has three benefits: it is far more efficient than building a separate model for each language pair; it allows for translation between languages that share no parallel data; and it improves results on low-resource languages by allowing them to implicitly share parameters with high-resource languages  Our g2p system is inspired by this approach, although it differs in that there is only one target “language”, IPA, and the artificial tokens identify the language of the source instead of the language of the target 
Other work has also made use of multilingually-trained neural networks  Phoneme-level polyglot language models BIBREF8 train a single model on multiple languages and additionally condition on externally constructed typological data about the language  ostling2017continuous used a similar approach, in which a character-level neural language model is trained on a massively multilingual corpus  A language embedding vector is concatenated to the input at each time step  The language embeddings their system learned correlate closely to the genetic relationships between languages  However, neither of these models was applied to g2p 
Grapheme-to-Phoneme
g2p is the problem of converting the orthographic representation of a word into a phonemic representation  A phoneme is an abstract unit of sound which may have different realizations in different contexts  For example, the English phoneme has two phonetic realizations (or allophones):
English speakers without linguistic training often struggle to perceive any difference between these sounds  Writing systems usually do not distinguish between allophones: and are both written as INLINEFORM0 p INLINEFORM1 in English  The sounds are written differently in languages where they contrast, such as Hindi and Eastern Armenian 
Most writing systems in use today are glottographic, meaning that their symbols encode solely phonological information  But despite being glottographic, in few writing systems do graphemes correspond one-to-one with phonemes  There are cases in which multiple graphemes represent a single phoneme, as in the word the in English:
There are cases in which a single grapheme represents multiple phonemes, such as syllabaries, in which each symbol represents a syllable 
In many languages, there are silent letters, as in the word hora in Spanish:
There are more complicated correspondences, such as the silent e in English that affects the pronunciation of the previous vowel, as seen in the pair of words cape and cap 
It is possible for an orthographic system to have any or all of the above phenomena while remaining unambiguous  However, some orthographic systems contain ambiguities  English is well-known for its spelling ambiguities  Abjads, used for Arabic and Hebrew, do not give full representation to vowels 
Consequently, g2p is harder than simply replacing each grapheme symbol with a corresponding phoneme symbol  It is the problem of replacing a grapheme sequence INLINEFORM0
with a phoneme sequence INLINEFORM0
where the sequences are not necessarily of the same length  Data-driven g2p is therefore the problem of finding the phoneme sequence that maximizes the likelihood of the grapheme sequence: INLINEFORM0
Data-driven approaches are especially useful for problems in which the rules that govern them are complex and difficult to engineer by hand  g2p for languages with ambiguous orthographies is such a problem  Multilingual g2p, in which the various languages have similar but different and possibly contradictory spelling rules, can be seen as an extreme case of that  Therefore, a data-driven sequence-to-sequence model is a natural choice 
Encoder–Decoder Models
In order to find the best phoneme sequence, we use a neural encoder–decoder model with attention BIBREF9   The model consists of two main parts: the encoder compresses each source grapheme sequence INLINEFORM0 into a fixed-length vector  The decoder, conditioned on this fixed-length vector, generates the output phoneme sequence INLINEFORM1  
The encoder and decoder are both implemented as recurrent neural networks, which have the advantage of being able to process sequences of arbitrary length and use long histories efficiently  They are trained jointly to minimize cross-entropy on the training data  We had our best results when using a bidirectional encoder, which consists of two separate encoders which process the input in forward and reverse directions  We used long short-term memory units BIBREF10 for both the encoder and decoder  For the attention mechanism, we used the general global attention architecture described by luong2015effective 
We implemented all models with OpenNMT BIBREF11   Our hyperparameters, which we determined by experimentation, are listed in Table TABREF8  
Training Multilingual Models
Presenting pronunciation data in several languages to the network might create problems because different languages have different pronunciation patterns  For example, the string `real' is pronounced differently in English, German, Spanish, and Portuguese  We solve this problem by prepending each grapheme sequence with an artificial token consisting of the language's ISO 639-3 code enclosed in angle brackets  The English word `real', for example, would be presented to the system as
INLINEFORM0 eng INLINEFORM1 r e a l
The artificial token is treated simply as an element of the grapheme sequence  This is similar to the approach taken by johnson2016google in their zero-shot NMT system  However, their source-side artificial tokens identify the target language, whereas ours identify the source language  An alternative approach, used by ostling2017continuous, would be to concatenate a language embedding to the input at each time step  They do not evaluate their approach on grapheme-to-phoneme conversion 
Data
In order to train a neural g2p system, one needs a large quantity of pronunciation data  A standard dataset for g2p is the Carnegie Mellon Pronouncing Dictionary BIBREF12   However, that is a monolingual English resource, so it is unsuitable for our multilingual task  Instead, we use the multilingual pronunciation corpus collected by deri2016grapheme for all experiments  This corpus consists of spelling–pronunciation pairs extracted from Wiktionary  It is already partitioned into training and test sets  Corpus statistics are presented in Table TABREF10  
In addition to the raw IPA transcriptions extracted from Wiktionary, the corpus provides an automatically cleaned version of transcriptions  Cleaning is a necessary step because web-scraped data is often noisy and may be transcribed at an inconsistent level of detail  The data cleaning used here attempts to make the transcriptions consistent with the phonemic inventories used in Phoible BIBREF4   When a transcription contains a phoneme that is not in its language's inventory in Phoible, that phoneme is replaced by the phoneme with the most similar articulatory features that is in the language's inventory  Sometimes this cleaning algorithm works well: in the German examples in Table TABREF11 , the raw German symbols and are both converted to   This is useful because the in Ansbach and the in Kaninchen are instances of the same phoneme, so their phonemic representations should use the same symbol  However, the cleaning algorithm can also have negative effects on the data quality  For example, the phoneme is not present in the Phoible inventory for German, but it is used in several German transcriptions in the corpus  The cleaning algorithm converts to in all German transcriptions, whereas would be a more reasonable guess  The cleaning algorithm also removes most suprasegmentals, even though these are often an important part of a language's phonology  Developing a more sophisticated procedure for cleaning pronunciation data is a direction for future work, but in this paper we use the corpus's provided cleaned transcriptions in order to ease comparison to previous results 
Experiments
We present experiments with two versions of our sequence-to-sequence model  LangID prepends each training, validation, and test sample with an artificial token identifying the language of the sample  NoLangID omits this token  LangID and NoLangID have identical structure otherwise  To translate the test corpus, we used a beam width of 100  Although this is an unusually wide beam and had negligible performance effects, it was necessary to compute our error metrics 
Evaluation
We use the following three evaluation metrics:
Phoneme Error Rate (PER) is the Levenshtein distance between the predicted phoneme sequences and the gold standard phoneme sequences, divided by the length of the gold standard phoneme sequences 
Word Error Rate (WER) is the percentage of words in which the predicted phoneme sequence does not exactly match the gold standard phoneme sequence 
Word Error Rate 100 (WER 100) is the percentage of words in the test set for which the correct guess is not in the first 100 guesses of the system 
In system evaluations, WER, WER 100, and PER numbers presented for multiple languages are averaged, weighting each language equally BIBREF13  
It would be interesting to compute error metrics that incorporate phoneme similarity, such as those proposed by hixon2011phonemic  PER weights all phoneme errors the same, even though some errors are more harmful than others: and are usually contrastive, whereas and almost never are  Such statistics would be especially interesting for evaluating a multilingual system, because different languages often map the same grapheme to phonemes that are only subtly different from each other  However, these statistics have not been widely reported for other g2p systems, so we omit them here 
Baseline
Results on LangID and NoLangID are compared to the system presented by deri2016grapheme, which is identified in our results as wFST  Their results can be divided into two parts:
High resource results, computed with wFSTs trained on a combination of Wiktionary pronunciation data and g2p rules extracted from Wikipedia IPA Help pages  They report high resource results for 85 languages 
Adapted results, where they apply various mapping strategies in order to adapt high resource models to other languages  The final adapted results they reported include most of the 85 languages with high resource results, as well as the various languages they were able to adapt them for, for a total of 229 languages  This test set omits 23 of the high resource languages that are written in unique scripts or for which language distance metrics could not be computed 
Training
We train the LangID and NoLangID versions of our model each on three subsets of the Wiktionary data:
LangID-High and NoLangID-High: Trained on data from the 85 languages for which BIBREF13 used non-adapted wFST models 
LangID-Adapted and NoLangID-Adapted: Trained on data from any of the 229 languages for which they built adapted models  Because many of these languages had no training data at all, the model is actually only trained on data in 157 languages  As is noted above, the Adapted set omits 23 languages which are in the High test set 
LangID-All and NoLangID-All: Trained on data in all 311 languages in the Wiktionary training corpus 
In order to ease comparison to Deri and Knight's system, we limited our use of the training corpus to 10,000 words per language  We set aside 10 percent of the data in each language for validation, so the maximum number of training words for any language is 9000 for our systems 
Adapted Results
On the 229 languages for which deri2016grapheme presented their final results, the LangID version of our system outperforms the baseline by a wide margin  The best performance came with the version of our model that was trained on data in all available languages, not just the languages it was tested on  Using a language ID token improves results considerably, but even NoLangID beats the baseline in WER and WER 100  Full results are presented in Table TABREF24  
High Resource Results
Having shown that our model exceeds the performance of the wFST-adaptation approach, we next compare it to the baseline models for just high resource languages  The wFST models here are purely monolingual – they do not use data adaptation because there is sufficient training data for each of them  Full results are presented in Table TABREF26   We omit models trained on the Adapted languages because they were not trained on high resource languages with unique writing systems, such as Georgian and Greek, and consequently performed very poorly on them 
In contrast to the larger-scale Adapted results, in the High Resource experiments none of the sequence-to-sequence approaches equal the performance of the wFST model in WER and PER, although LangID-High does come close  The LangID models do beat wFST in WER 100  A possible explanation is that a monolingual wFST model will never generate phonemes that are not part of the language's inventory  A multilingual model, on the other hand, could potentially generate phonemes from the inventories of any language it has been trained on 
Even if LangID-High does not present a more accurate result, it does present a more compact one: LangID-High is 15 4 MB, while the combined wFST high resource models are 197 5 MB 
Results on Unseen Languages
Finally, we report our models' results on unseen languages in Table TABREF28   The unseen languages are any that are present in the test corpus but absent from the training data  Deri and Knight did not report results specifically on these languages  Although the NoLangID models sometimes do better on WER 100, even here the LangID models have a slight advantage in WER and PER  This is somewhat surprising because the LangID models have not learned embeddings for the language ID tokens of unseen languages  Perhaps negative associations are also being learned, driving the model towards predicting more common pronunciations for unseen languages 
Language ID Tokens
Adding a language ID token always improves results in cases where an embedding has been learned for that token  The power of these embeddings is demonstrated by what happens when one feeds the same input word to the model with different language tokens, as is seen in Table TABREF30   Impressively, this even works when the source sequence is in the wrong script for the language, as is seen in the entry for Arabic 
Language Embeddings
Because these language ID tokens are so useful, it would be good if they could be effectively estimated for unseen languages  ostling2017continuous found that the language vectors their models learned correlated well to genetic relationships, so it would be interesting to see if the embeddings our source encoder learned for the language ID tokens showed anything similar  In a few cases they do (the languages closest to German in the vector space are Luxembourgish, Bavarian, and Yiddish, all close relatives)  However, for the most part the structure of these vectors is not interpretable  Therefore, it would be difficult to estimate the embedding for an unseen language, or to “borrow” the language ID token of a similar language  A more promising way forward is to find a model that uses an externally constructed typological representation of the language 
Phoneme Embeddings
In contrast to the language embeddings, the phoneme embeddings appear to show many regularities (see Table TABREF33 )  This is a sign that our multilingual model learns similar embeddings for phonemes that are written with the same grapheme in different languages  These phonemes tend to be phonetically similar to each other 
Perhaps the structure of the phoneme embedding space is what leads to our models' very good performance on WER 100  Even when the model's first predicted pronunciation is not correct, it tends to assign more probability mass to guesses that are more similar to the correct one  Applying some sort of filtering or reranking of the system output might therefore lead to better performance 
Future Work
Because the language ID token is so beneficial to performance, it would be very interesting to find ways to extend a similar benefit to unseen languages  One possible way to do so is with tokens that identify something other than the language, such as typological features about the language's phonemic inventory  This could enable better sharing of resources among languages  Such typological knowledge is readily available in databases like Phoible and WALS for a wide variety of languages  It would be interesting to explore if any of these features is a good predictor of a language's orthographic rules 
It would also be interesting to apply the artificial token approach to other problems besides multilingual g2p  One closely related application is monolingual English g2p  Some of the ambiguity of English spelling is due to the wide variety of loanwords in the language, many of which have unassimilated spellings  Knowing the origins of these loanwords could provide a useful hint for figuring out their pronunciations  The etymology of a word could be tagged in an analogous way to how language ID is tagged in multilingual g2p 

ground_truth: Android application

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What sentiment classification dataset is used?
Context 2:  We observed a speedup of 3 2x on IMDb train time per epoch compared to the optimized LSTM implementation provided in NVIDIA's cuDNN library  For specific batch sizes and sequence lengths, a 16x speed gain is possible  Figure FIGREF15 provides extensive speed comparisons 
In Figure FIGREF12 , we visualize the hidden state vectors INLINEFORM0 of the final QRNN layer on part of an example from the IMDb dataset  Even without any post-processing, changes in the hidden state are visible and interpretable in regards to the input  This is a consequence of the elementwise nature of the recurrent pooling function, which delays direct interaction between different channels of the hidden state until the computation of the next QRNN layer 
Language Modeling
We replicate the language modeling experiment of BIBREF2 and BIBREF13 to benchmark the QRNN architecture for natural language sequence prediction  The experiment uses a standard preprocessed version of the Penn Treebank (PTB) by BIBREF25  
We implemented a gated QRNN model with medium hidden size: 2 layers with 640 units in each layer  Both QRNN layers use a convolutional filter width INLINEFORM0 of two timesteps  While the “medium” models used in other work BIBREF2 , BIBREF13 consist of 650 units in each layer, it was more computationally convenient to use a multiple of 32  As the Penn Treebank is a relatively small dataset, preventing overfitting is of considerable importance and a major focus of recent research  It is not obvious in advance which of the many RNN regularization schemes would perform well when applied to the QRNN  Our tests showed encouraging results from zoneout applied to the QRNN's recurrent pooling layer, implemented as described in Section SECREF5  
The experimental settings largely followed the “medium” setup of BIBREF2   Optimization was performed by stochastic gradient descent (SGD) without momentum  The learning rate was set at 1 for six epochs, then decayed by 0 95 for each subsequent epoch, for a total of 72 epochs  We additionally used INLINEFORM0 regularization of INLINEFORM1 and rescaled gradients with norm above 10  Zoneout was applied by performing dropout with ratio 0 1 on the forget gates of the QRNN, without rescaling the output of the dropout function  Batches consist of 20 examples, each 105 timesteps 
Comparing our results on the gated QRNN with zoneout to the results of LSTMs with both ordinary and variational dropout in Table TABREF14 , we see that the QRNN is highly competitive  The QRNN without zoneout strongly outperforms both our medium LSTM and the medium LSTM of BIBREF2 which do not use recurrent dropout and is even competitive with variational LSTMs  This may be due to the limited computational capacity that the QRNN's pooling layer has relative to the LSTM's recurrent weights, providing structural regularization over the recurrence 
Without zoneout, early stopping based upon validation loss was required as the QRNN would begin overfitting  By applying a small amount of zoneout ( INLINEFORM0 ), no early stopping is required and the QRNN achieves competitive levels of perplexity to the variational LSTM of BIBREF13 , which had variational inference based dropout of 0 2 applied recurrently  Their best performing variation also used Monte Carlo (MC) dropout averaging at test time of 1000 different masks, making it computationally more expensive to run 
When training on the PTB dataset with an NVIDIA K40 GPU, we found that the QRNN is substantially faster than a standard LSTM, even when comparing against the optimized cuDNN LSTM  In Figure FIGREF15 we provide a breakdown of the time taken for Chainer's default LSTM, the cuDNN LSTM, and QRNN to perform a full forward and backward pass on a single batch during training of the RNN LM on PTB  For both LSTM implementations, running time was dominated by the RNN computations, even with the highly optimized cuDNN implementation  For the QRNN implementation, however, the “RNN” layers are no longer the bottleneck  Indeed, there are diminishing returns from further optimization of the QRNN itself as the softmax and optimization overhead take equal or greater time  Note that the softmax, over a vocabulary size of only 10,000 words, is relatively small; for tasks with larger vocabularies, the softmax would likely dominate computation time 
It is also important to note that the cuDNN library's RNN primitives do not natively support any form of recurrent dropout  That is, running an LSTM that uses a state-of-the-art regularization scheme at cuDNN-like speeds would likely require an entirely custom kernel 
Character-level Neural Machine Translation
We evaluate the sequence-to-sequence QRNN architecture described in SECREF5 on a challenging neural machine translation task, IWSLT German–English spoken-domain translation, applying fully character-level segmentation  This dataset consists of 209,772 sentence pairs of parallel training data from transcribed TED and TEDx presentations, with a mean sentence length of 103 characters for German and 93 for English  We remove training sentences with more than 300 characters in English or German, and use a unified vocabulary of 187 Unicode code points 
Our best performance on a development set (TED tst2013) was achieved using a four-layer encoder–decoder QRNN with 320 units per layer, no dropout or INLINEFORM0 regularization, and gradient rescaling to a maximum magnitude of 5  Inputs were supplied to the encoder reversed, while the encoder convolutions were not masked  The first encoder layer used convolutional filter width INLINEFORM1 , while the other encoder layers used INLINEFORM2   Optimization was performed for 10 epochs on minibatches of 16 examples using Adam BIBREF28 with INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , and INLINEFORM6   Decoding was performed using beam search with beam width 8 and length normalization INLINEFORM7   The modified log-probability ranking criterion is provided in the appendix 
Results using this architecture were compared to an equal-sized four-layer encoder–decoder LSTM with attention, applying dropout of 0 2  We again optimized using Adam; other hyperparameters were equal to their values for the QRNN and the same beam search procedure was applied  Table TABREF17 shows that the QRNN outperformed the character-level LSTM, almost matching the performance of a word-level attentional baseline 
Related Work
Exploring alternatives to traditional RNNs for sequence tasks is a major area of current research  Quasi-recurrent neural networks are related to several such recently described models, especially the strongly-typed recurrent neural networks (T-RNN) introduced by BIBREF12   While the motivation and constraints described in that work are different, BIBREF12 's concepts of “learnware” and “firmware” parallel our discussion of convolution-like and pooling-like subcomponents  As the use of a fully connected layer for recurrent connections violates the constraint of “strong typing”, all strongly-typed RNN architectures (including the T-RNN, T-GRU, and T-LSTM) are also quasi-recurrent  However, some QRNN models (including those with attention or skip-connections) are not “strongly typed”  In particular, a T-RNN differs from a QRNN as described in this paper with filter size 1 and f-pooling only in the absence of an activation function on INLINEFORM0   Similarly, T-GRUs and T-LSTMs differ from QRNNs with filter size 2 and fo- or ifo-pooling respectively in that they lack INLINEFORM1 on INLINEFORM2 and use INLINEFORM3 rather than sigmoid on INLINEFORM4  
The QRNN is also related to work in hybrid convolutional–recurrent models  BIBREF31 apply CNNs at the word level to generate INLINEFORM0 -gram features used by an LSTM for text classification  BIBREF32 also tackle text classification by applying convolutions at the character level, with a stride to reduce sequence length, then feeding these features into a bidirectional LSTM  A similar approach was taken by BIBREF10 for character-level machine translation  Their model's encoder uses a convolutional layer followed by max-pooling to reduce sequence length, a four-layer highway network, and a bidirectional GRU  The parallelism of the convolutional, pooling, and highway layers allows training speed comparable to subword-level models without hard-coded text segmentation 
The QRNN encoder–decoder model shares the favorable parallelism and path-length properties exhibited by the ByteNet BIBREF33 , an architecture for character-level machine translation based on residual convolutions over binary trees  Their model was constructed to achieve three desired properties: parallelism, linear-time computational complexity, and short paths between any pair of words in order to better propagate gradient signals 
Conclusion
Intuitively, many aspects of the semantics of long sequences are context-invariant and can be computed in parallel (e g , convolutionally), but some aspects require long-distance context and must be computed recurrently  Many existing neural network architectures either fail to take advantage of the contextual information or fail to take advantage of the parallelism  QRNNs exploit both parallelism and context, exhibiting advantages from both convolutional and recurrent neural networks  QRNNs have better predictive accuracy than LSTM-based models of equal hidden size, even though they use fewer parameters and run substantially faster  Our experiments show that the speed and accuracy advantages remain consistent across tasks and at both word and character levels 
Extensions to both CNNs and RNNs are often directly applicable to the QRNN, while the model's hidden states are more interpretable than those of other recurrent architectures as its channels maintain their independence across timesteps  We believe that QRNNs can serve as a building block for long-sequence tasks that were previously impractical with traditional RNNs 
Beam search ranking criterion
The modified log-probability ranking criterion we used in beam search for translation experiments is: DISPLAYFORM0
where INLINEFORM0 is a length normalization parameter BIBREF34 , INLINEFORM1 is the INLINEFORM2 th output character, and INLINEFORM3 is a “target length” equal to the source sentence length plus five characters  This reduces at INLINEFORM4 to ordinary beam search with probabilities: DISPLAYFORM0
and at INLINEFORM0 to beam search with probabilities normalized by length (up to the target length): DISPLAYFORM0
Conveniently, this ranking criterion can be computed at intermediate beam-search timesteps, obviating the need to apply a separate reranking on complete hypotheses 

ground_truth: the IMDb movie review dataset BIBREF17

IMDb movie review

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what datasets did they use?
Context 4:  This way, we obtain a synthetic parallel simplified-ordinary sentences  Both the synthetic sentences and the available parallel data are used as training data for the original NMT system 
Evaluation
We evaluate the performance of text simplification using neural machine translation on available parallel sentences and additional simplified sentences 
Dataset  We use two simplification datasets (WikiSmall and WikiLarge)  WikiSmall consists of ordinary and simplified sentences from the ordinary and simple English Wikipedias, which has been used as benchmark for evaluating text simplification BIBREF17 , BIBREF18 , BIBREF8   The training set has 89,042 sentence pairs, and the test set has 100 pairs  WikiLarge is also from Wikipedia corpus whose training set contains 296,402 sentence pairs BIBREF19 , BIBREF20   WikiLarge includes 8 (reference) simplifications for 2,359 sentences split into 2,000 for development and 359 for testing 
Metrics  Three metrics in text simplification are chosen in this paper  BLEU BIBREF5 is one traditional machine translation metric to assess the degree to which translated simplifications differed from reference simplifications  FKGL measures the readability of the output BIBREF23   A small FKGL represents simpler output  SARI is a recent text-simplification metric by comparing the output against the source and reference simplifications BIBREF20  
We evaluate the output of all systems using human evaluation  The metric is denoted as Simplicity BIBREF8   The three non-native fluent English speakers are shown reference sentences and output sentences  They are asked whether the output sentence is much simpler (+2), somewhat simpler (+1), equally (0), somewhat more difficult (-1), and much more difficult (-2) than the reference sentence 
Methods  We use OpenNMT BIBREF24 as the implementation of the NMT system for all experiments BIBREF5   We generally follow the default settings and training procedure described by Klein et al (2017)  We replace out-of-vocabulary words with a special UNK symbol  At prediction time, we replace UNK words with the highest probability score from the attention layer  OpenNMT system used on parallel data is the baseline system  To obtain a synthetic parallel training set, we back-translate a random sample of 100K sentences from the collected simplified corpora  OpenNMT used on parallel data and synthetic data is our model  The benchmarks are run on a Intel(R) Core(TM) i7-5930K CPU@3 50GHz, 32GB Mem, trained on 1 GPU GeForce GTX 1080 (Pascal) with CUDA v  8 0 
We choose three statistical text simplification systems  PBMT-R is a phrase-based method with a reranking post-processing step BIBREF18   Hybrid performs sentence splitting and deletion operations based on discourse representation structures, and then simplifies sentences with PBMT-R BIBREF25   SBMT-SARI BIBREF19 is syntax-based translation model using PPDB paraphrase database BIBREF26 and modifies tuning function (using SARI)  We choose two neural text simplification systems  NMT is a basic attention-based encoder-decoder model which uses OpenNMT framework to train with two LSTM layers, hidden states of size 500 and 500 hidden units, SGD optimizer, and a dropout rate of 0 3 BIBREF8   Dress is an encoder-decoder model coupled with a deep reinforcement learning framework, and the parameters are chosen according to the original paper BIBREF20   For the experiments with synthetic parallel data, we back-translate a random sample of 60 000 sentences from the collected simplified sentences into ordinary sentences  Our model is trained on synthetic data and the available parallel data, denoted as NMT+synthetic 
Results  Table 1 shows the results of all models on WikiLarge dataset  We can see that our method (NMT+synthetic) can obtain higher BLEU, lower FKGL and high SARI compared with other models, except Dress on FKGL and SBMT-SARI on SARI  It verified that including synthetic data during training is very effective, and yields an improvement over our baseline NMF by 2 11 BLEU, 1 7 FKGL and 1 07 SARI  We also substantially outperform Dress, who previously reported SOTA result  The results of our human evaluation using Simplicity are also presented in Table 1  NMT on synthetic data is significantly better than PBMT-R, Dress, and SBMT-SARI on Simplicity  It indicates that our method with simplified data is effective at creating simpler output 
Results on WikiSmall dataset are shown in Table 2  We see substantial improvements (6 37 BLEU) than NMT from adding simplified training data with synthetic ordinary sentences  Compared with statistical machine translation models (PBMT-R, Hybrid, SBMT-SARI), our method (NMT+synthetic) still have better results, but slightly worse FKGL and SARI  Similar to the results in WikiLarge, the results of our human evaluation using Simplicity outperforms the other models  In conclusion, Our method produces better results comparing with the baselines, which demonstrates the effectiveness of adding simplified training data 
Conclusion
In this paper, we propose one simple method to use simplified corpora during training of NMT systems, with no changes to the network architecture  In the experiments on two datasets, we achieve substantial gains in all tasks, and new SOTA results, via back-translation of simplified sentences into the ordinary sentences, and treating this synthetic data as additional training data  Because we do not change the neural network architecture to integrate simplified corpora, our method can be easily applied to other Neural Text Simplification (NTS) systems  We expect that the effectiveness of our method not only varies with the quality of the NTS system used for back-translation, but also depends on the amount of available parallel and simplified corpora  In the paper, we have only utilized data from Wikipedia for simplified sentences  In the future, many other text sources are available and the impact of not only size, but also of domain should be investigated 

ground_truth: the Carnegie Mellon Pronouncing Dictionary BIBREF12, the multilingual pronunciation corpus collected by deri2016grapheme , ranscriptions extracted from Wiktionary

multilingual pronunciation corpus collected by deri2016grapheme