prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the baselines?
Context 1: Introduction
Speech-to-Text translation (ST) is essential for a wide range of scenarios: for example in emergency calls, where agents have to respond emergent requests in a foreign language BIBREF0; or in online courses, where audiences and speakers use different languages BIBREF1  To tackle this problem, existing approaches can be categorized into cascaded method BIBREF2, BIBREF3, where a machine translation (MT) model translates outputs of an automatic speech recognition (ASR) system into target language, and end-to-end method BIBREF4, BIBREF5, where a single model learns acoustic frames to target word sequence mappings in one step towards the final objective of interest  Although the cascaded model remains the dominant approach due to its better performance, the end-to-end method becomes more and more popular because it has lower latency by avoiding inferences with two models and rectifies the error propagation in theory 
Since it is hard to obtain a large-scale ST dataset, multi-task learning BIBREF5, BIBREF6 and pre-training techniques BIBREF7 have been applied to end-to-end ST model to leverage large-scale datasets of ASR and MT  A common practice is to pre-train two encoder-decoder models for ASR and MT respectively, and then initialize the ST model with the encoder of the ASR model and the decoder of the MT model  Subsequently, the ST model is optimized with the multi-task learning by weighing the losses of ASR, MT, and ST  This approach, however, causes a huge gap between pre-training and fine-tuning, which are summarized into three folds:
Subnet Waste: The ST system just reuses the ASR encoder and the MT decoder, while discards other pre-trained subnets, such as the MT encoder  Consequently, valuable semantic information captured by the MT encoder cannot be inherited by the final ST system 
Role Mismatch: The speech encoder plays different roles in pre-training and fine-tuning  The encoder is a pure acoustic model in pre-training, while it has to extract semantic and linguistic features additionally in fine-tuning, which significantly increases the learning difficulty 
Non-pre-trained Attention Module: Previous work BIBREF6 trains attention modules for ASR, MT and ST respectively, hence, the attention module of ST does not benefit from the pre-training 
To address these issues, we propose a Tandem Connectionist Encoding Network (TCEN), which is able to reuse all subnets in pre-training, keep the roles of subnets consistent, and pre-train the attention module  Concretely, the TCEN consists of three components, a speech encoder, a text encoder, and a target text decoder  Different from the previous work that pre-trains an encoder-decoder based ASR model, we only pre-train an ASR encoder by optimizing the Connectionist Temporal Classification (CTC) BIBREF8 objective function  In this way, the additional decoder of ASR is not required while keeping the ability to read acoustic features into the source language space by the speech encoder  Besides, the text encoder and decoder can be pre-trained on a large MT dataset  After that, we employ common used multi-task learning method to jointly learn ASR, MT and ST tasks 
Compared to prior works, the encoder of TCEN is a concatenation of an ASR encoder and an MT encoder and our model does not have an ASR decoder, so the subnet waste issue is solved  Furthermore, the two encoders work at tandem, disentangling acoustic feature extraction and linguistic feature extraction, ensuring the role consistency between pre-training and fine-tuning  Moreover, we reuse the pre-trained MT attention module in ST, so we can leverage the alignment information learned in pre-training 
Since the text encoder consumes word embeddings of plausible texts in MT task but uses speech encoder outputs in ST task, another question is how one guarantees the speech encoder outputs are consistent with the word embeddings  We further modify our model to achieve semantic consistency and length consistency  Specifically, (1) the projection matrix at the CTC classification layer for ASR is shared with the word embedding matrix, ensuring that they are mapped to the same latent space, and (2) the length of the speech encoder output is proportional to the length of the input frame, so it is much longer than a natural sentence  To bridge the length gap, source sentences in MT are lengthened by adding word repetitions and blank tokens to mimic the CTC output sequences 
We conduct comprehensive experiments on the IWSLT18 speech translation benchmark BIBREF1, demonstrating the effectiveness of each component  Our model is significantly better than previous methods by 3 6 and 2 2 BLEU scores for the subword-level decoding and character-level decoding strategies, respectively 
Our contributions are three-folds: 1) we shed light on why previous ST models cannot sufficiently utilize the knowledge learned from the pre-training process; 2) we propose a new ST model, which alleviates shortcomings in existing methods; and 3) we empirically evaluate the proposed model on a large-scale public dataset 
Background ::: Problem Formulation
End-to-end speech translation aims to translate a piece of audio into a target-language translation in one step  The raw speech signals are usually converted to sequences of acoustic features, e g  Mel filterbank features  Here, we define the speech feature sequence as $\mathbf {x} = (x_1, \cdots , x_{T_x})$ The transcription and translation sequences are denoted as $\mathbf {y^{s}} = (y_1^{s}, \cdots , y_{T_s}^{s})$, and $\mathbf {y^{t}} = (y_1^{t}, \cdots , y_{T_t}^{t})$ repectively  Each symbol in $\mathbf {y^{s}}$ or $\mathbf {y^{t}}$ is an integer index of the symbol in a vocabulary $V_{src}$ or $V_{trg}$ respectively (e g  $y^s_i=k, k\in [0, |V_{src}|-1]$)  In this work, we suppose that an ASR dataset, an MT dataset, and a ST dataset are available, denoted as $\mathcal {A} = \lbrace (\mathbf {x_i}, \mathbf {y^{s}_i})\rbrace _{i=0}^I$, $\mathcal {M} =\lbrace (\mathbf {y^{s}_j}, \mathbf {y^{t}_j})\rbrace _{j=0}^J$ and $ \mathcal {S} =\lbrace (\mathbf {x_l}, \mathbf {y^{t}_l})\rbrace _{l=0}^L$ respectively  Given a new piece of audio $\mathbf {x}$, our goal is to learn an end to end model to generate a translation sentence $\mathbf {y^{t}}$ without generating an intermediate result $\mathbf {y^{s}}$ 
Background ::: Multi-Task Learning and Pre-training for ST
To leverage large scale ASR and MT data, multi-task learning and pre-training techniques are widely employed to improve the ST system  As shown in Figure FIGREF4, there are three popular multi-task strategies for ST, including 1) one-to-many setting, in which a speech encoder is shared between ASR and ST tasks; 2) many-to-one setting in which a decoder is shared between MT and ST tasks; and 3) many-to-many setting where both the encoder and decoder are shared 
A many-to-many multi-task model contains two encoders as well as two decoders  It can be jointly trained on ASR, MT, and ST tasks  As the attention module is task-specific, three attentions are defined 
Usually, the size of $\mathcal {A}$ and $\mathcal {M}$ is much larger than $\mathcal {S}$  Therefore, the common training practice is to pre-train the model on ASR and MT tasks and then fine-tune it with a multi-task learning manner  However, as aforementioned, this method suffers from subnet waste, role mismatch and non-pre-trained attention issues, which severely limits the end-to-end ST performance 
Our method
In this section, we first introduce the architecture of TCEN, which consists of two encoders connected in tandem, and one decoder with an attention module  Then we give the pre-training and fine-tuning strategy for TCEN  Finally, we propose our solutions for semantic and length inconsistency problems, which are caused by multi-task learning 
Our method ::: TCEN Architecture
Figure FIGREF5 sketches the overall architecture of TCEN, including a speech encoder $enc_s$, a text encoder $enc_t$ and a decoder $dec$ with an attention module $att$  During training, the $enc_s$ acts like an acoustic model which reads the input $\mathbf {x}$ to word or subword representations $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into hidden representations $\mathbf {h^t}$  Finally, the $dec$ defines a distribution probability over target words  The advantage of our architecture is that two encoders disentangle acoustic feature extraction and linguistic feature extraction, making sure that valuable knowledge learned from ASR and MT tasks can be effectively leveraged for ST training  Besides, every module in pre-training can be utilized in fine-tuning, alleviating the subnet waste problem 
Follow BIBREF9 inaguma2018speech, we use CNN-BiLSTM architecture to build our model  Specifically, the input features $\mathbf {x}$ are organized as a sequence of feature vectors in length $T_x$  Then, $\mathbf {x}$ is passed into a stack of two convolutional layers followed by max-pooling:
where $\mathbf {v}^{(l-1)}$ is feature maps in last layer and $\mathbf {W}^{(l)}$ is the filter  The max-pooling layers downsample the sequence in length by a total factor of four  The down-sampled feature sequence is further fed into a stack of five bidirectional $d$-dimensional LSTM layers:
where $[;]$ denotes the vector concatenation  The final output representation from the speech encoder is denoted as $\mathbf {h^s}=(h^s_1, \cdots , h^s_{\frac{T_x}{4}})$, where $h_i^s \in \mathbb {R}^d$ 
The text encoder $enc_t$ consists of two bidirectional LSTM layers  In ST task, $enc_t$ accepts speech encoder output $\mathbf {h}^s$ as input  While in MT, $enc_t$ consumes the word embedding representation $\mathbf {e^s}$ derived from $\mathbf {y^s}$, where each element $e^s_i$ is computed by choosing the $y_i^s$-th vector from the source embedding matrix $W_{E^s}$  The goal of $enc_t$ is to extract high-level linguistic features like syntactic features or semantic features from lower level subword representations $\mathbf {h}^s$ or $\mathbf {e}^s$  Since $\mathbf {h}^s$ and $\mathbf {e}^s$ belong to different latent space and have different lengths, there remain semantic and length inconsistency problems  We will provide our solutions in Section SECREF21  The output sequence of $enc_t$ is denoted as $\mathbf {h}^t$ 
The decoder is defined as two unidirectional LSTM layers with an additive attention $att$  It predicts target sequence $\mathbf {y^{t}}$ by estimating conditional probability $P(\mathbf {y^{t}}|\mathbf {x})$:
Here, $z_k$ is the the hidden state of the deocder RNN at $k$ step and $c_k$ is a time-dependent context vector computed by the attention $att$ 
Our method ::: Training Procedure
Following previous work, we split the training procedure to pre-training and fine-tuning stages  In pre-training stage, the speech encoder $enc_s$ is trained towards CTC objective using dataset $\mathcal {A}$, while the text encoder $enc_t$ and the decoder $dec$ are trained on MT dataset $\mathcal {M}$  In fine-tuning stage, we jointly train the model on ASR, MT, and ST tasks 
Our method ::: Training Procedure ::: Pre-training
To sufficiently utilize the large dataset $\mathcal {A}$ and $\mathcal {M}$, the model is pre-trained on CTC-based ASR task and MT task in the pre-training stage 
For ASR task, in order to get rid of the requirement for decoder and enable the $enc_s$ to generate subword representation, we leverage connectionist temporal classification (CTC) BIBREF8 loss to train the speech encoder 
Given an input $\mathbf {x}$, $enc_s$ emits a sequence of hidden vectors $\mathbf {h^s}$, then a softmax classification layer predicts a CTC path $\mathbf {\pi }$, where $\pi _t \in V_{src} \cup $ {`-'} is the observing label at particular RNN step $t$, and `-' is the blank token representing no observed labels:
where $W_{ctc} \in \mathbb {R}^{d \times (|V_{src}|+1)}$ is the weight matrix in the classification layer and $T$ is the total length of encoder RNN 
A legal CTC path $\mathbf {\pi }$ is a variation of the source transcription $\mathbf {y}^s$ by allowing occurrences of blank tokens and repetitions, as shown in Table TABREF14  For each transcription $\mathbf {y}$, there exist many legal CTC paths in length $T$  The CTC objective trains the model to maximize the probability of observing the golden sequence $\mathbf {y}^s$, which is calculated by summing the probabilities of all possible legal paths:
where $\Phi _T(y)$ is the set of all legal CTC paths for sequence $\mathbf {y}$ with length $T$  The loss can be easily computed using forward-backward algorithm  More details about CTC are provided in supplementary material 
For MT task, we use the cross-entropy loss as the training objective  During training, $\mathbf {y^s}$ is converted to embedding vectors $\mathbf {e^s}$ through embedding layer $W_{E^s}$, then $enc_t$ consumes $\mathbf {e^s}$ and pass the output $\mathbf {h^t}$ to decoder  The objective function is defined as:
Our method ::: Training Procedure ::: Fine-tune
In fine-tune stage, we jointly update the model on ASR, MT, and ST tasks  The training for ASR and MT follows the same process as it was in pre-training stage 
For ST task, the $enc_s$ reads the input $\mathbf {x}$ and generates $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into $\mathbf {h^t}$  Finally, the $dec$ predicts the target sentence  The ST loss function is defined as:
Following the update strategy proposed by BIBREF11 luong2015multi, we allocate a different training ratio $\alpha _i$ for each task  When switching between tasks, we select randomly a new task $i$ with probability $\frac{\alpha _i}{\sum _{j}\alpha _{j}}$ 
Our method ::: Subnet-Consistency
Our model keeps role consistency between pre-training and fine-tuning by connecting two encoders for ST task  However, this leads to some new problems: 1) The text encoder consumes $\mathbf {e^s}$ during MT training, while it accepts $\mathbf {h^s}$ during ST training  However, $\mathbf {e^s}$ and $\mathbf {h^s}$ may not follow the same distribution, resulting in the semantic inconsistency  2) Besides, the length of $\mathbf {h^s}$ is not the same order of magnitude with the length of $\mathbf {e^s}$, resulting in the length inconsistency 
In response to the above two challenges, we propose two countermeasures: 1) We share weights between CTC classification layer and source-end word embedding layer during training of ASR and MT, encouraging $\mathbf {e^s}$ and $\mathbf {h^s}$ in the same space  2)We feed the text encoder source sentences in the format of CTC path, which are generated from a seq2seq model, making it more robust toward long inputs 
Our method ::: Subnet-Consistency ::: Semantic Consistency
As shown in Figure FIGREF5, during multi-task training, two different hidden features will be fed into the text encoder $enc_t$: the embedding representation $\mathbf {e}^s$ in MT task, and the $enc_s$ output $\mathbf {h^s}$ in ST task  Without any regularization, they may belong to different latent spaces  Due to the space gap, the $enc_t$ has to compromise between two tasks, limiting its performance on individual tasks 
To bridge the space gap, our idea is to pull $\mathbf {h^s}$ into the latent space where $\mathbf {e}^s$ belong  Specifically, we share the weight $W_{ctc}$ in CTC classification layer with the source embedding weights $W_{E^s}$, which means $W_{ctc} = W_{E^s}$  In this way, when predicting the CTC path $\mathbf {\pi }$, the probability of observing the particular label $w_i \in V_{src}\cup ${`-'} at time step $t$, $p(\pi _t=w_i|\mathbf {x})$, is computed by normalizing the product of hidden vector $h_t^s$ and the $i$-th vector in $W_{E^s}$:
The loss function closes the distance between $h^s_t$ and golden embedding vector, encouraging $\mathbf {h}^s$ have the same distribution with $\mathbf {e}^s$ 
Our method ::: Subnet-Consistency ::: Length Consistency
Another existing problem is length inconsistency  The length of the sequence $\mathbf {h^s}$ is proportional to the length of the input frame $\mathbf {x}$, which is much longer than the length of $\mathbf {e^s}$  To solve this problem, we train an RNN-based seq2seq model to transform normal source sentences to noisy sentences in CTC path format, and replace standard MT with denoising MT for multi-tasking 
Specifically, we first train a CTC ASR model based on dataset $\mathcal {A} = \lbrace (\mathbf {x}_i, \mathbf {y}^s_i)\rbrace _{i=0}^{I}$, and generate a CTC-path $\mathbf {\pi }_i$ for each audio $\mathbf {x}_i$ by greedy decoding  Then we define an operation $S(\cdot )$, which converts a CTC path $\mathbf {\pi }$ to a sequence of the unique tokens $\mathbf {u}$ and a sequence of repetition times for each token $\mathbf {l}$, denoted as $S(\mathbf {\pi }) = (\mathbf {u}, \mathbf {l})$  Notably, the operation is reversible, meaning that $S^{-1} (\mathbf {u}, \mathbf {l})=\mathbf {\pi }$  We use the example $\mathbf {\pi _1}$ in Table TABREF14 and show the corresponding $\mathbf {u}$ and $\mathbf {l}$ in Table TABREF24 
Then we build a dataset $\mathcal {P} = \lbrace (\mathbf {y^s}_i, \mathbf {u}_i, \mathbf {l}_i)\rbrace _{i=0}^{I}$ by decoding all the audio pieces in $\mathcal {A}$ and transform the resulting path by the operation $S(\cdot )$  After that, we train a seq2seq model, as shown in Figure FIGREF25, which takes $ \mathbf {y^s}_i$ as input and decodes $\mathbf {u}_i, \mathbf {l}_i$ as outputs  With the seq2seq model, a noisy MT dataset $\mathcal {M}^{\prime }=\lbrace (\mathbf {\pi }_l, \mathbf {y^t}_l)\rbrace _{l=0}^{L}$ is obtained by converting every source sentence $\mathbf {y^s}_i \in \mathcal {M}$ to $\mathbf {\pi _i}$, where $\mathbf {\pi }_i = S^{-1}(\mathbf {u}_i, \mathbf {l}_i)$  We did not use the standard seq2seq model which takes $\mathbf {y^s}$ as input and generates $\mathbf {\pi }$ directly, since there are too many blank tokens `-' in $\mathbf {\pi }$ and the model tends to generate a long sequence with only blank tokens  During MT training, we randomly sample text pairs from $\mathcal {M}^{\prime }$ and $\mathcal {M}$ according to a hyper-parameter $k$  After tuning on the validation set, about $30\%$ pairs are sampled from $\mathcal {M}^{\prime }$  In this way, the $enc_t$ is more robust toward the longer inputs given by the $enc_s$ 
Experiments
We conduct experiments on the IWSLT18 speech translation task BIBREF1  Since IWSLT participators use different data pre-processing methods, we reproduce several competitive baselines based on the ESPnet BIBREF12 for a fair comparison 
Experiments ::: Dataset ::: Speech translation data:
The organizer provides a speech translation corpus extracting from the TED talk (ST-TED), which consists of raw English wave files, English transcriptions, and aligned German translations  The corpus contains 272 hours of English speech with 171k segments  We split 2k segments from the corpus as dev set and tst2010, tst2013, tst2014, tst2015 are used as test sets 
Speech recognition data: Aside from ST-TED, TED-LIUM2 corpus BIBREF13 is provided as speech recognition data, which contains 207 hours of English speech and 93k transcript sentences 
Text translation data: We use transcription and translation pairs in the ST-TED corpus and WIT3 as in-domain MT data, which contains 130k and 200k sentence pairs respectively  WMT2018 is used as out-of-domain training data which consists of 41M sentence pairs 
Data preprocessing: For speech data, the utterances are segmented into multiple frames with a 25 ms window size and a 10 ms step size  Then we extract 80-channel log-Mel filter bank and 3-dimensional pitch features using Kaldi BIBREF14, resulting in 83-dimensional input features  We normalize them by the mean and the standard deviation on the whole training set  The utterances with more than 3000 frames are discarded  The transcripts in ST-TED are in true-case with punctuation while in TED-LIUM2, transcripts are in lower-case and unpunctuated  Thus, we lowercase all the sentences and remove the punctuation to keep consistent  To increase the amount of training data, we perform speed perturbation on the raw signals with speed factors 0 9 and 1 1  For the text translation data, sentences longer than 80 words or shorter than 10 words are removed  Besides, we discard pairs whose length ratio between source and target sentence is smaller than 0 5 or larger than 2 0  Word tokenization is performed using the Moses scripts and both English and German words are in lower-case 
We use two different sets of vocabulary for our experiments  For the subword experiments, both English and German vocabularies are generated using sentencepiece BIBREF15 with a fixed size of 5k tokens  BIBREF9 inaguma2018speech show that increasing the vocabulary size is not helpful for ST task  For the character experiments, both English and German sentences are represented in the character level 
For evaluation, we segment each audio with the LIUM SpkDiarization tool BIBREF16 and then perform MWER segmentation with RWTH toolkit BIBREF17  We use lowercase BLEU as evaluation metric 
Experiments ::: Baseline Models and Implementation
We compare our method with following baselines 
Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder  It is trained from scratch on the ST-TED corpus 
Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained  The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus  The MT model has a text encoder and decoder with the same architecture of which in TCEN  It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data 
Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting  In the first two settings, we train the model with $\alpha _{st}=0 75$ while $\alpha _{asr}=0 25$ or $\alpha _{mt}=0 25$  For many-to-many setting, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$   For MT task, we use only in-domain data 
Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models  Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation  Their model solves the subnet waste issue by concatenating an ST decoder to an ASR encoder-decoder model  Notably, their ST decoder can consume representations from the speech encoder as well as the ASR decoder  For a fair comparison, the speech encoder and the ASR decoder are initialized from the pre-trained ASR model  The Triangle model is fine-tuned under their multi-task manner 
All our baselines as well as TCEN are implemented based on ESPnet BIBREF12, the RNN size is set as $d=1024$ for all models  We use a dropout of 0 3 for embeddings and encoders, and train using Adadelta with initial learning rate of 1 0 for a maximum of 10 epochs 
For training of TCEN, we set $\alpha _{asr}=0 2$ and $\alpha _{mt}=0 8$ in the pre-training stage, since the MT dataset is much larger than ASR dataset  For fine-tune, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$, same as the `many-to-many' baseline 
For testing, we select the model with the best accuracy on speech translation task on dev set  At inference time, we use a beam size of 10, and the beam scores include length normalization with a weight of 0 2 
Experiments ::: Experimental Results
Table TABREF29 shows the results on four test sets as well as the average performance  Our method significantly outperforms the strong `many-to-many+pretrain' baseline by 3 6 and 2 2 BLEU scores respectively, indicating the proposed method is very effective that substantially improves the translation quality  Besides, both pre-training and multi-task learning can improve translation quality, and the pre-training settings (2nd-4th rows) are more effective compared to multi-task settings (5th-8th rows)  We observe a performance degradation in the `triangle+pretrain' baseline  Compared to our method, where the decoder receives higher-level syntactic and semantic linguistic knowledge extracted from text encoder, their ASR decoder can only provide lower word-level linguistic information  Besides, since their model lacks text encoder and the architecture of ST decoder is different from MT decoder, their model cannot utilize the large-scale MT data in all the training stages  Interestingly, we find that the char-level models outperform the subword-level models in all settings, especially in vanilla baseline  A similar phenomenon is observed by BIBREF6 berard2018end  A possible explanation is that learning the alignments between speech frames and subword units in another language is notoriously difficult  Our method can bring more gains in the subword setting since our model is good at learning the text-to-text alignment and the subword-level alignment is more helpful to the translation quality 
Experiments ::: Discussion ::: Ablation Study
To better understand the contribution of each component, we perform an ablation study on subword-level experiments  The results are shown in Table TABREF37  In `-MT noise' setting, we do not add noise to source sentences for MT  In `-weight sharing' setting, we use different parameters in CTC classification layer and source embedding layer  These two experiments prove that both weight sharing and using noisy MT input benefit to the final translation quality  Performance degrades more in `-weight sharing', indicating the semantic consistency contributes more to our model  In the `-pretrain' experiment, we remove the pre-training stage and directly update the model on three tasks, leading to a dramatic decrease on BLEU score, indicating the pre-training is an indispensable step for end-to-end ST 
Experiments ::: Discussion ::: Learning Curve
It is interesting to investigate why our method is superior to baselines  We find that TCEN achieves a higher final result owing to a better start-point in fine-tuning  Figure FIGREF39 provides learning curves of subword accuracy on validation set  The x-axis denotes the fine-tuning training steps  The vanilla model starts at a low accuracy, because its networks are not pre-trained on the ASR and MT data  The trends of our model and `many-to-many+pretrain' are similar, but our model outperforms it about five points in the whole fine-tuning process  It indicates that the gain comes from bridging the gap between pre-training and fine-tuning rather than a better fine-tuning process 
Experiments ::: Discussion ::: Compared with a Cascaded System
Table TABREF29 compares our model with end-to-end baselines  Here, we compare our model with cascaded systems  We build a cascaded system by combining the ASR model and MT model used in pre-training baseline  Word error rate (WER) of the ASR system and BLEU score of the MT system are reported in the supplementary material  In addition to a simple combination of the ASR and MT systems, we also re-segment the ASR outputs before feeding to the MT system, denoted as cascaded+re-seg  Specifically, we train a seq2seq model BIBREF19 on the MT dataset, where the source side is a no punctuation sentence and the target side is a natural sentence  After that, we use the seq2seq model to add sentence boundaries and punctuation on ASR outputs  Experimental results are shown in Table TABREF41  Our end-to-end model outperforms the simple cascaded model over 2 BLEU scores, and it achieves a comparable performance with the cascaded model combining with a sentence re-segment model 
Related Work
Early works conduct speech translation in a pipeline manner BIBREF2, BIBREF20, where the ASR output lattices are fed into an MT system to generate target sentences  HMM BIBREF21, DenseNet BIBREF22, TDNN BIBREF23 are commonly used ASR systems, while RNN with attention BIBREF19 and Transformer BIBREF10 are top choices for MT  To enhance the robustness of the NMT model towards ASR errors, BIBREF24 DBLP:conf/eacl/TsvetkovMD14 and BIBREF25 DBLP:conf/asru/ChenHHL17 propose to simulate the noise in training and inference 
To avoid error propagation and high latency issues, recent works propose translating the acoustic speech into text in target language without yielding the source transcription BIBREF4  Since ST data is scarce, pre-training BIBREF7, multi-task learning BIBREF4, BIBREF6, curriculum learning BIBREF26, attention-passing BIBREF27, and knowledge distillation BIBREF28, BIBREF29 strategies have been explored to utilize ASR data and MT data  Specifically, BIBREF5 DBLP:conf/interspeech/WeissCJWC17 show improvements of performance by training the ST model jointly with the ASR and the MT model  BIBREF6 berard2018end observe faster convergence and better results due to pre-training and multi-task learning on a larger dataset  BIBREF7 DBLP:conf/naacl/BansalKLLG19 show that pre-training a speech encoder on one language can improve ST quality on a different source language  All of them follow the traditional multi-task training strategies  BIBREF26 DBLP:journals/corr/abs-1802-06003 propose to use curriculum learning to improve ST performance on syntactically distant language pairs  To effectively leverage transcriptions in ST data, BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 augment the multi-task model where the target decoder receives information from the source decoder and they show improvements on low-resource speech translation  Their model just consumes ASR and ST data, in contrast, our work sufficiently utilizes the large-scale MT data to capture the rich semantic knowledge  BIBREF30 DBLP:conf/icassp/JiaJMWCCALW19 use pre-trained MT and text-to-speech (TTS) synthesis models to convert weakly supervised data into ST pairs and demonstrate that an end-to-end MT model can be trained using only synthesised data 
Conclusion
This paper has investigated the end-to-end method for ST  It has discussed why there is a huge gap between pre-training and fine-tuning in previous methods  To alleviate these issues, we have proposed a method, which is capable of reusing every sub-net and keeping the role of sub-net consistent between pre-training and fine-tuning  Empirical studies have demonstrated that our model significantly outperforms baselines 
Context 2: Introduction
Medical search engines are an essential component for many online medical applications, such as online diagnosis systems and medical document databases  A typical online diagnosis system, for instance, relies on a medical search engine  The search engine takes as input a user query that describes some symptoms and then outputs clinical concept entries that provide relevant information to assist in diagnosing the problem  One challenge medical search engines face is the segmentation of individual clinical entities  When a user query consists of multiple clinical entities, a search engine would often fail to recognize them as separate entities  For example, the user query “fever joint pain weight loss headache” contains four separate clinical entities: “fever”, “joint pain”, “weight loss”, and “headache”  But when the search engine does not recognize them as separate entities and proceeds to retrieve results for each word in the query, it may find "pain" in body locations other than "joint pain", or it may miss "headache" altogether, for example  Some search engines allow the users to enter a single clinical concept by selecting from an auto-completion pick list  But this could also result in retrieving inaccurate or partial results and lead to poor user experience 
We want to improve the medical search engine so that it can accurately retrieve all the relevant clinical concepts mentioned in a user query, where relevant clinical concepts are defined with respect to the terminologies the search engine uses  The problem of extracting clinical concept mentions from a user query can be seen as a variant of the Concept Extraction (CE) task in the frequently-cited NLP challenges in healthcare, such as 2010 i2b2/VA BIBREF0 and 2013 ShARe/CLEF Task 1 BIBREF1  Both CE tasks in 2010 i2b2/VA and 2013 ShARe/CLEF Task 1 ask the participants to design an algorithm to tag a set of predefined entities of interest in clinical notes  These entity tagging tasks are also known as clinical Named Entity Recognition (NER)  For example, the CE task in 2010 i2b2/VA defines three types of entities: “problem”, “treatment”, and “test”  The CE task in 2013 ShARe/CLEF defines various types of disorder such as “injury or poisoning”, "disease or syndrome”, etc  In addition to tagging, the CE task in 2013 ShARe/CLEF has an encoding component which requires selecting one and only one Concept Unique Identifier (CUI) from Systematized Nomenclature Of Medicine Clinical Terms (SNOMED-CT) for each disorder entity tagged  Our problem, similar to the CE task in 2013 ShARe/CLEF, also contains two sub-problems: tagging mentions of entities of interest (entity tagging), and selecting appropriate terms from a glossary to match the mentions (term matching)  However, several major differences exist  First, compared to clinical notes, the user queries are much shorter, less technical, and often less coherent  Second, instead of encoding, we are dealing with term matching where we rank a few best terms that match an entity, instead of selecting only one  This is because the users who type the queries may not have a clear idea about what they are looking for, or could be laymen who know little terminology, it may be more helpful to provide a set of likely results and let the users choose  Third, the types of entities are different  Each medical search engine may have its own types of entities to tag  There is also one minor difference in the tagging scheme between our problem and the CE task in 2013 ShARe/CLEF - We limit our scope to dealing with entities of consecutive words and not disjoint entities   We use only Beginning, Inside, Outside (BIO) tags  Given the differences listed above, we need to customize a framework consisting of an entity tagging and term matching component for our CE problem 
Related Work
An effective model that has been commonly used for NER problem is a Bi-directional LSTM with a Conditional Random Field (CRF) on the top layer (BiLSTM-CRF), which is described in the next section  Combining LSTM’s power of representing relations between words and CRF’s capability of accounting for tag sequence constraints, Huang et al  BIBREF2 proposed the BiLSTM-CRF model and used handcrafted word features as the input to the model  Lample et al  BIBREF3 used a combination of character-level and word-level word embeddings as the input to BiLSTM-CRF  Since then, similar models with variation in types of word embeddings have been used extensively for clinical CE tasks and produced state-of-the-art results BIBREF4, BIBREF5, BIBREF6, BIBREF7  Word embeddings have become the cornerstone of the neural models in NLP since the famous Word2vec BIBREF8 model demonstrated its power in word analogy tasks  One well-known example is that after training Word2vec on a large amount of news data, we can get word relations such as $vector(^{\prime }king^{\prime }) - vector(^{\prime }queen^{\prime }) + vector(^{\prime }woman^{\prime }) \approx vector(^{\prime }man^{\prime })$  More sophisticated word embedding technique emerged since Word2vec  It has been shown empirically that better quality in word embeddings leads to better performance in many downstream NLP including entity tagging BIBREF9, BIBREF10  Recently, contextualized word embeddings generated by deep learning models, such as ELMo BIBREF11, BERT BIBREF12, and Flair BIBREF13, have been shown to be more effective in various NLP tasks  In our project, we make use of a fine-tuned ELMo model and a fine-tuned Flair model in the medical domain  We experiment with the word embeddings from the two fine-tuned models as the input to the BiLSTM-CRF model separately and compare the results 
Tang et al  BIBREF14 provided straightforward algorithm for term matching  The algorithm starts with finding candidate terms that contain ALL the entity words, with term frequency - inverse document frequency (tf-idf) weighting  Then the candidates are ranked based on the pairwise cosine distance between the word embeddings of the candidates and the entity 
Framework
We adopt the tagging - encoding pipeline framework from the CE task in 2013 ShARe/CLEF  We first tag the clinical entities in the user query and then select relevant terms from a glossary in dermatology to match the entities 
Framework ::: Entity Tagging
We use the same BiLSTM-CRF model proposed by Huang et al  BIBREF2  An illustration of the architecture is shown in Figure FIGREF6   Given a sequence (or sentence) of n tokens, $r = (w_1, w_2,   , w_n)$, we use a fine-tuned ELMo model to generate contextual word embeddings for all the tokens in the sentence, where a token refers to a word or punctuation  We denote the ELMo embedding, $x$, for a token $w$ in the sentence $r$ by $x = ELMo(w|r)$  The notation and the procedure described here can be adopted for Flair embeddings or other embeddings  Now, given a sequence of tokens in ELMo embeddings, $X =(x_1, x_2,    , x_n)$, the BiLSTM layer generates a matrix of scores, $P(\theta )$ of size $n \times k$, where $k$ is the number of tag types, and $\theta $ is the parameters of the BiLSTM  To simplify notation, we will omit the $\theta $ and write $P$  Then, $P_{i,j}$ denotes the score of the token, $x_i$, being assigned to the $j$th tag  Since certain constraints may exist in the transition between tags, an "O" tag should not be followed by an "I" tag, for example, a transition matrix, $A$, of dimension $(k+2)\times (k+2)$, is initialized to model the constraints  The learnable parameters, $A_{i,j}$, represent the probability of the $j$th tag follows the $i$th tag in a sequence  For example, if we index the tags by: 1:“B”, 2:“I”, and 3:“O”, then $A_{1,3}$ would be the probability that an “O” tag follows a “B” tag  A beginning transition and an end transition are inserted in $A$ and hence $A$ is of dimension $(k+2)\times (k+2)$ 
Given a sequence of tags, $Y=(y_1,y_2,   ,y_n)$, where each $y_i$, $1\le i \le n$, corresponds to an index of the tags, the score of the sequence is then given by
The probability of the sequence of tags is then calculated by a softmax,
where $\lbrace Y_x\rbrace $ denotes the set of all possible tag sequences  During training, the objective function is to maximize $\log (P(Y|X))$ by adjusting $A$ and $P$ 
Framework ::: Term Matching
The term matching algorithm of Tang et al  BIBREF14 is adopted with some major modifications  First, to identify candidate terms, we use a much looser string search algorithm where we stem the entity words with snowball stemmer and then find candidate terms which contain ANY non-stopword words in the entity  The stemming is mainly used to represent a word in its stem  For example, “legs” becomes “leg”, and “jammed” becomes “jam”  Thus, stemming can provide more tolerance when finding candidates  Similarly, finding candidates using the condition ANY (instead of ALL) also increases the tolerance  However, if the tagged entity contains stopwords such as “in”, “on”, etc , the pool of candidates will naturally grow very large and increase the computation cost for later part, because even completely irrelevant terms may contain stopwords  Therefore, we match based on non-stopwords only  To illustrate the points above, suppose a query is tagged with the entity
Ex  3 1 “severe burns on legs”,
and one relevant term is “leg burn”  After stemming, “burns” and “legs” in Ex UNKREF12 become “burn” and “leg”, respectively, allowing "leg burn" to be considered as a candidate  Although the word “severe” is not in the term “leg burn”, the term is still considered a candidate because we selected using ANY  The stopword “on” is ignored when finding candidate terms so that not every term that contains the word “on” is added to the candidate pool  When a candidate term, $C$, is found in this manner for the tagged entity, $E$, we calculate the semantic similarity score, $s$, between $C$ and $E$ in two steps  In the first step, calculate the maximum similarity score for each word in $C$ as shown in Figure FIGREF10  Given a word in the candidate term, $C_i$ ($1 \le i \le m$, $m$ is the number of words in the candidate term) and a word in the tagged entity, $E_j$ Their similarity score, $s_{ij}$ (shown as the element in the boxed matrix in Figure FIGREF10), is given by
where $ELMo(C_i|C)$ and $ELMo(E_j|E)$ are the ELMo embeddings for the word $C_i$ and $E_j$, respectively  The ELMo embeddings have the same dimension for all words when using the same fine-tuned ELMo model  Thus, we can use a distance function (e g
Context 3: Introduction
Cyberbullying has been defined by the National Crime Prevention Council as the use of the Internet, cell phones or other devices to send or post text or images intended to hurt or embarrass another person  Various studies have estimated that between to 10% to 40% of internet users are victims of cyberbullying BIBREF0   Effects of cyberbullying can range from temporary anxiety to suicide BIBREF1   Many high profile incidents have emphasized the prevalence of cyberbullying on social media  Most recently in October 2017, a Swedish model Arvida Byström was cyberbullied to the extent of receiving rape threats after she appeared in an advertisement with hairy legs 
Detection of cyberbullying in social media is a challenging task  Definition of what constitutes cyberbullying is quite subjective  For example, frequent use of swear words might be considered as bullying by the general population  However, for teen oriented social media platforms such as Formspring, this does not necessarily mean bullying (Table TABREF9 )  Across multiple SMPs, cyberbullies attack victims on different topics such as race, religion, and gender  Depending on the topic of cyberbullying, vocabulary and perceived meaning of words vary significantly across SMPs  For example, in our experiments we found that for word `fat', the most similar words as per Twitter dataset are `female' and `woman' (Table TABREF23 )  However, other two datasets do not show such particular bias against women  This platform specific semantic similarity between words is a key aspect of cyberbullying detection across SMPs  Style of communication varies significantly across SMPs  For example, Twitter posts are short and lack anonymity  Whereas posts on Q&A oriented SMPs are long and have option of anonymity (Table TABREF7 )  Fast evolving words and hashtags in social media make it difficult to detect cyberbullying using swear word list based simple filtering approaches  The option of anonymity in certain social networks also makes it harder to identify cyberbullying as profile and history of the bully might not be available 
Past works on cyberbullying detection have at least one of the following three bottlenecks  First (Bottleneck B1), they target only one particular social media platform  How these methods perform across other SMPs is unknown  Second (Bottleneck B2), they address only one topic of cyberbullying such as racism, and sexism  Depending on the topic, vocabulary and nature of cyberbullying changes  These models are not flexible in accommodating changes in the definition of cyberbullying  Third (Bottleneck B3), they rely on carefully handcrafted features such as swear word list and POS tagging  However, these handcrafted features are not robust against variations in writing style  In contrast to existing bottlenecks, this work targets three different types of social networks (Formspring: a Q&A forum, Twitter: microblogging, and Wikipedia: collaborative knowledge repository) for three topics of cyberbullying (personal attack, racism, and sexism) without doing any explicit feature engineering by developing deep learning based models along with transfer learning 
We experimented with diverse traditional machine learning models (logistic regression, support vector machine, random forest, naive Bayes) and deep neural network models (CNN, LSTM, BLSTM, BLSTM with Attention) using variety of representation methods for words (bag of character n-gram, bag of word unigram, GloVe embeddings, SSWE embeddings)  Summary of our findings and research contributions is as follows 
Datasets
Please refer to Table TABREF7 for summary of datasets used  We performed experiments using large, diverse, manually annotated, and publicly available datasets for cyberbullying detection in social media  We cover three different types of social networks: teen oriented Q&A forum (Formspring), large microblogging platform (Twitter), and collaborative knowledge repository (Wikipedia talk pages)  Each dataset addresses a different topic of cyberbullying  Twitter dataset contains examples of racism and sexism  Wikipedia dataset contains examples of personal attack  However, Formspring dataset is not specifically about any single topic  All three datasets have the problem of class imbalance where posts labeled as cyberbullying are in the minority as compared to neutral posts  Variation in the number of posts across datasets also affects vocabulary size that represents the number of distinct words encountered in the dataset  We measure the size of a post in terms of the number of words in the post  For each dataset, there are only a few posts with large size  We truncate such large posts to the size of post ranked at 95 percentile in that dataset  For example, in Wikipedia dataset, the largest post has 2846 words  However, size of post ranked at 95 percentile in that dataset is only 231  Any post larger than size 231 in Wikipedia dataset will be truncated by considering only first 231 words  This truncation affects only a small minority of posts in each dataset  However, it is required for efficiently training various models in our experiments  Details of each dataset are as follows 
Formspring BIBREF2 : It was a question and answer based website where users could openly invite others to ask and answer questions  The dataset includes 12K annotated question and answer pairs  Each post is manually labeled by three workers  Among these pairs, 825 were labeled as containing cyberbullying content by at least two Amazon Mechanical turk workers 
Twitter BIBREF3 : This dataset includes 16K annotated tweets  The authors bootstrapped the corpus collection, by performing an initial manual search of common slurs and terms used pertaining to religious, sexual, gender, and ethnic minorities  Of the 16K tweets, 3117 are labeled as sexist, 1937 as racist, and the remaining are marked as neither sexist nor racist 
Wikipedia BIBREF4 : For each page in Wikipedia, a corresponding talk page maintains the history of discussion among users who participated in its editing  This data set includes over 100k labeled discussion comments from English Wikipedia's talk pages  Each comment was labeled by 10 annotators via Crowdflower on whether it contains a personal attack  There are total 13590 comments labeled as personal attack 
Use of Swear Words and Anonymity
Please refer to Table TABREF9   We use the following short forms in this section: B=Bullying, S=Swearing, A=Anonymous  Some of the values for Twitter dataset are undefined as Twitter does not allow anonymous postings  Use of swear words has been repeatedly linked to cyberbullying  However, preliminary analysis of datasets reveals that depending on swear word usage can neither lead to high precision nor high recall for cyberbullying detection  Swear word list based methods will have low precision as P(B INLINEFORM0 S) is not close to 1  In fact, for teen oriented social network Formspring, 78% of the swearing posts are non-bullying  Swear words based filtering will be irritating to the users in such SMPs where swear words are used casually  Swear word list based methods will also have a low recall as P(S INLINEFORM1 B) is not close to 1  For Twitter dataset, 82% of bullying posts do not use any swear words  Such passive-aggressive cyberbullying will go undetected with swear word list based methods  Anonymity is another clue that is used for detecting cyberbullying as bully might prefer to hide its identity  Anonymity definitely leads to increased use of swear words (P(S INLINEFORM2 A) INLINEFORM3 P(S)) and cyberbullying (P(B INLINEFORM4 A) INLINEFORM5 P(B), and P(B INLINEFORM6 A&S)) INLINEFORM7 P(B))  However, significant fraction of anonymous posts are non-bullying (P(B INLINEFORM8 A) not close to 1) and many of bullying posts are not anonymous (P(A INLINEFORM9 B) not close to 1)  Further, anonymity might not be allowed by many SMPs such as Twitter 
Related Work
Cyberbullying is recognized as a phenomenon at least since 2003 BIBREF5   Use of social media exploded with launching of multiple platforms such as Wikipedia (2001), MySpace (2003), Orkut (2004), Facebook (2004), and Twitter (2005)  By 2006, researchers had pointed that cyberbullying was as serious phenomenon as offline bullying BIBREF6   However, automatic detection of cyberbullying was addressed only since 2009 BIBREF7   As a research topic, cyberbullying detection is a text classification problem  Most of the existing works fit in the following template: get training dataset from single SMP, engineer variety of features with certain style of cyberbullying as the target, apply a few traditional machine learning methods, and evaluate success in terms of measures such as F1 score and accuracy  These works heavily rely on handcrafted features such as use of swear words  These methods tend to have low precision for cyberbullying detection as handcrafted features are not robust against variations in bullying style across SMPs and bullying topics  Only recently, deep learning has been applied for cyberbullying detection BIBREF8   Table TABREF27 summarizes important related work 
Deep Neural Network (DNN) Based Models
We experimented with four DNN based models for cyberbullying detection: CNN, LSTM, BLSTM, and BLSTM with attention  These models are listed in the increasing complexity of their neural architecture and amount of information used by these models  Please refer to Figure 1 for general architecture that we have used across four models  Various models differ only in the Neural Architecture layer while having identical rest of the layers  CNNs are providing state-of-the-results on extracting contextual feature for classification tasks in images, videos, audios, and text  Recently, CNNs were used for sentiment classification BIBREF9   Long Short Term Memory networks are a special kind of RNN, capable of learning long-term dependencies  Their ability to use their internal memory to process arbitrary sequences of inputs has been found to be effective for text classification BIBREF10   Bidirectional LSTMs BIBREF11 further increase the amount of input information available to the network by encoding information in both forward and backward direction  By using two directions, input information from both the past and future of the current time frame can be used  Attention mechanisms allow for a more direct dependence between the state of the model at different points in time  Importantly, attention mechanism lets the model learn what to attend to based on the input sentence and what it has produced so far 
The embedding layer processes a fixed size sequence of words  Each word is represented as a real-valued vector, also known as word embeddings  We have experimented with three methods for initializing word embeddings: random, GloVe BIBREF12 , and SSWE BIBREF13   During the training, model improves upon the initial word embeddings to learn task specific word embeddings  We have observed that these task specific word embeddings capture the SMP specific and topic specific style of cyberbullying  Using GloVe vectors over random vector initialization has been reported to improve performance for some NLP tasks  Most of the word embedding methods such as GloVe, consider only syntactic context of the word while ignoring the sentiment conveyed by the text  SSWE method overcomes this problem by incorporating the text sentiment as one of the parameters for word embedding generation  We experimented with various dimension size for word embeddings  Experimental results reported here are with dimension size as 50  There was no significant variation in results with dimension size ranging from 30 to 200 
To avoid overfitting, we used two dropout layers, one before the neural architecture layer and one after, with dropout rates of 0 25 and 0 5 respectively  Fully connected layer is a dense output layer with the number of neurons equal to the number of classes, followed by softmax layer that provides softmax activation  All our models are trained using backpropagation  The optimizer used for training is Adam and the loss function is categorical cross-entropy  Besides learning the network weights, these methods also learn task-specific word embeddings tuned towards the bullying labels (See Section SECREF21 )  Our code is available at: https://github com/sweta20/Detecting-Cyberbullying-Across-SMPs 
Experiments
Existing works have heavily relied on traditional machine learning models for cyberbullying detection  However, they do not study the performance of these models across multiple SMPs  We experimented with four models: logistic regression (LR), support vector machine (SVM), random forest (RF), and naive Bayes (NB), as these are used in previous works (Table TABREF27 )  We used two data representation methods: character n-gram and word unigram  Past work in the domain of detecting abusive language have showed that simple n-gram features are more powerful than linguistic and syntactic features, hand-engineered lexicons, and word and paragraph embeddings BIBREF14   As compared to DNN models, performance of all four traditional machine learning models was significantly lower  Please refer to Table TABREF11  
All DNN models reported here were implemented using Keras  We pre-process the data, subjecting it to standard operations of removal of stop words, punctuation marks and lowercasing, before annotating it to assigning respective labels to each comment  For each trained model, we report its performance after doing five-fold cross-validation  We use following short forms 
Effect of Oversampling Bullying Instances
The training datasets had a major problem of class imbalance with posts marked as bullying in the minority  As a result, all models were biased towards labeling the posts as non-bullying  To remove this bias, we oversampled the data from bullying class thrice  That is, we replicated bullying posts thrice in the training data  This significantly improved the performance of all DNN models with major leap in all three evaluation measures  Table TABREF17 shows the effect of oversampling for a variety of word embedding methods with BLSTM Attention as the detection model  Results for other models are similar BIBREF15   We can notice that oversampled datasets (F+, T+, W+) have far better performance than their counterparts (F, T, W respectively)  Oversampling particularly helps the smallest dataset Formspring where number of training instances for bullying class is quite small (825) as compared to other two datasets (about 5K and 13K)  We also experimented with varying the replication rate for bullying posts BIBREF15   However, we observed that for bullying posts, replication rate of three is good enough 
Choice of Initial Word Embeddings and Model
Initial word embeddings decide data representation for DNN models  However during the training, DNN models modify these initial word embeddings to learn task specific word embeddings  We have experimented with three methods to initialize word embeddings  Please refer to Table TABREF19   This table shows the effect of varying initial word embeddings for multiple DNN models across datasets  We can notice that initial word embeddings do not have a significant effect on cyberbullying detection when oversampling of bullying posts is done (rows corresponding to F+, T+, W+)  In the absence of oversampling (rows corresponding to F, T W), there is a gap in performance of simplest (CNN) and most complex (BLSTM with attention) models  However, this gap goes on reducing with the increase in the size of datasets 
Table TABREF20 compares the performance of four DNN models for three evaluation measures while using SSWE as the initial word embeddings  We have noticed that most of the time LSTM performs weaker than other three models  However, performance gap in the other three models is not significant 
Task Specific Word Embeddings
DNN models learn word embeddings over the training data  These learned embeddings across multiple datasets show the difference in nature and style of bullying across cyberbullying topics and SMPs  Here we report results for BLSTM with attention model  Results for other models are similar  We first verify that important words for each topic of cyberbullying form clusters in the learned embeddings  To enable the visualization of grouping, we reduced dimensionality with t-SNE BIBREF16 , a well-known technique for dimensionality reduction particularly well suited for visualization of high dimensional datasets  Please refer to Table TABREF22   This table shows important clusters observed in t-SNE projection of learned word embeddings  Each cluster shows that words most relevant to a particular topic of bullying form cluster 
We also observed changes in the meanings of the words across topics of cyberbullying  Table TABREF23 shows most similar words for a given query word for two datasets  Twitter dataset which is heavy on sexism and racism, considers word slave as similar to targets of racism and sexism  However, Wikipedia dataset that is about personal attacks does not show such bias 
Transfer Learning
We used transfer learning to check if the knowledge gained by DNN models on one dataset can be used to improve cyberbullying detection performance on other datasets  We report results where BLSTM with attention is used as the DNN model  Results for other models are similar BIBREF15   We experimented with following three flavors of transfer learning 
Complete Transfer Learning (TL1): In this flavor, a model trained on one dataset was directly used to detect cyberbullying in other datasets without any extra training  TL1 resulted in significantly low recall indicating that three datasets have different nature of cyberbullying with low overlap (Table TABREF25 )  However precision was relatively higher for TL1, indicating that DNN models are cautious in labeling a post as bully (Table TABREF25 )  TL1 also helps to measure similarity in nature of cyberbullying across three datasets  We can observe that bullying nature in Formspring and Wikipedia datasets is more similar to each other than the Twitter dataset  This can be inferred from the fact that with TL1, cyberbullying detection performance for Formspring dataset is higher when base model is Wikipedia (precision =0 51 and recall=0 66)as compared to Twitter as the base model (precision=0 38 and recall=0 04)  Similarly, for Wikipedia dataset, Formspring acts as a better base model than Twitter while using TL1 flavor of transfer learning  Nature of SMP might be a factor behind this similarity in nature of cyberbullying  Both Formspring and Wikipedia are task oriented social networks (Q&A and collaborative knowledge repository respectively) that allow anonymity and larger posts  Whereas communication on Twitter is short, free of anonymity and not oriented towards a particular task 
Feature Level Transfer Learning (TL2): In this flavor, a model was trained on one dataset and only learned word embeddings were transferred to another dataset for training a new model  As compared to TL1, recall score improved dramatically with TL2 (Table TABREF25 )  Improvement in precision was also significant (Table TABREF25 )  These improvements indicate that learned word embeddings are an essential part of knowledge transfer across datasets for cyberbullying detection 
Model Level Transfer Learning (TL3): In this flavor, a model was trained on one dataset and learned word embeddings, as well as network weights, were transferred to another dataset for training a new model  TL3 does not result in any significant improvement over TL2  This lack of improvement indicates that transfer of network weights is not essential for cyberbullying detection and learned word embeddings is the key knowledge gained by the DNN models 
DNN based models coupled with transfer learning beat the best-known results for all three datasets  Previous best F1 scores for Wikipedia BIBREF4 and Twitter BIBREF8 datasets were 0 68 and 0 93 respectively  We achieve F1 scores of 0 94 for both these datasets using BLSTM with attention and feature level transfer learning (Table TABREF25 )  For Formspring dataset, authors have not reported F1 score  Their method has accuracy score of 78 5% BIBREF2   We achieve F1 score of 0 95 with accuracy score of 98% for the same dataset 
Conclusion and Future Work
We have shown that DNN models can be used for cyberbullying detection on various topics across multiple SMPs using three datasets and four DNN models  These models coupled with transfer learning beat state of the art results for all three datasets  These models can be further improved with extra data such as information about the profile and social graph of users  Most of the current datasets do not provide any information about the severity of bullying  If such fine-grained information is made available, then cyberbullying detection models can be further improved to take a variety of actions depending on the perceived seriousness of the posts 

ground_truth: Vanilla ST baseline, encoder pre-training, in which the ST encoder is initialized from an ASR model, decoder pre-training, in which the ST decoder is initialized from an MT model, encoder-decoder pre-training, where both the encoder and decoder are pre-trained, many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models, Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation

Vanilla ST baseline, Pre-training baselines, Multi-task baselines, Many-to-many+pre-training, Triangle+pre-train

Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder. It is trained from scratch on the ST-TED corpus.

Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained. The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus. The MT model has a text encoder and decoder with the same architecture of which in TCEN. It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data.

Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting. In the first two settings, we train the model with $\alpha _{st}=0.75$ while $\alpha _{asr}=0.25$ or $\alpha _{mt}=0.25$. For many-to-many setting, we use $\alpha _{st}=0.6, \alpha _{asr}=0.2$ and $\alpha _{mt}=0.2$.. For MT task, we use only in-domain data.

Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models. 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. where did they obtain the annotated clinical notes from?
Context 2: , the cosine distance), denoted $d(\cdot )$ in equation DISPLAY_FORM13, to compute the semantic similarity between words  In step 2, we calculate the candidate-entity relevance score (similarity) using the formula
where $s_c$ is a score threshold, and $\mathbb {I} \lbrace max(\vec{S_i}) > s_c\rbrace $ is an indicator function that equals 1 if $max(\vec{S_i}) > s_c$ or equals 0 if not  In equation DISPLAY_FORM14 we define a metric that measures “information coverage” of the candidate terms with respect to a tagged entity  If the constituent words of a candidate term are relevant to the constituent words in the tagged entity, then the candidate term offers more information coverage  Intuitively, the more relevant words present in the candidate term, the more relevant the candidate is to the tagged entity  The purpose of the cutoff, $s_c$, is to screen the $(C_i,E_j)$ word pairs that are dissimilar, so that they do not contribute to information coverage  One can adjust the strictness of the entity - terminology matching by adjusting $s_c$  The higher we set $s_c$, the fewer candidate terms will be selected for a tagged entity  A normalization factor, $\frac{1}{m}$, is added to give preference to more concise candidate terms given the same amount of information coverage 
We need to create an extra stopword list to include words such as “configuration” and “color”, and exclude these words from the word count for a candidate term  This is because the terms associated with the description of color or configuration usually have the word “color” or “configuration” in them  On the other hand, a user query normally does not contain such words  For example, a tagged entity in a user query could be “round yellow patches”, for which the relevant terminologies include “round configuration” and “yellow color”  Since we applied a normalization factor, $\frac{1}{m}$, to the relevance score, the word “color” and “configuration” would lower the relevance score because they do not have a counterpart in the tagged entity  Therefore, we need to exclude them from word count  Once the process is complete, calculate $s(C,E)$ for all candidate terms and then we can apply a threshold on all $s(C,E)$ to ignore candidate terms with low information coverage  Finally, rank the terms by their $s(C,E)$ and return the ranked list as the results 
Experiments ::: Data
Despite the greater similarity between our task and the 2013 ShARe/CLEF Task 1, we use the clinical notes from the CE task in 2010 i2b2/VA on account of 1) the data from 2010 i2b2/VA being easier to access and parse, 2) 2013 ShARe/CLEF containing disjoint entities and hence requiring more complicated tagging schemes  The synthesized user queries are generated using the aforementioned dermatology glossary  Tagged sentences are extracted from the clinical notes  Sentences with no clinical entity present are ignored  22,489 tagged sentences are extracted from the clinical notes  We will refer to these tagged sentences interchangeably as the i2b2 data  The sentences are shuffled and split into train/dev/test set with a ratio of 7:2:1  The synthesized user queries are composed by randomly selecting several clinical terms from the dermatology glossary and then combining them in no particular order  When combining the clinical terms, we attach the BIO tags to their constituent words  The synthesized user queries (13,697 in total) are then split into train/dev/test set with the same ratio  Next, each set in the i2b2 data and the corresponding set in the synthesized query data are combined to form a hybrid train/dev/test set, respectively  This way we ensure that in each hybrid train/dev/test set, the ratio between the i2b2 data and the synthesized query data is the same 
The reason for combining the two data is their drastic structural difference (See figure FIGREF16 for an example)  Previously, when trained on the i2b2 data only, the BiLSTM-CRF model was not able to segment clinical entities at the correct boundary  It would fail to recognize the user query in Figure FIGREF16(a) as four separate entities  On the other hand, if the model was trained solely on the synthesized user queries, we could imagine that it would fail miserably on any queries that resemble the sentence in Figure FIGREF16(b) because the model would have never seen an “O” tag in the training data  Therefore, it is necessary to use the hybrid training data containing both the i2b2 data and the synthesized user queries 
To make the hybrid training data, we need to unify the tags  Recall that in Section SECREF1 we point out that the tags are different for the different tasks and datasets  Since we use custom tags for dermatology glossary in our problem, we would need to convert the tags used in 2010 i2b2/VA  But this would be an infeasible job as we need experts to manually do that  An alternative is to avoid distinguishing the tag types and label all tags under the generic BIO tags 
Experiments ::: Setup
To show the effects of using the hybrid training data, we trained two models of the same architecture and hyperparameters  One model was trained on the hybrid data and will be referred to as hybrid NER model  The other model was trained on clinical notes only and will be referred to as i2b2 NER model  We evaluated the performance of the NER models by micro-F1 score on the test set of both the synthesized queries and the i2b2 data 
We used the BiLSTM-CRF implementation provided by the flair package BIBREF16  We set the hidden size value to be 256 in the LSTM structure and left everything else at default values for the SequenceTagger model on flair  For word embeddings, we used the ELMo embeddings fine-tuned on PubMed articles and flair embeddings BIBREF13 trained on $5\%$ of PubMed abstracts , respectively  We trained models for 10 epochs and experimented with different learning rate, mini batch size, and dropouts  We ran hyperparameter optimization tests to find the best combination  $S_c$ is set to be 0 6 in our experiment 
Experiments ::: Hyperparameter Tuning
We defined the following hyperparameter search space:
embeddings: [“ELMo on pubmed”, “stacked flair on pubmed”],
hidden_size: [128, 256],
learning_rate: [0 05, 0 1],
mini_batch_size: [32, 64, 128] 
The hyperparameter optimization was performed using Hyperopt   Three evaluations were run for each combination of hyperparameters  Each ran for 10 epochs  Then the results were averaged to give the performance for that particular combination of hyperparameters 
Experiments ::: Results
From the hyperparameter tuning we found that the best combination was
embeddings: “ELMo on pubmed”,
hidden_size: 256,
learning_rate: 0 05,
mini_batch_size: 32 
With the above hyperparameter setting, the hybrid NER model achieved a F1 score of $0 995$ on synthesized queries and $0 948$ on clinical notes while the i2b2 NER model achieved a F1 score of $0 441$ on synthesized queries and $0 927$ on clinical notes (See Table TABREF23) 
Since there was no ground truth available for the retrieved terms, we randomly picked a few samples to assess its performance  Some example outputs of our complete framework on real user queries are shown in Figure FIGREF24  For example, from the figure we see that the query "child fever double vision dizzy" was correctly tagged with four entities: "child", "fever", "double vision", and "dizzy"  A list of terms from our glossary was matched to each entity  In real world application, the lists of terms will be presented to the user as the retrieval results to their queries 
Discussion
In most real user queries we sampled, the entities were tagged at the correct boundary and the tagging was complete (such as the ones shown in Figure FIGREF24)  Only on a few user queries the tagging was controversial  For example, the query “Erythematous blanching round, oval patches on torso, extremities” was tagged as “Erythematous blanching” and “oval patches on torso”  The entity “extremities” was missing  The segmentation was also not correct  A more appropriate tagging would be “Erythematous blanching round, oval patches”, “torso”, and “extremities”  The tagging could be further improved by synthesizing more realistic user queries  Recall that the synthesized user queries were created by randomly combining terminologies from the dermatology glossary, which, while providing data that helped the model learn entity segmentation, did not reflect the co-occurrence information in real user queries  For example, there could be two clinical entities that often co-occur or never co-occur in a user query  But since the synthesized user queries we used combined terms randomly, the co-occurrence information was thus missing 
The final retrieval results of our framework were not evaluated quantitatively in terms of recall and precision, due the the lack of ground truth  When ground truth becomes available, we will be able to evaluate our framework more thoroughly  Recently, a fine-tuned BERT model in the medical domain called BioBERT BIBREF17 has attracted some attention in the medical NLP domain  We could experiment with BioBERT embeddings in the future  We could also include query expansion technique for term matching  When finding candidate terms for an entity, our first step was still based on string matching  Given that there might be multiple entities that could be matched to the same term, it could be hard to include all these entities in the glossary and hard to match terms to these entities 
Conclusion
In this project, we tackle the problem of extracting clinical concepts from user queries on medical search engines  By training a BiLSTM-CRF model on a hybrid data consisting of synthesized user queries and sentences from clinical note, we adopt a CE framework for clinical user queries with minimal effort spent on annotating user queries  We find that the hybrid data enables the NER model perform better on both tagging the user queries and the clinical note sentences  Furthermore, our framework is built on an easy-to-use deep learning NLP Python library, which lends it more prospective value to various online medical applications that employ medical search engines 
Acknowledgment
This paper results from a technical report of a project the authors have worked on with visualDx, a healthcare informatics company that provides web-based clinical decision support system  The authors would like to thank visualDx for providing them the opportunity to work on such an exciting project  In particular, the authors would like to thank Roy Robinson, the Vice President of Technology and Medical Informatics at visualDx, for providing the synthesized user queries, as well as preliminary feedback on the performance of our framework 

ground_truth: clinical notes from the CE task in 2010 i2b2/VA

clinical notes from the CE task in 2010 i2b2/VA 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What cyberbulling topics did they address?
Context 4:  For M2M systems, we over-sampled Ja INLINEFORM0 Ru and Ja INLINEFORM1 En training data so that their sizes match the largest Ru INLINEFORM2 En data  To reduce the number of unknown words, we used tensor2tensor's internal sub-word segmentation mechanism  Since we work in a low-resource setting, we used shared sub-word vocabularies of size 16k for the uni- and bi-directional models and 32k for the M2M models  The number of training iterations was determined by early-stopping: we evaluated our models on the development set every 1,000 updates, and stopped training if BLEU score for the development set was not improved for 10,000 updates (10 check-points)  Note that the development set was created by concatenating those for the individual translation directions without any over-sampling 
Having trained the models, we averaged the last 10 check-points and decoded the test sets with a beam size of 4 and a length penalty which was tuned by a linear search on the BLEU score for the development set 
Similarly to PBSMT, we also evaluated “Cascade” and “Synthesize” methods with uni-directional NMT models 
Results
We evaluated MT models using case-sensitive and tokenized BLEU BIBREF23 on test sets, using Moses's multi-bleu perl  Statistical significance ( INLINEFORM0 ) on the difference of BLEU scores was tested by Moses's bootstrap-hypothesis-difference-significance pl 
Tables TABREF27 and TABREF31 show BLEU scores of all the models, except the NMT systems augmented with back-translations  Whereas some models achieved reasonable BLEU scores for Ja INLINEFORM0 En and Ru INLINEFORM1 En translation, all the results for Ja INLINEFORM2 Ru, which is our main concern, were abysmal 
Among the NMT models, Transformer models (b INLINEFORM0 ) were proven to be better than RNMT models (a INLINEFORM1 )  RNMT models could not even outperform the uni-directional PBSMT models (c1)  M2M models (a3) and (b3) outperformed their corresponding uni- and bi-directional models in most cases  It is worth noting that in this extremely low-resource scenario, BLEU scores of the M2M RNMT model for the largest language pair, i e , Ru INLINEFORM2 En, were lower than those of the uni- and bi-directional RNMT models as in TACL1081  In contrast, with the M2M Transformer model, Ru INLINEFORM3 En also benefited from multilingualism 
Standard PBSMT models (c1) achieved higher BLEU scores than uni-directional NMT models (a1) and (b1), as reported by koehn-knowles:2017:NMT, whereas they underperform the M2M Transformer NMT model (b3)  As shown in Table TABREF31 , pivot-based PBSMT systems always achieved higher BLEU scores than (c1)  The best model with three phrase tables, labeled “Synthesize / Triangulate / Gold,” brought visible BLEU gains with substantial reduction of OOV tokens (3047 INLINEFORM0 1180 for Ja INLINEFORM1 Ru, 4463 INLINEFORM2 1812 for Ru INLINEFORM3 Ja)  However, further extension with phrase tables induced from monolingual data did not push the limit, despite their high coverage; only 336 and 677 OOV tokens were left for the two translation directions, respectively  This is due to the poor quality of the bilingual word embeddings used to extract the phrase table, as envisaged in Section SECREF3  
None of pivot-based approaches with uni-directional NMT models could even remotely rival the M2M Transformer NMT model (b3) 
Table TABREF46 shows the results of our multistage fine-tuning, where the IDs of each row refer to those described in Section SECREF41   First of all, the final models of our multistage fine-tuning, i e , V and VII, achieved significantly higher BLEU scores than (b3) in Table TABREF27 , a weak baseline without using any monolingual data, and #10 in Table TABREF33 , a strong baseline established with monolingual data 
The performance of the initial model (I) depends on the language pair  For Ja INLINEFORM0 Ru pair, it cannot achieve minimum level of quality since the model has never seen parallel data for this pair  The performance on Ja INLINEFORM1 En pair was much lower than the two baseline models, reflecting the crucial mismatch between training and testing domains  In contrast, Ru INLINEFORM2 En pair benefited the most and achieved surprisingly high BLEU scores  The reason might be due to the proximity of out-of-domain training data and in-domain test data 
The first fine-tuning stage significantly pushed up the translation quality for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs, in both cases with fine-tuning (II) and mixed fine-tuning (III)  At this stage, both models performed only poorly for Ja INLINEFORM2 Ru pair as they have not yet seen Ja INLINEFORM3 Ru parallel data  Either model had a consistent advantage to the other 
When these models were further fine-tuned only on the in-domain Ja INLINEFORM0 Ru parallel data (IV and VI), we obtained translations of better quality than the two baselines for Ja INLINEFORM1 Ru pair  However, as a result of complete ignorance of Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs, the models only produced translations of poor quality for these language pairs  In contrast, mixed fine-tuning for the second fine-tuning stage (V and VII) resulted in consistently better models than conventional fine-tuning (IV and VI), irrespective of the choice at the first stage, thanks to the gradual shift of parameters realized by in-domain Ja INLINEFORM4 En and Ru INLINEFORM5 En parallel data  Unfortunately, the translation quality for Ja INLINEFORM6 En and Ru INLINEFORM7 En pairs sometimes degraded from II and III  Nevertheless, the BLEU scores still retain the large margin against two baselines 
The last three rows in Table TABREF46 present BLEU scores obtained by the methods with fewer fine-tuning steps  The most naive model I', trained on the balanced mixture of whole five types of corpora from scratch, and the model II', obtained through a single-step conventional fine-tuning of I on all the in-domain data, achieved only BLEU scores consistently worse than VII  In contrast, when we merged our two fine-tuning steps into a single mixed fine-tuning on I, we obtained a model III' which is better for the Ja INLINEFORM0 Ru pair than VII  Nevertheless, they are still comparable to those of VII and the BLEU scores for the other two language pairs are much lower than VII  As such, we conclude that our multistage fine-tuning leads to a more robust in-domain multilingual model 
Augmentation with Back-translation
Given that the M2M Transformer NMT model (b3) achieved best results for most of the translation directions and competitive results for the rest, we further explored it through back-translation 
We examined the utility of pseudo-parallel data for all the six translation directions, unlike the work of lakew2017improving and lakew2018comparison, which concentrate only on the zero-shot language pair, and the work of W18-2710, which compares only uni- or bi-directional models  We investigated whether each translation direction in M2M models will benefit from pseudo-parallel data and if so, what kind of improvement takes place 
First, we selected sentences to be back-translated from in-domain monolingual data (Table TABREF13 ), relying on the score proposed by moore:intelligent via the following procedure 
For each language, train two 4-gram language models, using KenLM: an in-domain one on all the Global Voices data, i e , both parallel and monolingual data, and a general-domain one on the concatenation of Global Voices, IWSLT, and Tatoeba data 
For each language, discard sentences containing OOVs according to the in-domain language model 
For each translation direction, select the INLINEFORM0 -best monolingual sentences in the news domain, according to the difference between cross-entropy scores given by the in-domain and general-domain language models 
Whereas W18-2710 exploited monolingual data much larger than parallel data, we maintained a 1:1 ratio between them BIBREF8 , setting INLINEFORM0 to the number of lines of parallel data of given language pair 
Selected monolingual sentences were then translated using the M2M Transformer NMT model (b3) to compose pseudo-parallel data  Then, the pseudo-parallel data were enlarged by over-sampling as summarized in Table TABREF32   Finally, new NMT models were trained on the concatenation of the original parallel and pseudo-parallel data from scratch in the same manner as the previous NMT models with the same hyper-parameters 
Table TABREF33 shows the BLEU scores achieved by several reasonable combinations of six-way pseudo-parallel data  We observed that the use of all six-way pseudo-parallel data (#10) significantly improved the base model for all the translation directions, except En INLINEFORM0 Ru  A translation direction often benefited when the pseudo-parallel data for that specific direction was used 
Summary
We have evaluated an extensive variation of MT models that rely only on in-domain parallel and monolingual data  However, the resulting BLEU scores for Ja INLINEFORM2 Ru and Ru INLINEFORM3 Ja tasks do not exceed 10 BLEU points, implying the inherent limitation of the in-domain data as well as the difficulty of these translation directions 
Exploiting Large Out-of-Domain Data Involving a Helping Language
The limitation of relying only on in-domain data demonstrated in Section SECREF4 motivates us to explore other types of parallel data  As raised in our second research question, [RQ2], we considered the effective ways to exploit out-of-domain data 
According to language pair and domain, parallel data can be classified into four categories in Table TABREF40   Among all the categories, out-of-domain data for the language pair of interest have been exploited in the domain adaptation scenarios (C INLINEFORM0 A) BIBREF9   However, for Ja INLINEFORM1 Ru, no out-of-domain data is available  To exploit out-of-domain parallel data for Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs instead, we propose a multistage fine-tuning method, which combines two types of transfer learning, i e , domain adaptation for Ja INLINEFORM4 En and Ru INLINEFORM5 En (D INLINEFORM6 B) and multilingual transfer (B INLINEFORM7 A), relying on the M2M model examined in Section SECREF4   We also examined the utility of fine-tuning for iteratively generating and using pseudo-parallel data 
Multistage Fine-tuning
Simply using NMT systems trained on out-of-domain data for in-domain translation is known to perform badly  In order to effectively use large-scale out-of-domain data for our extremely low-resource task, we propose to perform domain adaptation through either (a) conventional fine-tuning, where an NMT system trained on out-of-domain data is fine-tuned only on in-domain data, or (b) mixed fine-tuning BIBREF9 , where pre-trained out-of-domain NMT system is fine-tuned using a mixture of in-domain and out-of-domain data  The same options are available for transferring from Ja INLINEFORM0 En and Ru INLINEFORM1 En to Ja INLINEFORM2 Ru 
We inevitably involve two types of transfer learning, i e , domain adaptation for Ja INLINEFORM0 En and Ru INLINEFORM1 En and multilingual transfer for Ja INLINEFORM2 Ru pair  Among several conceivable options for managing these two problems, we examined the following multistage fine-tuning 
Pre-train a multilingual model only on the Ja INLINEFORM0 En and Ru INLINEFORM1 En out-of-domain parallel data (I), where the vocabulary of the model is determined on the basis of the in-domain parallel data in the same manner as the M2M NMT models examined in Section SECREF4  
Fine-tune the pre-trained model (I) on the in-domain Ja INLINEFORM0 En and Ru INLINEFORM1 En parallel data (fine-tuning, II) or on the mixture of in-domain and out-of-domain Ja INLINEFORM2 En and Ru INLINEFORM3 En parallel data (mixed fine-tuning, III) 
Further fine-tune the models (each of II and III) for Ja INLINEFORM0 Ru on in-domain parallel data for this language pair only (fine-tuning, IV and VI) or on all the in-domain parallel data (mixed fine-tuning, V and VII) 
We chose this way due to the following two reasons  First, we need to take a balance between several different parallel corpora sizes  The other reason is division of labor; we assume that solving each sub-problem one by one should enable gradual shift of parameters 
Data Selection
As an additional large-scale out-of-domain parallel data for Ja INLINEFORM0 En, we used the cleanest 1 5M sentences from the Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF24   As for Ru INLINEFORM1 En, we used the UN and Yandex corpora released for the WMT 2018 News Translation Task  We retained Ru INLINEFORM2 En sentence pairs that contain at least one OOV token in both sides, according to the in-domain language model trained in Section SECREF34   Table TABREF45 summarizes the statistics on the remaining out-of-domain parallel data 
Further Augmentation with Back-translation
Having obtained a better model, we examined again the utility of back-translation  More precisely, we investigated (a) whether the pseudo-parallel data generated by an improved NMT model leads to a further improvement, and (b) whether one more stage of fine-tuning on the mixture of original parallel and pseudo-parallel data will result in a model better than training a new model from scratch as examined in Section SECREF34  
Given an NMT model, we first generated six-way pseudo-parallel data by translating monolingual data  For the sake of comparability, we used the identical monolingual sentences sampled in Section SECREF34   Then, we further fine-tuned the given model on the mixture of the generated pseudo-parallel data and the original parallel data, following the same over-sampling procedure in Section SECREF34   We repeated these steps five times 
Table TABREF51 shows the results  “new #10” in the second row indicates an M2M Transformer model trained from scratch on the mixture of six-way pseudo-parallel data generated by VII and the original parallel data  It achieved higher BLEU scores than #10 in Table TABREF33 thanks to the pseudo-parallel data of better quality, but underperformed the base NMT model VII  In contrast, our fine-tuned model VIII successfully surpassed VII, and one more iteration (IX) further improved BLEU scores for all translation directions, except Ru INLINEFORM0 En  Although further iterations did not necessarily gain BLEU scores, we came to a much higher plateau compared to the results in Section SECREF4  
Conclusion
In this paper, we challenged the difficult task of Ja INLINEFORM0 Ru news domain translation in an extremely low-resource setting  We empirically confirmed the limited success of well-established solutions when restricted to in-domain data  Then, to incorporate out-of-domain data, we proposed a multilingual multistage fine-tuning approach and observed that it substantially improves Ja INLINEFORM1 Ru translation by over 3 7 BLEU points compared to a strong baseline, as summarized in Table TABREF53   This paper contains an empirical comparison of several existing approaches and hence we hope that our paper can act as a guideline to researchers attempting to tackle extremely low-resource translation 
In the future, we plan to confirm further fine-tuning for each of specific translation directions  We will also explore the way to exploit out-of-domain pseudo-parallel data, better domain-adaptation approaches, and additional challenging language pairs 
Acknowledgments
This work was carried out when Aizhan Imankulova was taking up an internship at NICT, Japan  We would like to thank the reviewers for their insightful comments  A part of this work was conducted under the program “Promotion of Global Communications Plan: Research, Development, and Social Demonstration of Multilingual Speech Translation Technology” of the Ministry of Internal Affairs and Communications (MIC), Japan 

ground_truth: personal attack, racism, and sexism

racism, sexism, personal attack, not specifically about any single topic