prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they use attention?
Context 1: Background
Teaching machine to read and comprehend a given passage/paragraph and answer its corresponding questions is a challenging task  It is also one of the long-term goals of natural language understanding, and has important applications in e g , building intelligent agents for conversation and customer service support  In a real world setting, it is necessary to judge whether the given questions are answerable given the available knowledge, and then generate correct answers for the ones which are able to infer an answer in the passage or an empty answer (as an unanswerable question) otherwise 
In comparison with many existing MRC systems BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , which extract answers by finding a sub-string in the passages/paragraphs, we propose a model that not only extracts answers but also predicts whether such an answer should exist  Using a multi-task learning approach (c f  BIBREF5 ), we extend the Stochastic Answer Network (SAN) BIBREF1 for MRC answer span detector to include a classifier that whether the question is unanswerable  The unanswerable classifier is a pair-wise classification model BIBREF6 which predicts a label indicating whether the given pair of a passage and a question is unanswerable  The two models share the same lower layer to save the number of parameters, and separate the top layers for different tasks (the span detector and binary classifier) 
Our model is pretty simple and intuitive, yet efficient  Without relying on the large pre-trained language models (ELMo) BIBREF7 , the proposed model achieves competitive results to the state-of-the-art on Stanford Question Answering Dataset (SQuAD) 2 0 
The contribution of this work is summarized as follows  First, we propose a simple yet efficient model for MRC that handles unanswerable questions and is optimized jointly  Second, our model achieves competitive results on SQuAD v2 0 
Model
The Machine Reading Comprehension is a task which takes a question INLINEFORM0 and a passage/paragraph INLINEFORM1 as inputs, and aims to find an answer span INLINEFORM2 in INLINEFORM3   We assume that if the question is answerable, the answer INLINEFORM4 exists in INLINEFORM5 as a contiguous text string; otherwise, INLINEFORM6 is an empty string indicating an unanswerable question  Note that to handle the unanswerable questions, we manually append a dumpy text string NULL at the end of each corresponding passage/paragraph  Formally, the answer is formulated as INLINEFORM7   In case of unanswerable questions, INLINEFORM8 points to the last token of the passage 
Our model is a variation of SAN BIBREF1 , as shown in Figure FIGREF2   The main difference is the additional binary classifier added in the model justifying whether the question is unanswerable  Roughly, the model includes two different layers: the shared layer and task specific layer  The shared layer is almost identical to the lower layers of SAN, which has a lexicon encoding layer, a contextual layer and a memory generation layer  On top of it, there are different answer modules for different tasks  We employ the SAN answer module for the span detector and a one-layer feed forward neural network for the binary classification task  It can also be viewed as a multi-task learning BIBREF8 , BIBREF5 , BIBREF9   We will briefly describe the model from ground up as follows  Detailed descriptions can be found in BIBREF1  
Lexicon Encoding Layer  We map the symbolic/surface feature of INLINEFORM0 and INLINEFORM1 into neural space via word embeddings , 16-dim part-of-speech (POS) tagging embeddings, 8-dim named-entity embeddings and 4-dim hard-rule features  Note that we use small embedding size of POS and NER to reduce model size and they mainly serve the role of coarse-grained word clusters  Additionally, we use question enhanced passages word embeddings which can viewwed as soft matching between questions and passages  At last, we use two separate two-layer position-wise Feed-Forward Networks (FFN) BIBREF11 , BIBREF1 to map both question and passage encodings into the same dimension  As results, we obtain the final lexicon embeddings for the tokens for INLINEFORM2 as a matrix INLINEFORM3 , and tokens in INLINEFORM4 as INLINEFORM5  
Contextual Encoding Layer  A shared two-layers BiLSTM is used on the top to encode the contextual information of both passages and questions  To avoid overfitting, we concatenate a pre-trained 600-dimensional CoVe vectors BIBREF12 trained on German-English machine translation dataset, with the aforementioned lexicon embeddings as the final input of the contextual encoding layer, and also with the output of the first contextual encoding layer as the input of its second encoding layer  Thus, we obtain the final representation of the contextual encoding layer by a concatenation of the outputs of two BiLSTM: INLINEFORM0 for questions and INLINEFORM1 for passages 
Memory Generation Layer  In this layer, we generate a working memory by fusing information from both passages INLINEFORM0 and questions INLINEFORM1   The attention function BIBREF11 is used to compute the similarity score between passages and questions as: INLINEFORM2
Note that INLINEFORM0 and INLINEFORM1 is transformed from INLINEFORM2 and INLINEFORM3 by one layer neural network INLINEFORM4 , respectively  A question-aware passage representation is computed as INLINEFORM5   After that, we use the method of BIBREF13 to apply self attention to the passage: INLINEFORM6
where INLINEFORM0 means that we only drop diagonal elements on the similarity matrix (i e , attention with itself)  At last, INLINEFORM1 and INLINEFORM2 are concatenated and are passed through a BiLSTM to form the final memory: INLINEFORM3  
Span detector  We adopt a multi-turn answer module for the span detector BIBREF1   Formally, at time step INLINEFORM0 in the range of INLINEFORM1 , the state is defined by INLINEFORM2   The initial state INLINEFORM3 is the summary of the INLINEFORM4 : INLINEFORM5 , where INLINEFORM6   Here, INLINEFORM7 is computed from the previous state INLINEFORM8 and memory INLINEFORM9 : INLINEFORM10 and INLINEFORM11   Finally, a bilinear function is used to find the begin and end point of answer spans at each reasoning step INLINEFORM12 : DISPLAYFORM0 DISPLAYFORM1
The final prediction is the average of each time step: INLINEFORM0   We randomly apply dropout on the step level in each time step during training, as done in BIBREF1  
Unanswerable classifier  We adopt a one-layer neural network as our unanswerable binary classifier: DISPLAYFORM0
, where INLINEFORM0 is the summary of the memory: INLINEFORM1 , where INLINEFORM2   INLINEFORM3 denotes the probability of the question which is unanswerable 
Objective The objective function of the joint model has two parts: DISPLAYFORM0
Following BIBREF0 , the span loss function is defined: DISPLAYFORM0
The objective function of the binary classifier is defined: DISPLAYFORM0
where INLINEFORM0 is a binary variable: INLINEFORM1 indicates the question is unanswerable and INLINEFORM2 denotes the question is answerable 
Setup
We evaluate our system on SQuAD 2 0 dataset BIBREF14 , a new MRC dataset which is a combination of Stanford Question Answering Dataset (SQuAD) 1 0 BIBREF15 and additional unanswerable question-answer pairs  The answerable pairs are around 100K; while the unanswerable questions are around 53K  This dataset contains about 23K passages and they come from approximately 500 Wikipedia articles  All the questions and answers are obtained by crowd-sourcing  Two evaluation metrics are used: Exact Match (EM) and Macro-averaged F1 score (F1) BIBREF14  
Implementation details
We utilize spaCy tool to tokenize the both passages and questions, and generate lemma, part-of-speech and named entity tags  The word embeddings are initialized with pre-trained 300-dimensional GloVe BIBREF10   A 2-layer BiLSTM is used encoding the contextual information of both questions and passages  Regarding the hidden size of our model, we search greedily among INLINEFORM0   During training, Adamax BIBREF16 is used as our optimizer  The min-batch size is set to 32  The learning rate is initialized to 0 002 and it is halved after every 10 epochs  The dropout rate is set to 0 1  To prevent overfitting, we also randomly set 0 5% words in both passages and questions as unknown words during the training  Here, we use a special token unk to indicate a word which doesn't appear in GloVe  INLINEFORM1 in Eq EQREF9 is set to 1 
Results
We would like to investigate effectiveness the proposed joint model  To do so, the same shared layer/architecture is employed in the following variants of the proposed model:
The results in terms of EM and F1 is summarized in Table TABREF20   We observe that Joint SAN outperforms the SAN baseline with a large margin, e g , 67 89 vs 69 27 (+1 38) and 70 68 vs 72 20 (+1 52) in terms of EM and F1 scores respectively, so it demonstrates the effectiveness of the joint optimization  By incorporating the output information of classifier into Joint SAN, it obtains a slight improvement, e g , 72 2 vs 72 66 (+0 46) in terms of F1 score  By analyzing the results, we found that in most cases when our model extract an NULL string answer, the classifier also predicts it as an unanswerable question with a high probability 
Table TABREF21 reports comparison results in literature published   Our model achieves state-of-the-art on development dataset in setting without pre-trained large language model (ELMo)  Comparing with the much complicated model R M -Reader + Verifier, which includes several components, our model still outperforms by 0 7 in terms of F1 score  Furthermore, we observe that ELMo gives a great boosting on the performance, e g , 2 8 points in terms of F1 for DocQA  This encourages us to incorporate ELMo into our model in future 
Analysis  To better understand our model, we analyze the accuracy of the classifier in our joint model  We obtain 75 3 classification accuracy on the development with the threshold 0 5  By increasing value of INLINEFORM0 in Eq EQREF9 , the classification accuracy reached to 76 8 ( INLINEFORM1 ), however the final results of our model only have a small improvement (+0 2 in terms of F1 score)  It shows that it is important to make balance between these two components: the span detector and unanswerable classifier 
Conclusion
To sum up, we proposed a simple yet efficient model based on SAN  It showed that the joint learning algorithm boosted the performance on SQuAD 2 0  We also would like to incorporate ELMo into our model in future 
Acknowledgments
We thank Yichong Xu, Shuohang Wang and Sheng Zhang for valuable discussions and comments  We also thank Robin Jia for the help on SQuAD evaluations 
Context 2: Introduction
Speech-to-Text translation (ST) is essential for a wide range of scenarios: for example in emergency calls, where agents have to respond emergent requests in a foreign language BIBREF0; or in online courses, where audiences and speakers use different languages BIBREF1  To tackle this problem, existing approaches can be categorized into cascaded method BIBREF2, BIBREF3, where a machine translation (MT) model translates outputs of an automatic speech recognition (ASR) system into target language, and end-to-end method BIBREF4, BIBREF5, where a single model learns acoustic frames to target word sequence mappings in one step towards the final objective of interest  Although the cascaded model remains the dominant approach due to its better performance, the end-to-end method becomes more and more popular because it has lower latency by avoiding inferences with two models and rectifies the error propagation in theory 
Since it is hard to obtain a large-scale ST dataset, multi-task learning BIBREF5, BIBREF6 and pre-training techniques BIBREF7 have been applied to end-to-end ST model to leverage large-scale datasets of ASR and MT  A common practice is to pre-train two encoder-decoder models for ASR and MT respectively, and then initialize the ST model with the encoder of the ASR model and the decoder of the MT model  Subsequently, the ST model is optimized with the multi-task learning by weighing the losses of ASR, MT, and ST  This approach, however, causes a huge gap between pre-training and fine-tuning, which are summarized into three folds:
Subnet Waste: The ST system just reuses the ASR encoder and the MT decoder, while discards other pre-trained subnets, such as the MT encoder  Consequently, valuable semantic information captured by the MT encoder cannot be inherited by the final ST system 
Role Mismatch: The speech encoder plays different roles in pre-training and fine-tuning  The encoder is a pure acoustic model in pre-training, while it has to extract semantic and linguistic features additionally in fine-tuning, which significantly increases the learning difficulty 
Non-pre-trained Attention Module: Previous work BIBREF6 trains attention modules for ASR, MT and ST respectively, hence, the attention module of ST does not benefit from the pre-training 
To address these issues, we propose a Tandem Connectionist Encoding Network (TCEN), which is able to reuse all subnets in pre-training, keep the roles of subnets consistent, and pre-train the attention module  Concretely, the TCEN consists of three components, a speech encoder, a text encoder, and a target text decoder  Different from the previous work that pre-trains an encoder-decoder based ASR model, we only pre-train an ASR encoder by optimizing the Connectionist Temporal Classification (CTC) BIBREF8 objective function  In this way, the additional decoder of ASR is not required while keeping the ability to read acoustic features into the source language space by the speech encoder  Besides, the text encoder and decoder can be pre-trained on a large MT dataset  After that, we employ common used multi-task learning method to jointly learn ASR, MT and ST tasks 
Compared to prior works, the encoder of TCEN is a concatenation of an ASR encoder and an MT encoder and our model does not have an ASR decoder, so the subnet waste issue is solved  Furthermore, the two encoders work at tandem, disentangling acoustic feature extraction and linguistic feature extraction, ensuring the role consistency between pre-training and fine-tuning  Moreover, we reuse the pre-trained MT attention module in ST, so we can leverage the alignment information learned in pre-training 
Since the text encoder consumes word embeddings of plausible texts in MT task but uses speech encoder outputs in ST task, another question is how one guarantees the speech encoder outputs are consistent with the word embeddings  We further modify our model to achieve semantic consistency and length consistency  Specifically, (1) the projection matrix at the CTC classification layer for ASR is shared with the word embedding matrix, ensuring that they are mapped to the same latent space, and (2) the length of the speech encoder output is proportional to the length of the input frame, so it is much longer than a natural sentence  To bridge the length gap, source sentences in MT are lengthened by adding word repetitions and blank tokens to mimic the CTC output sequences 
We conduct comprehensive experiments on the IWSLT18 speech translation benchmark BIBREF1, demonstrating the effectiveness of each component  Our model is significantly better than previous methods by 3 6 and 2 2 BLEU scores for the subword-level decoding and character-level decoding strategies, respectively 
Our contributions are three-folds: 1) we shed light on why previous ST models cannot sufficiently utilize the knowledge learned from the pre-training process; 2) we propose a new ST model, which alleviates shortcomings in existing methods; and 3) we empirically evaluate the proposed model on a large-scale public dataset 
Background ::: Problem Formulation
End-to-end speech translation aims to translate a piece of audio into a target-language translation in one step  The raw speech signals are usually converted to sequences of acoustic features, e g  Mel filterbank features  Here, we define the speech feature sequence as $\mathbf {x} = (x_1, \cdots , x_{T_x})$ The transcription and translation sequences are denoted as $\mathbf {y^{s}} = (y_1^{s}, \cdots , y_{T_s}^{s})$, and $\mathbf {y^{t}} = (y_1^{t}, \cdots , y_{T_t}^{t})$ repectively  Each symbol in $\mathbf {y^{s}}$ or $\mathbf {y^{t}}$ is an integer index of the symbol in a vocabulary $V_{src}$ or $V_{trg}$ respectively (e g  $y^s_i=k, k\in [0, |V_{src}|-1]$)  In this work, we suppose that an ASR dataset, an MT dataset, and a ST dataset are available, denoted as $\mathcal {A} = \lbrace (\mathbf {x_i}, \mathbf {y^{s}_i})\rbrace _{i=0}^I$, $\mathcal {M} =\lbrace (\mathbf {y^{s}_j}, \mathbf {y^{t}_j})\rbrace _{j=0}^J$ and $ \mathcal {S} =\lbrace (\mathbf {x_l}, \mathbf {y^{t}_l})\rbrace _{l=0}^L$ respectively  Given a new piece of audio $\mathbf {x}$, our goal is to learn an end to end model to generate a translation sentence $\mathbf {y^{t}}$ without generating an intermediate result $\mathbf {y^{s}}$ 
Background ::: Multi-Task Learning and Pre-training for ST
To leverage large scale ASR and MT data, multi-task learning and pre-training techniques are widely employed to improve the ST system  As shown in Figure FIGREF4, there are three popular multi-task strategies for ST, including 1) one-to-many setting, in which a speech encoder is shared between ASR and ST tasks; 2) many-to-one setting in which a decoder is shared between MT and ST tasks; and 3) many-to-many setting where both the encoder and decoder are shared 
A many-to-many multi-task model contains two encoders as well as two decoders  It can be jointly trained on ASR, MT, and ST tasks  As the attention module is task-specific, three attentions are defined 
Usually, the size of $\mathcal {A}$ and $\mathcal {M}$ is much larger than $\mathcal {S}$  Therefore, the common training practice is to pre-train the model on ASR and MT tasks and then fine-tune it with a multi-task learning manner  However, as aforementioned, this method suffers from subnet waste, role mismatch and non-pre-trained attention issues, which severely limits the end-to-end ST performance 
Our method
In this section, we first introduce the architecture of TCEN, which consists of two encoders connected in tandem, and one decoder with an attention module  Then we give the pre-training and fine-tuning strategy for TCEN  Finally, we propose our solutions for semantic and length inconsistency problems, which are caused by multi-task learning 
Our method ::: TCEN Architecture
Figure FIGREF5 sketches the overall architecture of TCEN, including a speech encoder $enc_s$, a text encoder $enc_t$ and a decoder $dec$ with an attention module $att$  During training, the $enc_s$ acts like an acoustic model which reads the input $\mathbf {x}$ to word or subword representations $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into hidden representations $\mathbf {h^t}$  Finally, the $dec$ defines a distribution probability over target words  The advantage of our architecture is that two encoders disentangle acoustic feature extraction and linguistic feature extraction, making sure that valuable knowledge learned from ASR and MT tasks can be effectively leveraged for ST training  Besides, every module in pre-training can be utilized in fine-tuning, alleviating the subnet waste problem 
Follow BIBREF9 inaguma2018speech, we use CNN-BiLSTM architecture to build our model  Specifically, the input features $\mathbf {x}$ are organized as a sequence of feature vectors in length $T_x$  Then, $\mathbf {x}$ is passed into a stack of two convolutional layers followed by max-pooling:
where $\mathbf {v}^{(l-1)}$ is feature maps in last layer and $\mathbf {W}^{(l)}$ is the filter  The max-pooling layers downsample the sequence in length by a total factor of four  The down-sampled feature sequence is further fed into a stack of five bidirectional $d$-dimensional LSTM layers:
where $[;]$ denotes the vector concatenation  The final output representation from the speech encoder is denoted as $\mathbf {h^s}=(h^s_1, \cdots , h^s_{\frac{T_x}{4}})$, where $h_i^s \in \mathbb {R}^d$ 
The text encoder $enc_t$ consists of two bidirectional LSTM layers  In ST task, $enc_t$ accepts speech encoder output $\mathbf {h}^s$ as input  While in MT, $enc_t$ consumes the word embedding representation $\mathbf {e^s}$ derived from $\mathbf {y^s}$, where each element $e^s_i$ is computed by choosing the $y_i^s$-th vector from the source embedding matrix $W_{E^s}$  The goal of $enc_t$ is to extract high-level linguistic features like syntactic features or semantic features from lower level subword representations $\mathbf {h}^s$ or $\mathbf {e}^s$  Since $\mathbf {h}^s$ and $\mathbf {e}^s$ belong to different latent space and have different lengths, there remain semantic and length inconsistency problems  We will provide our solutions in Section SECREF21  The output sequence of $enc_t$ is denoted as $\mathbf {h}^t$ 
The decoder is defined as two unidirectional LSTM layers with an additive attention $att$  It predicts target sequence $\mathbf {y^{t}}$ by estimating conditional probability $P(\mathbf {y^{t}}|\mathbf {x})$:
Here, $z_k$ is the the hidden state of the deocder RNN at $k$ step and $c_k$ is a time-dependent context vector computed by the attention $att$ 
Our method ::: Training Procedure
Following previous work, we split the training procedure to pre-training and fine-tuning stages  In pre-training stage, the speech encoder $enc_s$ is trained towards CTC objective using dataset $\mathcal {A}$, while the text encoder $enc_t$ and the decoder $dec$ are trained on MT dataset $\mathcal {M}$  In fine-tuning stage, we jointly train the model on ASR, MT, and ST tasks 
Our method ::: Training Procedure ::: Pre-training
To sufficiently utilize the large dataset $\mathcal {A}$ and $\mathcal {M}$, the model is pre-trained on CTC-based ASR task and MT task in the pre-training stage 
For ASR task, in order to get rid of the requirement for decoder and enable the $enc_s$ to generate subword representation, we leverage connectionist temporal classification (CTC) BIBREF8 loss to train the speech encoder 
Given an input $\mathbf {x}$, $enc_s$ emits a sequence of hidden vectors $\mathbf {h^s}$, then a softmax classification layer predicts a CTC path $\mathbf {\pi }$, where $\pi _t \in V_{src} \cup $ {`-'} is the observing label at particular RNN step $t$, and `-' is the blank token representing no observed labels:
where $W_{ctc} \in \mathbb {R}^{d \times (|V_{src}|+1)}$ is the weight matrix in the classification layer and $T$ is the total length of encoder RNN 
A legal CTC path $\mathbf {\pi }$ is a variation of the source transcription $\mathbf {y}^s$ by allowing occurrences of blank tokens and repetitions, as shown in Table TABREF14  For each transcription $\mathbf {y}$, there exist many legal CTC paths in length $T$  The CTC objective trains the model to maximize the probability of observing the golden sequence $\mathbf {y}^s$, which is calculated by summing the probabilities of all possible legal paths:
where $\Phi _T(y)$ is the set of all legal CTC paths for sequence $\mathbf {y}$ with length $T$  The loss can be easily computed using forward-backward algorithm  More details about CTC are provided in supplementary material 
For MT task, we use the cross-entropy loss as the training objective  During training, $\mathbf {y^s}$ is converted to embedding vectors $\mathbf {e^s}$ through embedding layer $W_{E^s}$, then $enc_t$ consumes $\mathbf {e^s}$ and pass the output $\mathbf {h^t}$ to decoder  The objective function is defined as:
Our method ::: Training Procedure ::: Fine-tune
In fine-tune stage, we jointly update the model on ASR, MT, and ST tasks  The training for ASR and MT follows the same process as it was in pre-training stage 
For ST task, the $enc_s$ reads the input $\mathbf {x}$ and generates $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into $\mathbf {h^t}$  Finally, the $dec$ predicts the target sentence  The ST loss function is defined as:
Following the update strategy proposed by BIBREF11 luong2015multi, we allocate a different training ratio $\alpha _i$ for each task  When switching between tasks, we select randomly a new task $i$ with probability $\frac{\alpha _i}{\sum _{j}\alpha _{j}}$ 
Our method ::: Subnet-Consistency
Our model keeps role consistency between pre-training and fine-tuning by connecting two encoders for ST task  However, this leads to some new problems: 1) The text encoder consumes $\mathbf {e^s}$ during MT training, while it accepts $\mathbf {h^s}$ during ST training  However, $\mathbf {e^s}$ and $\mathbf {h^s}$ may not follow the same distribution, resulting in the semantic inconsistency  2) Besides, the length of $\mathbf {h^s}$ is not the same order of magnitude with the length of $\mathbf {e^s}$, resulting in the length inconsistency 
In response to the above two challenges, we propose two countermeasures: 1) We share weights between CTC classification layer and source-end word embedding layer during training of ASR and MT, encouraging $\mathbf {e^s}$ and $\mathbf {h^s}$ in the same space  2)We feed the text encoder source sentences in the format of CTC path, which are generated from a seq2seq model, making it more robust toward long inputs 
Our method ::: Subnet-Consistency ::: Semantic Consistency
As shown in Figure FIGREF5, during multi-task training, two different hidden features will be fed into the text encoder $enc_t$: the embedding representation $\mathbf {e}^s$ in MT task, and the $enc_s$ output $\mathbf {h^s}$ in ST task  Without any regularization, they may belong to different latent spaces  Due to the space gap, the $enc_t$ has to compromise between two tasks, limiting its performance on individual tasks 
To bridge the space gap, our idea is to pull $\mathbf {h^s}$ into the latent space where $\mathbf {e}^s$ belong  Specifically, we share the weight $W_{ctc}$ in CTC classification layer with the source embedding weights $W_{E^s}$, which means $W_{ctc} = W_{E^s}$  In this way, when predicting the CTC path $\mathbf {\pi }$, the probability of observing the particular label $w_i \in V_{src}\cup ${`-'} at time step $t$, $p(\pi _t=w_i|\mathbf {x})$, is computed by normalizing the product of hidden vector $h_t^s$ and the $i$-th vector in $W_{E^s}$:
The loss function closes the distance between $h^s_t$ and golden embedding vector, encouraging $\mathbf {h}^s$ have the same distribution with $\mathbf {e}^s$ 
Our method ::: Subnet-Consistency ::: Length Consistency
Another existing problem is length inconsistency  The length of the sequence $\mathbf {h^s}$ is proportional to the length of the input frame $\mathbf {x}$, which is much longer than the length of $\mathbf {e^s}$  To solve this problem, we train an RNN-based seq2seq model to transform normal source sentences to noisy sentences in CTC path format, and replace standard MT with denoising MT for multi-tasking 
Specifically, we first train a CTC ASR model based on dataset $\mathcal {A} = \lbrace (\mathbf {x}_i, \mathbf {y}^s_i)\rbrace _{i=0}^{I}$, and generate a CTC-path $\mathbf {\pi }_i$ for each audio $\mathbf {x}_i$ by greedy decoding  Then we define an operation $S(\cdot )$, which converts a CTC path $\mathbf {\pi }$ to a sequence of the unique tokens $\mathbf {u}$ and a sequence of repetition times for each token $\mathbf {l}$, denoted as $S(\mathbf {\pi }) = (\mathbf {u}, \mathbf {l})$  Notably, the operation is reversible, meaning that $S^{-1} (\mathbf {u}, \mathbf {l})=\mathbf {\pi }$  We use the example $\mathbf {\pi _1}$ in Table TABREF14 and show the corresponding $\mathbf {u}$ and $\mathbf {l}$ in Table TABREF24 
Then we build a dataset $\mathcal {P} = \lbrace (\mathbf {y^s}_i, \mathbf {u}_i, \mathbf {l}_i)\rbrace _{i=0}^{I}$ by decoding all the audio pieces in $\mathcal {A}$ and transform the resulting path by the operation $S(\cdot )$  After that, we train a seq2seq model, as shown in Figure FIGREF25, which takes $ \mathbf {y^s}_i$ as input and decodes $\mathbf {u}_i, \mathbf {l}_i$ as outputs  With the seq2seq model, a noisy MT dataset $\mathcal {M}^{\prime }=\lbrace (\mathbf {\pi }_l, \mathbf {y^t}_l)\rbrace _{l=0}^{L}$ is obtained by converting every source sentence $\mathbf {y^s}_i \in \mathcal {M}$ to $\mathbf {\pi _i}$, where $\mathbf {\pi }_i = S^{-1}(\mathbf {u}_i, \mathbf {l}_i)$  We did not use the standard seq2seq model which takes $\mathbf {y^s}$ as input and generates $\mathbf {\pi }$ directly, since there are too many blank tokens `-' in $\mathbf {\pi }$ and the model tends to generate a long sequence with only blank tokens  During MT training, we randomly sample text pairs from $\mathcal {M}^{\prime }$ and $\mathcal {M}$ according to a hyper-parameter $k$  After tuning on the validation set, about $30\%$ pairs are sampled from $\mathcal {M}^{\prime }$  In this way, the $enc_t$ is more robust toward the longer inputs given by the $enc_s$ 
Experiments
We conduct experiments on the IWSLT18 speech translation task BIBREF1  Since IWSLT participators use different data pre-processing methods, we reproduce several competitive baselines based on the ESPnet BIBREF12 for a fair comparison 
Experiments ::: Dataset ::: Speech translation data:
The organizer provides a speech translation corpus extracting from the TED talk (ST-TED), which consists of raw English wave files, English transcriptions, and aligned German translations  The corpus contains 272 hours of English speech with 171k segments  We split 2k segments from the corpus as dev set and tst2010, tst2013, tst2014, tst2015 are used as test sets 
Speech recognition data: Aside from ST-TED, TED-LIUM2 corpus BIBREF13 is provided as speech recognition data, which contains 207 hours of English speech and 93k transcript sentences 
Text translation data: We use transcription and translation pairs in the ST-TED corpus and WIT3 as in-domain MT data, which contains 130k and 200k sentence pairs respectively  WMT2018 is used as out-of-domain training data which consists of 41M sentence pairs 
Data preprocessing: For speech data, the utterances are segmented into multiple frames with a 25 ms window size and a 10 ms step size  Then we extract 80-channel log-Mel filter bank and 3-dimensional pitch features using Kaldi BIBREF14, resulting in 83-dimensional input features  We normalize them by the mean and the standard deviation on the whole training set  The utterances with more than 3000 frames are discarded  The transcripts in ST-TED are in true-case with punctuation while in TED-LIUM2, transcripts are in lower-case and unpunctuated  Thus, we lowercase all the sentences and remove the punctuation to keep consistent  To increase the amount of training data, we perform speed perturbation on the raw signals with speed factors 0 9 and 1 1  For the text translation data, sentences longer than 80 words or shorter than 10 words are removed  Besides, we discard pairs whose length ratio between source and target sentence is smaller than 0 5 or larger than 2 0  Word tokenization is performed using the Moses scripts and both English and German words are in lower-case 
We use two different sets of vocabulary for our experiments  For the subword experiments, both English and German vocabularies are generated using sentencepiece BIBREF15 with a fixed size of 5k tokens  BIBREF9 inaguma2018speech show that increasing the vocabulary size is not helpful for ST task  For the character experiments, both English and German sentences are represented in the character level 
For evaluation, we segment each audio with the LIUM SpkDiarization tool BIBREF16 and then perform MWER segmentation with RWTH toolkit BIBREF17  We use lowercase BLEU as evaluation metric 
Experiments ::: Baseline Models and Implementation
We compare our method with following baselines 
Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder  It is trained from scratch on the ST-TED corpus 
Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained  The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus  The MT model has a text encoder and decoder with the same architecture of which in TCEN  It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data 
Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting  In the first two settings, we train the model with $\alpha _{st}=0 75$ while $\alpha _{asr}=0 25$ or $\alpha _{mt}=0 25$  For many-to-many setting, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$   For MT task, we use only in-domain data 
Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models  Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation  Their model solves the subnet waste issue by concatenating an ST decoder to an ASR encoder-decoder model  Notably, their ST decoder can consume representations from the speech encoder as well as the ASR decoder  For a fair comparison, the speech encoder and the ASR decoder are initialized from the pre-trained ASR model  The Triangle model is fine-tuned under their multi-task manner 
All our baselines as well as TCEN are implemented based on ESPnet BIBREF12, the RNN size is set as $d=1024$ for all models  We use a dropout of 0 3 for embeddings and encoders, and train using Adadelta with initial learning rate of 1 0 for a maximum of 10 epochs 
For training of TCEN, we set $\alpha _{asr}=0 2$ and $\alpha _{mt}=0 8$ in the pre-training stage, since the MT dataset is much larger than ASR dataset  For fine-tune, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$, same as the `many-to-many' baseline 
For testing, we select the model with the best accuracy on speech translation task on dev set  At inference time, we use a beam size of 10, and the beam scores include length normalization with a weight of 0 2 
Experiments ::: Experimental Results
Table TABREF29 shows the results on four test sets as well as the average performance  Our method significantly outperforms the strong `many-to-many+pretrain' baseline by 3 6 and 2 2 BLEU scores respectively, indicating the proposed method is very effective that substantially improves the translation quality  Besides, both pre-training and multi-task learning can improve translation quality, and the pre-training settings (2nd-4th rows) are more effective compared to multi-task settings (5th-8th rows)  We observe a performance degradation in the `triangle+pretrain' baseline  Compared to our method, where the decoder receives higher-level syntactic and semantic linguistic knowledge extracted from text encoder, their ASR decoder can only provide lower word-level linguistic information  Besides, since their model lacks text encoder and the architecture of ST decoder is different from MT decoder, their model cannot utilize the large-scale MT data in all the training stages  Interestingly, we find that the char-level models outperform the subword-level models in all settings, especially in vanilla baseline  A similar phenomenon is observed by BIBREF6 berard2018end  A possible explanation is that learning the alignments between speech frames and subword units in another language is notoriously difficult  Our method can bring more gains in the subword setting since our model is good at learning the text-to-text alignment and the subword-level alignment is more helpful to the translation quality 
Experiments ::: Discussion ::: Ablation Study
To better understand the contribution of each component, we perform an ablation study on subword-level experiments  The results are shown in Table TABREF37  In `-MT noise' setting, we do not add noise to source sentences for MT  In `-weight sharing' setting, we use different parameters in CTC classification layer and source embedding layer  These two experiments prove that both weight sharing and using noisy MT input benefit to the final translation quality  Performance degrades more in `-weight sharing', indicating the semantic consistency contributes more to our model  In the `-pretrain' experiment, we remove the pre-training stage and directly update the model on three tasks, leading to a dramatic decrease on BLEU score, indicating the pre-training is an indispensable step for end-to-end ST 
Experiments ::: Discussion ::: Learning Curve
It is interesting to investigate why our method is superior to baselines  We find that TCEN achieves a higher final result owing to a better start-point in fine-tuning  Figure FIGREF39 provides learning curves of subword accuracy on validation set  The x-axis denotes the fine-tuning training steps  The vanilla model starts at a low accuracy, because its networks are not pre-trained on the ASR and MT data  The trends of our model and `many-to-many+pretrain' are similar, but our model outperforms it about five points in the whole fine-tuning process  It indicates that the gain comes from bridging the gap between pre-training and fine-tuning rather than a better fine-tuning process 
Experiments ::: Discussion ::: Compared with a Cascaded System
Table TABREF29 compares our model with end-to-end baselines  Here, we compare our model with cascaded systems  We build a cascaded system by combining the ASR model and MT model used in pre-training baseline  Word error rate (WER) of the ASR system and BLEU score of the MT system are reported in the supplementary material  In addition to a simple combination of the ASR and MT systems, we also re-segment the ASR outputs before feeding to the MT system, denoted as cascaded+re-seg  Specifically, we train a seq2seq model BIBREF19 on the MT dataset, where the source side is a no punctuation sentence and the target side is a natural sentence  After that, we use the seq2seq model to add sentence boundaries and punctuation on ASR outputs  Experimental results are shown in Table TABREF41  Our end-to-end model outperforms the simple cascaded model over 2 BLEU scores, and it achieves a comparable performance with the cascaded model combining with a sentence re-segment model 
Related Work
Early works conduct speech translation in a pipeline manner BIBREF2, BIBREF20, where the ASR output lattices are fed into an MT system to generate target sentences  HMM BIBREF21, DenseNet BIBREF22, TDNN BIBREF23 are commonly used ASR systems, while RNN with attention BIBREF19 and Transformer BIBREF10 are top choices for MT  To enhance the robustness of the NMT model towards ASR errors, BIBREF24 DBLP:conf/eacl/TsvetkovMD14 and BIBREF25 DBLP:conf/asru/ChenHHL17 propose to simulate the noise in training and inference 
To avoid error propagation and high latency issues, recent works propose translating the acoustic speech into text in target language without yielding the source transcription BIBREF4  Since ST data is scarce, pre-training BIBREF7, multi-task learning BIBREF4, BIBREF6, curriculum learning BIBREF26, attention-passing BIBREF27, and knowledge distillation BIBREF28, BIBREF29 strategies have been explored to utilize ASR data and MT data  Specifically, BIBREF5 DBLP:conf/interspeech/WeissCJWC17 show improvements of performance by training the ST model jointly with the ASR and the MT model  BIBREF6 berard2018end observe faster convergence and better results due to pre-training and multi-task learning on a larger dataset  BIBREF7 DBLP:conf/naacl/BansalKLLG19 show that pre-training a speech encoder on one language can improve ST quality on a different source language  All of them follow the traditional multi-task training strategies  BIBREF26 DBLP:journals/corr/abs-1802-06003 propose to use curriculum learning to improve ST performance on syntactically distant language pairs  To effectively leverage transcriptions in ST data, BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 augment the multi-task model where the target decoder receives information from the source decoder and they show improvements on low-resource speech translation  Their model just consumes ASR and ST data, in contrast, our work sufficiently utilizes the large-scale MT data to capture the rich semantic knowledge  BIBREF30 DBLP:conf/icassp/JiaJMWCCALW19 use pre-trained MT and text-to-speech (TTS) synthesis models to convert weakly supervised data into ST pairs and demonstrate that an end-to-end MT model can be trained using only synthesised data 
Conclusion
This paper has investigated the end-to-end method for ST  It has discussed why there is a huge gap between pre-training and fine-tuning in previous methods  To alleviate these issues, we have proposed a method, which is capable of reusing every sub-net and keeping the role of sub-net consistent between pre-training and fine-tuning  Empirical studies have demonstrated that our model significantly outperforms baselines 

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the attention module pretrained on?
Context 4: Introduction
There exists a class of language construction known as pun in natural language texts and utterances, where a certain word or other lexical items are used to exploit two or more separate meanings  It has been shown that understanding of puns is an important research question with various real-world applications, such as human-computer interaction BIBREF0 , BIBREF1 and machine translation BIBREF2   Recently, many researchers show their interests in studying puns, like detecting pun sentences BIBREF3 , locating puns in the text BIBREF4 , interpreting pun sentences BIBREF5 and generating sentences containing puns BIBREF6 , BIBREF7 , BIBREF8   A pun is a wordplay in which a certain word suggests two or more meanings by exploiting polysemy, homonymy, or phonological similarity to another sign, for an intended humorous or rhetorical effect  Puns can be generally categorized into two groups, namely heterographic puns (where the pun and its latent target are phonologically similar) and homographic puns (where the two meanings of the pun reflect its two distinct senses) BIBREF9   Consider the following two examples:
The first punning joke exploits the sound similarity between the word “propane" and the latent target “profane", which can be categorized into the group of heterographic puns  Another categorization of English puns is homographic pun, exemplified by the second instance leveraging distinct senses of the word “gut" 
Pun detection is the task of detecting whether there is a pun residing in the given text  The goal of pun location is to find the exact word appearing in the text that implies more than one meanings  Most previous work addresses such two tasks separately and develop separate systems BIBREF10 , BIBREF5   Typically, a system for pun detection is built to make a binary prediction on whether a sentence contains a pun or not, where all instances (with or without puns) are taken into account during training  For the task of pun location, a separate system is used to make a single prediction as to which word in the given sentence in the text that trigger more than one semantic interpretations of the text, where the training data involves only sentences that contain a pun  Therefore, if one is interested in solving both problems at the same time, a pipeline approach that performs pun detection followed by pun location can be used 
Compared to the pipeline methods, joint learning has been shown effective BIBREF11 , BIBREF12 since it is able to reduce error propagation and allows information exchange between tasks which is potentially beneficial to all the tasks  In this work, we demonstrate that the detection and location of puns can be jointly addressed by a single model  The pun detection and location tasks can be combined as a sequence labeling problem, which allows us to jointly detect and locate a pun in a sentence by assigning each word a tag  Since each context contains a maximum of one pun BIBREF9 , we design a novel tagging scheme to capture this structural constraint  Statistics on the corpora also show that a pun tends to appear in the second half of a context  To capture such a structural property, we also incorporate word position knowledge into our structured prediction model  Experiments on the benchmark datasets show that detection and location tasks can reinforce each other, leading to new state-of-the-art performance on these two tasks  To the best of our knowledge, this is the first work that performs joint detection and location of English puns by using a sequence labeling approach 
Problem Definition
We first design a simple tagging scheme consisting of two tags { INLINEFORM0 }:
INLINEFORM0 tag means the current word is not a pun 
INLINEFORM0 tag means the current word is a pun 
If the tag sequence of a sentence contains a INLINEFORM0 tag, then the text contains a pun and the word corresponding to INLINEFORM1 is the pun 
The contexts have the characteristic that each context contains a maximum of one pun BIBREF9   In other words, there exists only one pun if the given sentence is detected as the one containing a pun  Otherwise, there is no pun residing in the text  To capture this interesting property, we propose a new tagging scheme consisting of three tags, namely { INLINEFORM0 } 
INLINEFORM0 tag indicates that the current word appears before the pun in the given context 
INLINEFORM0 tag highlights the current word is a pun 
INLINEFORM0 tag indicates that the current word appears after the pun 
We empirically show that the INLINEFORM0 scheme can guarantee the context property that there exists a maximum of one pun residing in the text 
Given a context from the training set, we will be able to generate its corresponding gold tag sequence using a deterministic procedure  Under the two schemes, if a sentence does not contain any puns, all words will be tagged with INLINEFORM0 or INLINEFORM1 , respectively  Exemplified by the second sentence “Some diets cause a gut reaction," the pun is given as “gut " Thus, under the INLINEFORM2 scheme, it should be tagged with INLINEFORM3 , while the words before it are assigned with the tag INLINEFORM4 and words after it are with INLINEFORM5 , as illustrated in Figure FIGREF8   Likewise, the INLINEFORM6 scheme tags the word “gut" with INLINEFORM7 , while other words are tagged with INLINEFORM8   Therefore, we can combine the pun detection and location tasks into one problem which can be solved by the sequence labeling approach 
Model
Neural models have shown their effectiveness on sequence labeling tasks BIBREF13 , BIBREF14 , BIBREF15   In this work, we adopt the bidirectional Long Short Term Memory (BiLSTM) BIBREF16 networks on top of the Conditional Random Fields BIBREF17 (CRF) architecture to make labeling decisions, which is one of the classical models for sequence labeling  Our model architecture is illustrated in Figure FIGREF8 with a running example  Given a context/sentence INLINEFORM0 where INLINEFORM1 is the length of the context, we generate the corresponding tag sequence INLINEFORM2 based on our designed tagging schemes and the original annotations for pun detection and location provided by the corpora  Our model is then trained on pairs of INLINEFORM3  
Input  The contexts in the pun corpus hold the property that each pun contains exactly one content word, which can be either a noun, a verb, an adjective, or an adverb  To capture this characteristic, we consider lexical features at the character level  Similar to the work of BIBREF15 , the character embeddings are trained by the character-level LSTM networks on the unannotated input sequences  Nonlinear transformations are then applied to the character embeddings by highway networks BIBREF18 , which map the character-level features into different semantic spaces 
We also observe that a pun tends to appear at the end of a sentence  Specifically, based on the statistics, we found that sentences with a pun that locate at the second half of the text account for around 88% and 92% in homographic and heterographic datasets, respectively  We thus introduce a binary feature that indicates if a word is located at the first or the second half of an input sentence to capture such positional information  A binary indicator can be mapped to a vector representation using a randomly initialized embedding table BIBREF19 , BIBREF20   In this work, we directly adopt the value of the binary indicator as part of the input 
The concatenation of the transformed character embeddings, the pre-trained word embeddings BIBREF21 , and the position indicators are taken as input of our model 
Tagging  The input is then fed into a BiLSTM network, which will be able to capture contextual information  For a training instance INLINEFORM0 , we suppose the output by the word-level BiLSTM is INLINEFORM1   The CRF layer is adopted to capture label dependencies and make final tagging decisions at each position, which has been included in many state-of-the-art sequence labeling models BIBREF14 , BIBREF15   The conditional probability is defined as:
where INLINEFORM0 is a set of all possible label sequences consisting of tags from INLINEFORM1 (or INLINEFORM2 ), INLINEFORM3 and INLINEFORM4 are weight and bias parameters corresponding to the label pair INLINEFORM5   During training, we minimize the negative log-likelihood summed over all training instances:
where INLINEFORM0 refers to the INLINEFORM1 -th instance in the training set  During testing, we aim to find the optimal label sequence for a new input INLINEFORM2 :
This search process can be done efficiently using the Viterbi algorithm 
Datasets and Settings
We evaluate our model on two benchmark datasets BIBREF9   The homographic dataset contains 2,250 contexts, 1,607 of which contain a pun  The heterographic dataset consists of 1,780 contexts with 1,271 containing a pun  We notice there is no standard splitting information provided for both datasets  Thus we apply 10-fold cross validation  To make direct comparisons with prior studies, following BIBREF4 , we accumulated the predictions for all ten folds and calculate the scores in the end 
For each fold, we randomly select 10% of the instances from the training set for development  Word embeddings are initialized with the 100-dimensional Glove BIBREF21   The dimension of character embeddings is 30 and they are randomly initialized, which can be fine tuned during training  The pre-trained word embeddings are not updated during training  The dimensions of hidden vectors for both char-level and word-level LSTM units are set to 300  We adopt stochastic gradient descent (SGD) BIBREF26 with a learning rate of 0

ground_truth: the model is pre-trained on CTC-based ASR task and MT task in the pre-training stage.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which component is the least impactful?
Context 3: Introduction
The concept of message passing over graphs has been around for many years BIBREF0, BIBREF1, as well as that of graph neural networks (GNNs) BIBREF2, BIBREF3  However, GNNs have only recently started to be closely investigated, following the advent of deep learning  Some notable examples include BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12  These approaches are known as spectral  Their similarity with message passing (MP) was observed by BIBREF9 and formalized by BIBREF13 and BIBREF14 
The MP framework is based on the core idea of recursive neighborhood aggregation  That is, at every iteration, the representation of each vertex is updated based on messages received from its neighbors  All spectral GNNs can be described in terms of the MP framework 
GNNs have been applied with great success to bioinformatics and social network data, for node classification, link prediction, and graph classification  However, a few studies only have focused on the application of the MP framework to representation learning on text  This paper proposes one such application  More precisely, we represent documents as word co-occurrence networks, and develop an expressive MP GNN tailored to document understanding, the Message Passing Attention network for Document understanding (MPAD)  We also propose several hierarchical variants of MPAD  Evaluation on 10 document classification datasets show that our architectures learn representations that are competitive with the state-of-the-art  Furthermore, ablation experiments shed light on the impact of various architectural choices 
In what follows, we first provide some background about the MP framework (in sec  SECREF2), thoroughly describe and explain MPAD (sec  SECREF3), present our experimental framework (sec  SECREF4), report and interpret our results (sec  SECREF5), and provide a review of the relevant literature (sec  SECREF6) 
Message Passing Neural Networks
BIBREF13 proposed a MP framework under which many of the recently introduced GNNs can be reformulated  MP consists in an aggregation phase followed by a combination phase BIBREF14  More precisely, let $G(V,E)$ be a graph, and let us consider $v \in V$  At time $t+1$, a message vector $\mathbf {m}_v^{t+1}$ is computed from the representations of the neighbors $\mathcal {N}(v)$ of $v$:
The new representation $\mathbf {h}^{t+1}_v$ of $v$ is then computed by combining its current feature vector $\mathbf {h}^{t}_v$ with the message vector $\mathbf {m}_v^{t+1}$:
Messages are passed for $T$ time steps  Each step is implemented by a different layer of the MP network  Hence, iterations correspond to network depth  The final feature vector $\mathbf {h}_v^T$ of $v$ is based on messages propagated from all the nodes in the subtree of height $T$ rooted at $v$  It captures both the topology of the neighborhood of $v$ and the distribution of the vertex representations in it 
If a graph-level feature vector is needed, e g , for classification or regression, a READOUT pooling function, that must be invariant to permutations, is applied:
Next, we present the MP network we developed for document understanding 
Message Passing Attention network for Document understanding (MPAD) ::: Word co-occurrence networks
We represent a document as a statistical word co-occurrence network BIBREF18, BIBREF19 with a sliding window of size 2 overspanning sentences  Let us denote that graph $G(V,E)$  Each unique word in the preprocessed document is represented by a node in $G$, and an edge is added between two nodes if they are found together in at least one instantiation of the window  $G$ is directed and weighted: edge directions and weights respectively capture text flow and co-occurrence counts 
$G$ is a compact representation of its document  In $G$, immediate neighbors are consecutive words in the same sentence  That is, paths of length 2 correspond to bigrams  Paths of length more than 2 can correspond either to traditional $n$-grams or to relaxed $n$-grams, that is, words that never appear in the same sentence but co-occur with the same word(s)  Such nodes are linked through common neighbors 
Master node  Inspired by BIBREF3, our $G$ also includes a special document node, linked to all other nodes via unit weight bi-directional edges  In what follows, let us denote by $n$ the number of nodes in $G$, including the master node 
Message Passing Attention network for Document understanding (MPAD) ::: Message passing
We formulate our AGGREGATE function as:
where $\mathbf {H}^t \in \mathbb {R}^{n \times d}$ contains node features ($d$ is a hyperparameter), and $\mathbf {A} \in \mathbb {R}^{n \times n}$ is the adjacency matrix of $G$  Since $G$ is directed, $\mathbf {A}$ is asymmetric  Also, $\mathbf {A}$ has zero diagonal as we choose not to consider the feature of the node itself, only that of its incoming neighbors, when updating its representation  Since $G$ is weighted, the $i^{th}$ row of $A$ contains the weights of the edges incoming on node $v_i$  $\mathbf {D} \in \mathbb {R}^{n \times n}$ is the diagonal in-degree matrix of $G$  MLP denotes a multi-layer perceptron, and $\mathbf {M}^{t+1} \in \mathbb {R}^{n \times d}$ is the message matrix 
The use of a MLP was motivated by the observation that for graph classification, MP neural nets with 1-layer perceptrons are inferior to their MLP counterparts BIBREF14  Indeed, 1-layer perceptrons are not universal approximators of multiset functions  Note that like in BIBREF14, we use a different MLP at each layer 
Renormalization  The rows of $\mathbf {D}^{-1}\mathbf {A}$ sum to 1  This is equivalent to the renormalization trick of BIBREF9, but using only the in-degrees  That is, instead of computing a weighted sum of the incoming neighbors' feature vectors, we compute a weighted average of them  The coefficients are proportional to the strength of co-occurrence between words  One should note that by averaging, we lose the ability to distinguish between different neighborhood structures in some special cases, that is, we lose injectivity  Such cases include neighborhoods in which all nodes have the same representations, and neighborhoods of different sizes containing various representations in equal proportions BIBREF14  As suggested by the results of an ablation experiment, averaging is better than summing in our application (see subsection SECREF30)  Note that instead of simply summing/averaging, we also tried using GAT-like attention BIBREF11 in early experiments, without obtaining better results 
As far as our COMBINE function, we use the Gated Recurrent Unit BIBREF20, BIBREF21:
Omitting biases for readability, we have:
where the $\mathbf {W}$ and $\mathbf {U}$ matrices are trainable weight matrices not shared across time steps, $\sigma (\mathbf {x}) = 1/(1+\exp (-\mathbf {x}))$ is the sigmoid function, and $\mathbf {R}$ and $\mathbf {Z}$ are the parameters of the reset and update gates  The reset gate controls the amount of information from the previous time step (in $\mathbf {H}^t$) that should propagate to the candidate representations, $\tilde{\mathbf {H}}^{t+1}$  The new representations $\mathbf {H}^{t+1}$ are finally obtained by linearly interpolating between the previous and the candidate ones, using the coefficients returned by the update gate 
Interpretation  Updating node representations through a GRU should in principle allow nodes to encode a combination of local and global signals (low and high values of $t$, resp ), by allowing them to remember about past iterations  In addition, we also explicitly consider node representations at all iterations when reading out (see Eq  DISPLAY_FORM18) 
Message Passing Attention network for Document understanding (MPAD) ::: Readout
After passing messages and performing updates for $T$ iterations, we obtain a matrix $\mathbf {H}^T \in \mathbb {R}^{n \times d}$ containing the final vertex representations  Let $\hat{G}$ be graph $G$ without the special document node, and matrix $\mathbf {\hat{H}}^T \in \mathbb {R}^{(n-1) \times d}$ be the corresponding representation matrix (i e , $\mathbf {H}^T$ without the row of the document node) 
We use as our READOUT function the concatenation of self-attention applied to $\mathbf {\hat{H}}^T$ with the final document node representation  More precisely, we apply a global self-attention mechanism BIBREF22 to the rows of $\mathbf {\hat{H}}^T$  As shown in Eq  DISPLAY_FORM17, $\mathbf {\hat{H}}^T$ is first passed to a dense layer parameterized by matrix $\mathbf {W}_A^T \in \mathbb {R}^{d \times d}$  An alignment vector $\mathbf {a}$ is then derived by comparing, via dot products, the rows of the output of the dense layer $\mathbf {Y}^T \in \mathbb {R}^{(n-1) \times d}$ with a trainable vector $\mathbf {v}^T \in \mathbb {R}^d$ (initialized randomly) and normalizing with a softmax  The normalized alignment coefficients are finally used to compute the attentional vector $\mathbf {u}^T \in \mathbb {R}^d$ as a weighted sum of the final representations $\mathbf {\hat{H}}^T$ 
Note that we tried with multiple context vectors, i e , with a matrix $\mathbf {V}^T$ instead of a vector $\mathbf {v}^T$, like in BIBREF22, but results were not convincing, even when adding a regularization term to the loss to favor diversity among the rows of $\mathbf {V}^T$ 
Master node skip connection  $\mathbf {h}_G^T \in \mathbb {R}^{2d}$ is obtained by concatenating $\mathbf {u}^T$ and the final master node representation  That is, the master node vector bypasses the attention mechanism  This is equivalent to a skip or shortcut connection BIBREF23  The reason behind this choice is that we expect the special document node to learn a high-level summary about the document, such as its size, vocabulary, etc  (more details are given in subsection SECREF30)  Therefore, by making the master node bypass the attention layer, we directly inject global information about the document into its final representation 
Multi-readout  BIBREF14, inspired by Jumping Knowledge Networks BIBREF12, recommend to not only use the final representations when performing readout, but also that of the earlier steps  Indeed, as one iterates, node features capture more and more global information  However, retaining more local, intermediary information might be useful too  Thus, instead of applying the readout function only to $t=T$, we apply it to all time steps and concatenate the results, finally obtaining $\mathbf {h}_G \in \mathbb {R}^{T \times 2d}$ :
In effect, with this modification, we take into account features based on information aggregated from subtrees of different heights (from 1 to $T$), corresponding to local and global features 
Message Passing Attention network for Document understanding (MPAD) ::: Hierarchical variants of MPAD
Through the successive MP iterations, it could be argued that MPAD implicitly captures some soft notion of the hierarchical structure of documents (words $\rightarrow $ bigrams $\rightarrow $ compositions of bigrams, etc )  However, it might be beneficial to explicitly capture document hierarchy  Hierarchical architectures have brought significant improvements to many NLP tasks, such as language modeling and generation BIBREF24, BIBREF25, sentiment and topic classification BIBREF26, BIBREF27, and spoken language understanding BIBREF28, BIBREF29  Inspired by this line of research, we propose several hierarchical variants of MPAD, detailed in what follows  In all of them, we represent each sentence in the document as a word co-occurrence network, and obtain an embedding for it by applying MPAD as previously described 
MPAD-sentence-att  Here, the sentence embeddings are simply combined through self-attention 
MPAD-clique  In this variant, we build a complete graph where each node represents a sentence  We then feed that graph to MPAD, where the feature vectors of the nodes are initialized with the sentence embeddings previously obtained 
MPAD-path  This variant is similar to the clique one, except that instead of a complete graph, we build a path according to the natural flow of the text  That is, two nodes are linked by a directed edge if the two sentences they represent follow each other in the document 
Experiments ::: Datasets
We evaluate the quality of the document embeddings learned by MPAD on 10 document classification datasets, covering the topic identification, coarse and fine sentiment analysis and opinion mining, and subjectivity detection tasks  We briefly introduce the datasets next  Their statistics are reported in Table TABREF21 
(1) Reuters  This dataset contains stories collected from the Reuters news agency in 1987  Following common practice, we used the ModApte split and considered only the 10 classes with the highest number of positive training examples  We also removed documents belonging to more than one class and then classes left with no document (2 classes) 
(2) BBCSport BIBREF30 contains documents from the BBC Sport website corresponding to 2004-2005 sports news articles 
(3) Polarity BIBREF31 features positive and negative labeled snippets from Rotten Tomatoes 
(4) Subjectivity BIBREF32 contains movie review snippets from Rotten Tomatoes (subjective sentences), and Internet Movie Database plot summaries (objective sentences) 
(5) MPQA BIBREF33 is made of positive and negative phrases, annotated as part of the summer 2002 NRRC Workshop on Multi-Perspective Question Answering 
(6) IMDB BIBREF34 is a collection of highly polarized movie reviews from IMDB (positive and negative)  There are at most 30 reviews for each movie 
(7) TREC BIBREF35 consists of questions that are classified into 6 different categories 
(8) SST-1 BIBREF36 contains the same snippets as Polarity  The authors used the Stanford Parser to parse the snippets and split them into multiple sentences  They then used Amazon Mechanical Turk to annotate the resulting phrases according to their polarity (very negative, negative, neutral, positive, very positive) 
(9) SST-2 BIBREF36 is the same as SST-1 but with neutral reviews removed and snippets classified as positive or negative 
(10) Yelp2013 BIBREF26 features reviews obtained from the 2013 Yelp Dataset Challenge 
Experiments ::: Baselines
We evaluate MPAD against multiple state-of-the-art baseline models, including hierarchical ones, to enable fair comparison with the hierarchical MPAD variants 
doc2vec BIBREF37  Doc2vec (or paragraph vector) is an extension of word2vec that learns vectors for documents in a fully unsupervised manner  Document embeddings are then fed to a logistic regression classifier 
CNN BIBREF38  The convolutional neural network architecture, well-known in computer vision, is applied to text  There is one spatial dimension and the word embeddings are used as channels (depth dimensions) 
DAN BIBREF39  The Deep Averaging Network passes the unweighted average of the embeddings of the input words through multiple dense layers and a final softmax 
Tree-LSTM BIBREF40 is a generalization of the standard LSTM architecture to constituency and dependency parse trees 
DRNN BIBREF41  Recursive neural networks are stacked and applied to parse trees 
LSTMN BIBREF42 is an extension of the LSTM model where the memory cell is replaced by a memory network which stores word representations 
C-LSTM BIBREF43 combines convolutional and recurrent neural networks  The region embeddings provided by a CNN are fed to a LSTM 
SPGK BIBREF44 also models documents as word co-occurrence networks  It computes a graph kernel that compares shortest paths extracted from the word co-occurrence networks and then uses a SVM to categorize documents 
WMD BIBREF45 is an application of the well-known Earth Mover's Distance to text  A k-nearest neighbor classifier is used 
S-WMD BIBREF46 is a supervised extension of the Word Mover's Distance 
Semantic-CNN BIBREF47  Here, a CNN is applied to semantic units obtained by clustering words in the embedding space 
LSTM-GRNN BIBREF26 is a hierarchical model where sentence embeddings are obtained with a CNN and a GRU-RNN is fed the sentence representations to obtain a document vector 
HN-ATT BIBREF27 is another hierarchical model, where the same encoder architecture (a bidirectional GRU-RNN) is used for both sentences and documents, with different parameters  A self-attention mechanism is applied to the RNN annotations at each level 
Experiments ::: Model configuration and training
We preprocess all datasets using the code of BIBREF38  On Yelp2013, we also replace all tokens appearing strictly less than 6 times with a special UNK token, like in BIBREF27  We then build a directed word co-occurrence network from each document, with a window of size 2 
We use two MP iterations ($T$=2) for the basic MPAD, and two MP iterations at each level, for the hierarchical variants  We set $d$ to 64, except on IMDB and Yelp on which $d=128$, and use a two-layer MLP  The final graph representations are passed through a softmax for classification  We train MPAD in an end-to-end fashion by minimizing the cross-entropy loss function with the Adam optimizer BIBREF48 and an initial learning rate of 0 001 
To regulate potential differences in magnitude, we apply batch normalization after concatenating the feature vector of the master node with the self-attentional vector, that is, after the skip connection (see subsection SECREF16)  To prevent overfitting, we use dropout BIBREF49 with a rate of 0 5  We select the best epoch, capped at 200, based on the validation accuracy  When cross-validation is used (see 3rd column of Table TABREF21), we construct a validation set by randomly sampling 10% of the training set of each fold 
On all datasets except Yelp2013, we use the publicly available 300-dimensional pre-trained Google News vectors ($D$=300) BIBREF50 to initialize the node representations $\mathbf {H}^0$  On Yelp2013, we follow BIBREF27 and learn our own word vectors from the training and validation sets with the gensim implementation of word2vec BIBREF51 
MPAD was implemented in Python 3 6 using the PyTorch library BIBREF52  All experiments were run on a single machine consisting of a 3 4 GHz Intel Core i7 CPU with 16 GB of RAM and an NVidia GeForce Titan Xp GPU 
Results and ablations ::: Results
Experimental results are shown in Table TABREF28  For the baselines, the best scores reported in each original paper are shown  MPAD reaches best performance on 7 out of 10 datasets, and is close second elsewhere  Moreover, the 7 datasets on which MPAD ranks first widely differ in training set size, number of categories, and prediction task (topic, sentiment, subjectivity), which indicates that MPAD can perform well in different settings 
MPAD vs  hierarchical variants  On 9 datasets out of 10, one or more of the hierarchical variants outperform the vanilla MPAD architecture, highlighting the benefit of explicitly modeling the hierarchical nature of documents 
However, on Subjectivity, standard MPAD outperforms all hierarchical variants  On TREC, it reaches the same accuracy  We hypothesize that in some cases, using a different graph to separately encode each sentence might be worse than using one single graph to directly encode the document  Indeed, in the single document graph, some words that never appear in the same sentence can be connected through common neighbors, as was explained in subsection SECREF7  So, this way, some notion of cross-sentence context is captured while learning representations of words, bigrams, etc  at each MP iteration  This creates better informed representations, resulting in a better document embedding  With the hierarchical variants, on the other hand, each sentence vector is produced in isolation, without any contextual information about the other sentences in the document  Therefore, the final sentence embeddings might be of lower quality, and as a group might also contain redundant/repeated information  When the sentence vectors are finally combined into a document representation, it is too late to take context into account 
Results and ablations ::: Ablation studies
To understand the impact of some hyperparameters on performance, we conducted additional experiments on the Reuters, Polarity, and IMDB datasets, with the non-hierarchical version of MPAD  Results are shown in Table TABREF29 
Number of MP iterations  First, we varied the number of message passing iterations from 1 to 4  We can clearly see in Table TABREF29 that having more iterations improves performance  We attribute this to the fact that we are reading out at each iteration from 1 to $T$ (see Eq  DISPLAY_FORM18), which enables the final graph representation to encode a mixture of low-level and high-level features  Indeed, in initial experiments involving readout at $t$=$T$ only, setting $T\ge 2$ was always decreasing performance, despite the GRU-based updates (Eq  DISPLAY_FORM14)  These results were consistent with that of BIBREF53 and BIBREF9, who both are reading out only at $t$=$T$ too  We hypothesize that node features at $T\ge 2$ are too diffuse to be entirely relied upon during readout  More precisely, initially at $t$=0, node representations capture information about words, at $t$=1, about their 1-hop neighborhood (bigrams), at $t$=2, about compositions of bigrams, etc  Thus, pretty quickly, node features become general and diffuse  In such cases, considering also the lower-level, more precise features of the earlier iterations when reading out may be necessary 
Undirected edges  On Reuters, using an undirected graph leads to better performance, while on Polarity and IMDB, it is the opposite  This can be explained by the fact that Reuters is a topic classification task, for which the presence or absence of some patterns is important, but not necessarily the order in which they appear, while Polarity and IMDB are sentiment analysis tasks  To capture sentiment, modeling word order is crucial, e g , in detecting negation 
No master node  Removing the master node deteriorates performance across all datasets, clearly showing the value of having such a node  We hypothesize that since the special document node is connected to all other nodes, it is able to encode during message passing a summary of the document 
No renormalization  Here, we do not use the renormalization trick of BIBREF9 during MP (see subsection SECREF10)  That is, Eq  DISPLAY_FORM11 becomes $\mathbf {M}^{t+1} = \textsc {MLP}^{t+1}\big (\mathbf {A}\mathbf {H}^{t}\big )$  In other words, instead of computing a weighted average of the incoming neighbors' feature vectors, we compute a weighted sum of them  Unlike the mean, which captures distributions, the sum captures structural information BIBREF14  As shown in Table TABREF29, using sum instead of mean decreases performance everywhere, suggesting that in our application, capturing the distribution of neighbor representations is more important that capturing their structure  We hypothesize that this is the case because statistical word co-occurrence networks tend to have similar structural properties, regardless of the topic, polarity, sentiment, etc  of the corresponding documents 
Neighbors-only  In this experiment, we replaced the GRU combine function (see Eq  DISPLAY_FORM14) with the identity function  That is, we simply have $\mathbf {H}^{t+1}$=$\mathbf {M}^{t+1}$  Since $\mathbf {A}$ has zero diagonal, by doing so, we completely ignore the previous feature of the node itself when updating its representation  That is, the update is based entirely on its neighbors  Except on Reuters (almost no change), performance always suffers, stressing the need to take into account the root node during updates, not only its neighborhood 
Related work
In what follows, we offer a brief review of relevant studies, ranked by increasing order of similarity with our work 
BIBREF9, BIBREF54, BIBREF11, BIBREF10 conduct some node classification experiments on citation networks, where nodes are scientific papers, i e , textual data  However, text is only used to derive node feature vectors  The external graph structure, which plays a central role in determining node labels, is completely unrelated to text 
On the other hand, BIBREF55, BIBREF7 experiment on traditional document classification tasks  They both build $k$-nearest neighbor similarity graphs based on the Gaussian diffusion kernel  More precisely, BIBREF55 build one single graph where nodes are documents and distance is computed in the BoW space  Node features are then used for classification  Closer to our work, BIBREF7 represent each document as a graph  All document graphs are derived from the same underlying structure  Only node features, corresponding to the entries of the documents' BoW vectors, vary  The underlying, shared structure is that of a $k$-NN graph where nodes are vocabulary terms and similarity is the cosine of the word embedding vectors  BIBREF7 then perform graph classification  However they found performance to be lower than that of a naive Bayes classifier 
BIBREF56 use a GNN for hierarchical classification into a large taxonomy of topics  This task differs from traditional document classification  The authors represent documents as unweighted, undirected word co-occurrence networks with word embeddings as node features  They then use the spatial GNN of BIBREF15 to perform graph classification 
The work closest to ours is probably that of BIBREF53  The authors adopt the semi-supervised node classification approach of BIBREF9  They build one single undirected graph from the entire dataset, with both word and document nodes  Document-word edges are weighted by TF-IDF and word-word edges are weighted by pointwise mutual information derived from co-occurrence within a sliding window  There are no document-document edges  The GNN is trained based on the cross-entropy loss computed only for the labeled nodes, that is, the documents in the training set  When the final node representations are obtained, one can use that of the test documents to classify them and evaluate prediction performance 
There are significant differences between BIBREF53 and our work  First, our approach is inductive, not transductive  Indeed, while the node classification approach of BIBREF53 requires all test documents at training time, our graph classification model is able to perform inference on new, never-seen documents  The downside of representing documents as separate graphs, however, is that we lose the ability to capture corpus-level dependencies  Also, our directed graphs capture word ordering, which is ignored by BIBREF53  Finally, the approach of BIBREF53 requires computing the PMI for every word pair in the vocabulary, which may be prohibitive on datasets with very large vocabularies  On the other hand, the complexity of MPAD does not depend on vocabulary size 
Conclusion
We have proposed an application of the message passing framework to NLP, the Message Passing Attention network for Document understanding (MPAD)  Experiments conducted on 10 standard text classification datasets show that our architecture is competitive with the state-of-the-art  By processing weighted, directed word co-occurrence networks, MPAD is sensitive to word order and word-word relationship strength  To explicitly capture the hierarchical structure of documents, we also propose three hierarchical variants of MPAD, that we show bring improvements over the vanilla architecture 
Acknowledgments
We thank the NVidia corporation for the donation of a GPU as part of their GPU grant program 

ground_truth: Based on table results provided changing directed to undirected edges had least impact - max abs difference of 0.33 points on all three datasets.