prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What models are used for painting embedding and what for language style transfer?
Context 1: Introduction
Neural networks have been successfully used to describe images with text using sequence-to-sequence models BIBREF0  However, the results are simple and dry captions which are one or two phrases long  Humans looking at a painting see more than just objects  Paintings stimulate sentiments, metaphors and stories as well  Therefore, our goal is to have a neural network describe the painting artistically in a style of choice  As a proof of concept, we present a model which generates Shakespearean prose for a given painting as shown in Figure FIGREF1  Accomplishing this task is difficult with traditional sequence to sequence models since there does not exist a large collection of Shakespearean prose which describes paintings: Shakespeare's works describes a single painting shown in Figure FIGREF3  Fortunately we have a dataset of modern English poems which describe images BIBREF1 and line-by-line modern paraphrases of Shakespeare's plays BIBREF2  Our solution is therefore to combine two separately trained models to synthesize Shakespearean prose for a given painting 
Introduction ::: Related work
A general end-to-end approach to sequence learning BIBREF3 makes minimal assumptions on the sequence structure  This model is widely used in tasks such as machine translation, text summarization, conversational modeling, and image captioning  A generative model using a deep recurrent architecture BIBREF0 has also beeen used for generating phrases describing an image  The task of synthesizing multiple lines of poetry for a given image BIBREF1 is accomplished by extracting poetic clues from images  Given the context image, the network associates image attributes with poetic descriptions using a convolutional neural net  The poem is generated using a recurrent neural net which is trained using multi-adversarial training via policy gradient 
Transforming text from modern English to Shakespearean English using text "style transfer" is challenging  An end to end approach using a sequence-to-sequence model over a parallel text corpus BIBREF2 has been proposed based on machine translation  In the absence of a parallel text corpus, generative adversarial networks (GANs) have been used, which simultaneously train two models: a generative model which captures the data distribution, and a discriminative model which evaluates the performance of the generator  Using a target domain language model as a discriminator has also been employed BIBREF4, providing richer and more stable token-level feedback during the learning process  A key challenge in both image and text style transfer is separating content from style BIBREF5, BIBREF6, BIBREF7  Cross-aligned auto-encoder models have focused on style transfer using non-parallel text BIBREF7  Recently, a fine grained model for text style transfer has been proposed BIBREF8 which controls several factors of variation in textual data by using back-translation  This allows control over multiple attributes, such as gender and sentiment, and fine-grained control over the trade-off between content and style 
Methods
We use a total three datasets: two datasets for generating an English poem from an image, and Shakespeare plays and their English translations for text style transfer 
We train a model for generating poems from images based on two datasets BIBREF1  The first dataset consists of image and poem pairs, namely a multi-modal poem dataset (MultiM-Poem), and the second dataset is a large poem corpus, namely a uni-modal poem dataset (UniM-Poem)  The image and poem pairs are extended by adding the nearest three neighbor poems from the poem corpus without redundancy, and an extended image and poem pair dataset is constructed and denoted as MultiM-Poem(Ex)BIBREF1 
We use a collection of line-by-line modern paraphrases for 16 of Shakespeare’s plays BIBREF2, for training a style transfer network from English poems to Shakespearean prose  We use 18,395 sentences from the training data split  We keep 1,218 sentences in the validation data set and 1,462 sentences in our test set 
Methods ::: Image To Poem Actor-Critic Model
For generating a poem from images we use an existing actor-critic architecture BIBREF1  This involves 3 parallel CNNs: an object CNN, sentiment CNN, and scene CNN, for feature extraction  These features are combined with a skip-thought model which provides poetic clues, which are then fed into a sequence-to-sequence model trained by policy gradient with 2 discriminator networks for rewards  This as a whole forms a pipeline that takes in an image and outputs a poem as shown on the top left of Figure FIGREF4  A CNN-RNN generative model acts as an agent  The parameters of this agent define a policy whose execution determines which word is selected as an action  When the agent selects all words in a poem, it receives a reward  Two discriminative networks, shown on the top right of Figure FIGREF4, are defined to serve as rewards concerning whether the generated poem properly describes the input image and whether the generated poem is poetic  The goal of the poem generation model is to generate a sequence of words as a poem for an image to maximize the expected return 
Methods ::: Shakespearizing Poetic Captions
For Shakespearizing modern English texts, we experimented with various types of sequence to sequence models  Since the size of the parallel translation data available is small, we leverage a dictionary providing a mapping between Shakespearean words and modern English words to retrofit pre-trained word embeddings  Incorporating this extra information improves the translation task  The large number of shared word types between the source and target sentences indicates that sharing the representation between them is beneficial 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with Attention
We use a sequence-to-sequence model which consists of a single layer unidrectional LSTM encoder and a single layer LSTM decoder and pre-trained retrofitted word embeddings shared between source and target sentences  We experimented with two different types of attention: global attention BIBREF9, in which the model makes use of the output from the encoder and decoder for the current time step only, and Bahdanau attention BIBREF10, where computing attention requires the output of the decoder from the prior time step  We found that global attention performs better in practice for our task of text style transfer 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with a Pointer Network
Since a pair of corresponding Shakespeare and modern English sentences have significant vocabulary overlap we extend the sequence-to-sequence model mentioned above using pointer networks BIBREF11 that provide location based attention and have been used to enable copying of tokens directly from the input  Moreover, there are lot of proper nouns and rare words which might not be predicted by a vanilla sequence to sequence model 
Methods ::: Shakespearizing Poetic Captions ::: Prediction
For both seq2seq models, we use the attention matrices returned at each decoder time step during inference, to compute the next word in the translated sequence if the decoder output at the current time step is the UNK token  We replace the UNKs in the target output with the highest aligned, maximum attention, source word  The seq2seq model with global attention gives the best results with an average target BLEU score of 29 65 on the style transfer dataset, compared with an average target BLEU score of 26 97 using the seq2seq model with pointer networks 
Results
We perform a qualitative analysis of the Shakespearean prose generated for the input paintings  We conducted a survey, in which we presented famous paintings including those shown in Figures FIGREF1 and FIGREF10 and the corresponding Shakespearean prose generated by the model, and asked 32 students to rate them on the basis of content, creativity and similarity to Shakespearean style on a Likert scale of 1-5  Figure FIGREF12 shows the result of our human evaluation 
The average content score across the paintings is 3 7 which demonstrates that the prose generated is relevant to the painting  The average creativity score is 3 9 which demonstrates that the model captures more than basic objects in the painting successfully using poetic clues in the scene  The average style score is 3 9 which demonstrates that the prose generated is perceived to be in the style of Shakespeare 
We also perform a quantitative analysis of style transfer by generating BLEU scores for the model output using the style transfer dataset  The variation of the BLEU scores with the source sentence lengths is shown in Figure FIGREF11  As expected, the BLEU scores decrease with increase in source sentence lengths 
Results ::: Implementation
All models were trained on Google Colab with a single GPU using Python 3 6 and Tensorflow 2 0  The number of hidden units for the encoder and decoder is 1,576 and 256 for seq2seq with global attention and seq2seq with pointer networks respectively  Adam optimizer was used with the default learning rate of 0 001  The model was trained for 25 epochs  We use pre-trained retrofitted word embeddings of dimension 192 
Results ::: Limitations
Since we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer as shown in Figure FIGREF12 for "Starry Night" with a low average content score  This happens when the style transfer dataset does not have similar words in the training set of sentences  A solution would be to expand the style transfer dataset, for a better representation of the poem data 
Results ::: Conclusions and Future Work
In conclusion, combining two pipelines with an intermediate representation works well in practice  We observe that a CNN-RNN based image-to-poem net combined with a seq2seq model with parallel text corpus for text style transfer synthesizes Shakespeare-style prose for a given painting  For the seq2seq model used, we observe that it performs better in practice using global attention as compared with local attention  We make our models and code publicly available BIBREF12  In future work we would like to experiment with GANs in the absence of non-parallel datasets, so that we can use varied styles for text style transfer  We would also like to experiment with cross aligned auto-encoders, which form a latent content representation, to efficiently separate style and content 
Context 2: Introduction
One of the most fundamental topics in natural language processing is how best to derive high-level representations from constituent parts, as natural language meanings are a function of their constituent parts  How best to construct a sentence representation from distributed word embeddings is an example domain of this larger issue  Even though sequential neural models such as recurrent neural networks (RNN) BIBREF0 and their variants including Long Short-Term Memory (LSTM) BIBREF1 and Gated Recurrent Unit (GRU) BIBREF2 have become the de-facto standard for condensing sentence-level information from a sequence of words into a fixed vector, there have been many lines of research towards better sentence representation using other neural architectures, e g  convolutional neural networks (CNN) BIBREF3 or self-attention based models BIBREF4  
From a linguistic point of view, the underlying tree structure—as expressed by its constituency and dependency trees—of a sentence is an integral part of its meaning  Inspired by this fact, some recursive neural network (RvNN) models are designed to reflect the syntactic tree structure, achieving impressive results on several sentence-level tasks such as sentiment analysis BIBREF5 , BIBREF6 , machine translation BIBREF7 , natural language inference BIBREF8 , and discourse relation classification BIBREF9  
However, some recent works have BIBREF10 , BIBREF11 proposed latent tree models, which learn to construct task-specific tree structures without explicit supervision, bringing into question the value of linguistically-motivated recursive neural models  Witnessing the surprising performance of the latent tree models on some sentence-level tasks, there arises a natural question: Are linguistic tree structures the optimal way of composing sentence representations for NLP tasks?
In this paper, we demonstrate that linguistic priors are in fact useful for devising effective neural models for sentence representations, showing that our novel architecture based on constituency trees and their tag information obtains superior performance on several sentence-level tasks, including sentiment analysis and natural language inference 
A chief novelty of our approach is that we introduce a small separate tag-level tree-LSTM to control the composition function of the existing word-level tree-LSTM, which is in charge of extracting helpful syntactic signals for meaningful semantic composition of constituents by considering both the structures and linguistic tags of constituency trees simultaneously  In addition, we demonstrate that applying a typical LSTM to preprocess the leaf nodes of a tree-LSTM greatly improves the performance of the tree models  Moreover, we propose a clustered tag set to replace the existing tags on the assumption that the original syntactic tags are too fined-grained to be useful in neural models 
In short, our contributions in this work are as follows:
Related Work
Recursive neural networks (RvNN) are a kind of neural architecture which model sentences by exploiting syntactic structure  While earlier RvNN models proposed utilizing diverse composition functions, including feed-forward neural networks BIBREF12 , matrix-vector multiplication BIBREF5 , and tensor computation BIBREF6 , tree-LSTMs BIBREF13 remain the standard for several sentence-level tasks 
Even though classic RvNNs have demonstrated superior performance on a variety of tasks, their inflexibility, i e  their inability to handle dynamic compositionality for different syntactic configurations, is a considerable weakness  For instance, it would be desirable if our model could distinguish e g  adjective-noun composition from that of verb-noun or preposition-noun composition, as models failing to make such a distinction ignore real-world syntactic considerations such as `-arity' of function words (i e  types), and the adjunct/argument distinction 
To enable dynamic compositionality in recursive neural networks, many previous works BIBREF14 , BIBREF15 , BIBREF16 , BIBREF9 , BIBREF17 , BIBREF18 , BIBREF19 have proposed various methods 
One main direction of research leverages tag information, which is produced as a by-product of parsing  In detail, BIBREF16 ( BIBREF16 ) suggested TG-RNN, a model employing different composition functions according to POS tags, and TE-RNN/TE-RNTN, models which leverage tag embeddings as additional inputs for the existing tree-structured models  Despite the novelty of utilizing tag information, the explosion of the number of parameters (in case of the TG-RNN) and the limited performance of the original models (in case of the TE-RNN/TE-RNTN) have prevented these models from being widely adopted  Meanwhile, BIBREF9 ( BIBREF9 ) and BIBREF18 ( BIBREF18 ) proposed models based on a tree-LSTM which also uses the tag vectors to control the gate functions of the tree-LSTM  In spite of their impressive results, there is a limitation that the trained tag embeddings are too simple to reflect the rich information which tags provide in different syntactic structures  To alleviate this problem, we introduce structure-aware tag representations in the next section 
Another way of building dynamic compositionality into RvNNs is to take advantage of a meta-network (or hyper-network)  Inspired by recent works on dynamic parameter prediction, DC-TreeLSTMs BIBREF17 dynamically create the parameters for compositional functions in a tree-LSTM  Specifically, the model has two separate tree-LSTM networks whose architectures are similar, but the smaller of the two is utilized to calculate the weights of the bigger one  A possible problem for this model is that it may be easy to be trained such that the role of each tree-LSTM is ambiguous, as they share the same input, i e  word information  Therefore, we design two disentangled tree-LSTMs in our model so that one focuses on extracting useful features from only syntactic information while the other composes semantic units with the aid of the features  Furthermore, our model reduces the complexity of computation by utilizing typical tree-LSTM frameworks instead of computing the weights for each example 
Finally, some recent works BIBREF10 , BIBREF11 have proposed latent tree-structured models that learn how to formulate tree structures from only sequences of tokens, without the aid of syntactic trees or linguistic information  The latent tree models have the advantage of being able to find the optimized task-specific order of composition rather than a sequential or syntactic one  In experiments, we compare our model with not only syntactic tree-based models but also latent tree models, demonstrating that modeling with explicit linguistic knowledge can be an attractive option 
Model
In this section, we introduce a novel RvNN architecture, called SATA Tree-LSTM (Structure-Aware Tag Augmented Tree-LSTM)  This model is similar to typical Tree-LSTMs, but provides dynamic compositionality by augmenting a separate tag-level tree-LSTM which produces structure-aware tag representations for each node in a tree  In other words, our model has two independent tree-structured modules based on the same constituency tree, one of which (word-level tree-LSTM) is responsible for constructing sentence representations given a sequence of words as usual, while the other (tag-level tree-LSTM) provides supplementary syntactic information to the former 
In section 3 1, we first review tree-LSTM architectures  Then in section 3 2, we introduce a tag-level tree-LSTM and structure-aware tag representations  In section 3 3, we discuss an additional technique to boost the performance of tree-structured models, and in section 3 4, we describe the entire architecture of our model in detail 
Tree-LSTM
The LSTM BIBREF1 architecture was first introduced as an extension of the RNN architecture to mitigate the vanishing and exploding gradient problems  In addition, several works have discovered that applying the LSTM cell into tree structures can be an effective means of modeling sentence representations 
To be formal, the composition function of the cell in a tree-LSTM can be formulated as follows:
$$  \begin{bmatrix} \mathbf {i} \\ \mathbf {f}_l \\ \mathbf {f}_r \\ \mathbf {o} \\ \mathbf {g} \end{bmatrix} = \begin{bmatrix} \sigma \\ \sigma \\ \sigma \\ \sigma \\ \tanh \end{bmatrix} \Bigg ( \mathbf {W} \begin{bmatrix} \mathbf {h}_l \\ \mathbf {h}_r \\ \end{bmatrix} + \mathbf {b} \Bigg )$$   (Eq  8)
$$  \mathbf {c} = \mathbf {f}_l \odot \mathbf {c}_l + \mathbf {f}_r \odot \mathbf {c}_r + \mathbf {i} \odot \mathbf {g}\\$$   (Eq  9)
where $\mathbf {h}, \mathbf {c} \in \mathbb {R}^{d}$ indicate the hidden state and cell state of the LSTM cell, and $\mathbf {h}_l, \mathbf {h}_r, \mathbf {c}_l, \mathbf {c}_r \in \mathbb {R}^{d}$ the hidden states and cell states of a left and right child  $\mathbf {g} \in \mathbb {R}^{d}$ is the newly composed input for the cell and $\mathbf {i}, \mathbf {f}_{l}, \mathbf {f}_{r}, \mathbf {o} \in \mathbb {R}^{d}$ represent an input gate, two forget gates (left, right), and an output gate respectively  $\mathbf {W} \in \mathbb {R}^{5d\times 2d}$ and $\mathbf {b} \in \mathbb {R}^{5d}$ are trainable parameters  $\sigma $ corresponds to the sigmoid function, $\tanh $ to the hyperbolic tangent, and $\odot $ to element-wise multiplication 
Note the equations assume that there are only two children for each node, i e  binary or binarized trees, following the standard in the literature  While RvNN models can be constructed on any tree structure, in this work we only consider constituency trees as inputs 
In spite of the obvious upside that recursive models have in being so flexible, they are known for being difficult to fully utilize with batch computations as compared to other neural architectures because of the diversity of structure found across sentences  To alleviate this problem, BIBREF8 ( BIBREF8 ) proposed the SPINN model, which brings a shift-reduce algorithm to the tree-LSTM  As SPINN simplifies the process of constructing a tree into only two operations, i e  shift and reduce, it can support more effective parallel computations while enjoying the advantages of tree structures  For efficiency, our model also starts from our own SPINN re-implementation, whose function is exactly the same as that of the tree-LSTM 
Structure-aware Tag Representation
In most previous works using linguistic tag information BIBREF16 , BIBREF9 , BIBREF18 , tags are usually represented as simple low-dimensional dense vectors, similar to word embeddings  This approach seems reasonable in the case of POS tags that are attached to the corresponding words, but phrase-level constituent tags (e g  NP, VP, ADJP) vary greatly in size and shape, making them less amenable to uniform treatment  For instance, even the same phrase tags within different syntactic contexts can vary greatly in size and internal structure, as the case of NP tags in Figure 1 shows  Here, the NP consisting of DT[the]-NN[stories] has a different internal structure than the NP consisting of NP[the film 's]-NNS[shortcomings] 
One way of deriving structure-aware tag representations from the original tag embeddings is to introduce a separate tag-level tree-LSTM which accepts the typical tag embeddings at each node of a tree and outputs the computed structure-aware tag representations for the nodes  Note that the module concentrates on extracting useful syntactic features by considering only the tags and structures of the trees, excluding word information 
Formally, we denote a tag embedding for the tag attached to each node in a tree as $\textbf {e} \in \mathbb {R}^{d_\text{T}}$   Then, the function of each cell in the tag tree-LSTM is defined in the following way  Leaf nodes are defined by the following:
$$  \begin{bmatrix} \hat{\mathbf {c}} \\ \hat{\mathbf {h}} \\ \end{bmatrix} = \tanh {\left(\mathbf {U}_\text{T} \mathbf {e} + \mathbf {a}_\text{T}\right)}$$   (Eq  13)
while non-leaf nodes are defined by the following:
$$  \begin{bmatrix} \hat{\mathbf {i}} \\ \hat{\mathbf {f}}_l \\ \hat{\mathbf {f}}_r \\ \hat{\mathbf {o}} \\ \hat{\mathbf {g}} \end{bmatrix} = \begin{bmatrix} \sigma \\ \sigma \\ \sigma \\ \sigma \\ \tanh \end{bmatrix} \Bigg ( \mathbf {W_\text{T}} \begin{bmatrix} \hat{\mathbf {h}}_l \\ \hat{\mathbf {h}}_r \\ \mathbf {e} \\ \end{bmatrix} + \mathbf {b}_\text{T} \Bigg )$$   (Eq  14)
$$  \hat{\mathbf {c}} = \hat{\mathbf {f}}_l \odot \hat{\mathbf {c}}_l + \hat{\mathbf {f}}_r \odot \hat{\mathbf {c}}_r + \hat{\mathbf {i}} \odot \hat{\mathbf {g}}\\$$   (Eq  15)
where $\hat{\mathbf {h}}, \hat{\mathbf {c}} \in \mathbb {R}^{d_\text{T}}$ represent the hidden state and cell state of each node in the tag tree-LSTM  We regard the hidden state ( $\hat{\mathbf {h}}$ ) as a structure-aware tag representation for the node  $ \mathbf {U}_\text{T} \in \mathbb {R}^{2d_\text{T} \times d_\text{T}}, \textbf {a}_\text{T} \in \mathbb {R}^{2d_\text{T}}, \mathbf {W}_\text{T} \in \mathbb {R}^{5d_\text{T} \times 3d_\text{T}}$ , and $\mathbf {b}_\text{T} \in \mathbb {R}^{5d_\text{T}}$ are trainable parameters  The rest of the notation follows equations 8 , 9 , and 10   In case of leaf nodes, the states are computed by a simple non-linear transformation  Meanwhile, the composition function in a non-leaf node absorbs the tag embedding ( $\mathbf {e}$ ) as an additional input as well as the hidden states of the two children nodes  The benefit of revising tag representations according to the internal structure is that the derived embedding is a function of the corresponding makeup of the node, rather than a monolithic, categorical tag 
With regard to the tags themselves, we conjecture that the taxonomy of the tags currently in use in many NLP systems is too complex to be utilized effectively in deep neural models, considering the specificity of many tag sets and the limited amount of data with which to train  Thus, we cluster POS (word-level) tags into 12 groups following the universal POS tagset BIBREF20 and phrase-level tags into 11 groups according to criteria analogous to the case of words, resulting in 23 tag categories in total  In this work, we use the revised coarse-grained tags instead of the original ones  For more details, we refer readers to the supplemental materials 
Leaf-LSTM
An inherent shortcoming of RvNNs relative to sequential models is that each intermediate representation in a tree is unaware of its external context until all the information is gathered together at the root node  In other words, each composition process is prone to be locally optimized rather than globally optimized 
To mitigate this problem, we propose using a leaf-LSTM following the convention of some previous works BIBREF21 , BIBREF7 , BIBREF11 , which is a typical LSTM that accepts a sequence of words in order  Instead of leveraging word embeddings directly, we can use each hidden state and cell state of the leaf-LSTM as input tokens for leaf nodes in a tree-LSTM, anticipating the proper contextualization of the input sequence 
Formally, we denote a sequence of words in an input sentence as $w_{1:n}$ ( $n$ : the length of the sentence), and the corresponding word embeddings as $\mathbf {x}_{1:n}$   Then, the operation of the leaf-LSTM at time $t$ can be formulated as,
$$  \begin{bmatrix} \tilde{\mathbf {i}} \\ \tilde{\mathbf {f}} \\ \tilde{\mathbf {o}} \\ \tilde{\mathbf {g}} \end{bmatrix} = \begin{bmatrix} \sigma \\ \sigma \\ \sigma \\ \tanh \end{bmatrix} \Bigg ( \mathbf {W}_\text{L} \begin{bmatrix} \tilde{\mathbf {h}}_{t-1} \\ \mathbf {x}_t \\ \end{bmatrix} + \mathbf {b}_\text{L} \Bigg )$$   (Eq  18)
$$  \tilde{\mathbf {c}}_t = \tilde{\mathbf {f}} \odot \tilde{\mathbf {c}}_{t-1} + \tilde{\mathbf {i}} \odot \tilde{\mathbf {g}}\\$$   (Eq  19)
where $\mathbf {x}_t \in \mathbb {R}^{d_w}$ indicates an input word vector and $\tilde{\mathbf {h}}_t$ , $\tilde{\mathbf {c}}_t \in \mathbb {R}^{d_h}$ represent the hidden and cell state of the LSTM at time $t$ ( $\tilde{\mathbf {h}}_{t-1}$ corresponds to the hidden state at time $t$ -1)  $\mathbf {W}_\text{L}$ and $\mathbf {b}_\text{L} $ are learnable parameters  The remaining notation follows that of the tree-LSTM above 
In experiments, we demonstrate that introducing a leaf-LSTM fares better at processing the input words of a tree-LSTM compared to using a feed-forward neural network  We also explore the possibility of its bidirectional setting in ablation study 
SATA Tree-LSTM
In this section, we define SATA Tree-LSTM (Structure-Aware Tag Augmented Tree-LSTM, see Figure 2 ) which joins a tag-level tree-LSTM (section 3 2), a leaf-LSTM (section 3 3), and the original word tree-LSTM together 
As above we denote a sequence of words in an input sentence as $w_{1:n}$ and the corresponding word embeddings as $\mathbf {x}_{1:n}$ 

ground_truth: generating a poem from images we use an existing actor-critic architecture, various types of sequence to sequence models

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which baselines did they compare against?
Context 2:  In addition, a tag embedding for the tag attached to each node in a tree is denoted by $\textbf {e} \in \mathbb {R}^{d_\text{T}}$   Then, we derive the final sentence representation for the input sentence with our model in two steps 
First, we compute structure-aware tag representations ( $\hat{\mathbf {h}}$ ) for each node of a tree using the tag tree-LSTM (the right side of Figure 2 ) as follows:
$$  \begin{bmatrix} \hat{\mathbf {c}} \\ \hat{\mathbf {h}} \\ \end{bmatrix} = {\left\lbrace \begin{array}{ll} \text{Tag-Tree-LSTM}(\mathbf {e}) & \text{if a leaf node} \\ \text{Tag-Tree-LSTM}(\hat{\mathbf {h}}_l, \hat{\mathbf {h}}_r, \mathbf {e}) & \text{otherwise} \end{array}\right }$$   (Eq  23)
where Tag-Tree-LSTM indicates the module we described in section 3 2 
Second, we combine semantic units recursively on the word tree-LSTM in a bottom-up fashion  For leaf nodes, we leverage the Leaf-LSTM (the bottom-left of Figure 2 , explained in section 3 3) to compute $\tilde{\mathbf {c}}_{t}$ and $\tilde{\mathbf {h}}_{t}$ in sequential order, with the corresponding input $\mathbf {x}_t$  
$$  \begin{bmatrix} \tilde{\mathbf {c}}_{t} \\ \tilde{\mathbf {h}}_{t} \\ \end{bmatrix} = \text{Leaf-LSTM}(\tilde{\textbf {h}}_{t-1}, \textbf {x}_t)$$   (Eq  24)
Then, the $\tilde{\mathbf {c}}_{t}$ and $\tilde{\mathbf {h}}_{t}$ can be utilized as input tokens to the word tree-LSTM, with the left (right) child of the target node corresponding to the $t$ th word in the input sentence 
$$  \begin{bmatrix} \check{\textbf {c}}_{\lbrace l, r\rbrace } \\ \check{\textbf {h}}_{\lbrace l, r\rbrace } \end{bmatrix} = \begin{bmatrix} \tilde{\textbf {c}}_{t} \\ \tilde{\textbf {h}}_{t} \end{bmatrix}$$   (Eq  25)
In the non-leaf node case, we calculate phrase representations for each node in the word tree-LSTM (the upper-left of Figure 2 ) recursively as follows:
$$  \check{\mathbf {g}} = \tanh {\left( \mathbf {U}_\text{w} \begin{bmatrix} \check{\mathbf {h}}_l \\ \check{\mathbf {h}}_r \\ \end{bmatrix} + \mathbf {a}_\text{w} \right)}$$   (Eq  26)
$$  \begin{bmatrix} \check{\mathbf {i}} \\ \check{\mathbf {f}}_l \\ \check{\mathbf {f}}_r \\ \check{\mathbf {o}} \end{bmatrix} = \begin{bmatrix} \sigma \\ \sigma \\ \sigma \\ \sigma \end{bmatrix} \Bigg ( \mathbf {W_\text{w}} \begin{bmatrix} \check{\mathbf {h}}_l \\ \check{\mathbf {h}}_r \\ \hat{\mathbf {h}} \\ \end{bmatrix} + \mathbf {b}_\text{w} \Bigg )$$   (Eq  27)
where $\check{\mathbf {h}}$ , $\check{\mathbf {c}} \in \mathbb {R}^{d_h}$ represent the hidden and cell state of each node in the word tree-LSTM  $\mathbf {U}_\text{w} \in \mathbb {R}^{d_h \times 2d_h}$ , $\mathbf {W}_\text{w} \in \mathbb {R}^{4d_h \times \left(2d_h+d_\text{T}\right)}$ , $\mathbf {a}_\text{w} \in \mathbb {R}^{d_h}$ , $\mathbf {b}_\text{w} \in \mathbb {R}^{4d_h}$ are learned parameters  The remaining notation follows those of the previous sections  Note that the structure-aware tag representations ( $\hat{\mathbf {h}}$ ) are only utilized to control the gate functions of the word tree-LSTM in the form of additional inputs, and are not involved in the semantic composition ( $\check{\mathbf {g}}$ ) directly 
Finally, the hidden state of the root node ( $\check{\mathbf {h}}_\text{root}$ ) in the word-level tree-LSTM becomes the final sentence representation of the input sentence 
Quantitative Analysis
One of the most basic approaches to evaluate a sentence encoder is to measure the classification performance with the sentence representations made by the encoder  Thus, we conduct experiments on the following five datasets  (Summary statistics for the datasets are reported in the supplemental materials )
MR: A group of movie reviews with binary (positive / negative) classes  BIBREF22
SST-2: Stanford Sentiment Treebank BIBREF6   Similar to MR, but each review is provided in the form of a binary parse tree whose nodes are annotated with numeric sentiment values  For SST-2, we only consider binary (positive / negative) classes 
SST-5: Identical to SST-2, but the reviews are grouped into fine-grained (very negative, negative, neutral, positive, very positive) classes 
SUBJ: Sentences grouped as being either subjective or objective (binary classes)  BIBREF23
TREC: A dataset which groups questions into six different question types (classes)  BIBREF24
As a preprocessing step, we construct parse trees for the sentences in the datasets using the Stanford PCFG parser BIBREF25   Because syntactic tags are by-products of constituency parsing, we do not need further preprocessing 
To classify the sentence given our sentence representation ( $\check{\mathbf {h}}_\text{root}$ ), we use one fully-connected layer with a ReLU activation, followed by a softmax classifier  The final predicted probability distribution of the class $y$ given the sentence $w_{1:n}$ is defined as follows,
$$\mathbf {s} = \text{ReLU}(\mathbf {W}_\text{s} \check{\mathbf {h}}_\text{root}+ \mathbf {b}_\text{s})$$   (Eq  37)
$$p(y|w_{1:n}) = \text{softmax}(\mathbf {W}_\text{c}\mathbf {s} + \mathbf {b}_\text{c})$$   (Eq  38)
where $\textbf {s} \in \mathbb {R}^{d_\text{s}}$ is the computed task-specific sentence representation for the classifier, and $\textbf {W}_\text{s} \in \mathbb {R}^{d_\text{s} \times d_h}$ , $\textbf {W}_\text{c} \in \mathbb {R}^{d_\text{c} \times d_s}$ , $\textbf {b}_\text{s} \in \mathbb {R}^{d_s}$ , $\textbf {b}_\text{c} \in \mathbb {R}^{d_c}$ are trainable parameters  As an objective function, we use the cross entropy of the predicted and true class distributions 
The results of the experiments on the five datasets are shown in table 1   In this table, we report the test accuracy of our model and various other models on each dataset in terms of percentage  To consider the effects of random initialization, we report the best numbers obtained from each several runs with hyper-parameters fixed 
Compared with the previous syntactic tree-based models as well as other neural models, our SATA Tree-LSTM shows superior or competitive performance on all tasks  Specifically, our model achieves new state-of-the-art results within the tree-structured model class on 4 out of 5 sentence classification tasks—SST-2, SST-5, MR, and TREC  The model shows its strength, in particular, when the datasets provide phrase-level supervision to facilitate tree structure learning (i e  SST-2, SST-5)  Moreover, the numbers we report for SST-5 and TREC are competitive to the existing state-of-the-art results including ones from structurally pre-trained models such as ELMo BIBREF26 , proving our model's superiority  Note that the SATA Tree-LSTM also outperforms the recent latent tree-based model, indicating that modeling a neural model with explicit linguistic knowledge can be an attractive option 
On the other hand, a remaining concern is that our SATA Tree-LSTM is not robust to random seeds when the size of a dataset is relatively small, as tag embeddings are randomly initialized rather than relying on pre-trained ones in contrast with the case of words  From this observation, we could find out there needs a direction of research towards pre-trained tag embeddings 
To estimate the performance of our model beyond the tasks requiring only one sentence at a time, we conduct an experiment on the Stanford Natural Language Inference BIBREF34 dataset, each example of which consists of two sentences, the premise and the hypothesis  Our objective given the data is to predict the correct relationship between the two sentences among three options— contradiction, neutral, or entailment 
We use the siamese architecture to encode both the premise ( $p_{1:m}$ ) and hypothesis ( $h_{1:n}$ ) following the standard of sentence-encoding models in the literature  (Specifically, $p_{1:m}$ is encoded as $\check{\mathbf {h}}_\text{root}^p \in \mathbb {R}^{d_h}$ and $h_{1:n}$ is encoded as $\check{\mathbf {h}}_\text{root}^h \in \mathbb {R}^{d_h}$ with the same encoder ) Then, we leverage some heuristics BIBREF35 , followed by one fully-connected layer with a ReLU activation and a softmax classifier  Specifically,
$$\mathbf {z} = \left[ \check{\mathbf {h}}_\text{root}^p; \check{\mathbf {h}}_\text{root}^h; | \check{\mathbf {h}}_\text{root}^p - \check{\mathbf {h}}_\text{root}^h |; \check{\mathbf {h}}_\text{root}^p \odot \check{\mathbf {h}}_\text{root}^h \right]$$   (Eq  41)
$$\mathbf {s} = \text{ReLU}(\mathbf {W}_\text{s} \mathbf {z} + \mathbf {b}_\text{s})$$   (Eq  42)
where $\textbf {z} \in \mathbb {R}^{4d_h}$ , $\textbf {s} \in \mathbb {R}^{d_s}$ are intermediate features for the classifier and $\textbf {W}_\text{s} \in \mathbb {R}^{d_\text{s} \times 4d_h}$ , $\textbf {W}_\text{c} \in \mathbb {R}^{d_\text{c} \times d_s}$ , $\textbf {b}_\text{s} \in \mathbb {R}^{d_s}$ , $\textbf {b}_\text{c} \in \mathbb {R}^{d_c}$ are again trainable parameters 
Our experimental results on the SNLI dataset are shown in table 2   In this table, we report the test accuracy and number of trainable parameters for each model  Our SATA-LSTM again demonstrates its decent performance compared against the neural models built on both syntactic trees and latent trees, as well as the non-tree models  (Latent Syntax Tree-LSTM: BIBREF10 ( BIBREF10 ), Tree-based CNN: BIBREF35 ( BIBREF35 ), Gumbel Tree-LSTM: BIBREF11 ( BIBREF11 ), NSE: BIBREF36 ( BIBREF36 ), Reinforced Self-Attention Network: BIBREF4 ( BIBREF4 ), Residual stacked encoders: BIBREF37 ( BIBREF37 ), BiLSTM with generalized pooling: BIBREF38 ( BIBREF38 ) ) Note that the number of learned parameters in our model is also comparable to other sophisticated models, showing the efficiency of our model 
Even though our model has proven its mettle, the effect of tag information seems relatively weak in the case of SNLI, which contains a large amount of data compared to the others  One possible explanation is that neural models may learn some syntactic rules from large amounts of text when the text size is large enough, reducing the necessity of external linguistic knowledge  We leave the exploration of the effectiveness of tags relative to data size for future work 
Here we go over the settings common across our models during experimentation  For more task-specific details, refer to the supplemental materials 
For our input embeddings, we used 300 dimensional 840B GloVe BIBREF39 as pre-trained word embeddings, and tag representations were randomly sampled from the uniform distribution [-0 005, 0 005]  Tag vectors are revised during training while the fine-tuning of the word embedding depends on the task  Our models were trained using the Adam BIBREF40 or Adadelta BIBREF41 optimizer, depending on task  For regularization, weight decay is added to the loss function except for SNLI following BIBREF42 ( BIBREF42 ) and Dropout BIBREF43 is also applied for the word embeddings and task-specific classifiers  Moreover, batch normalization BIBREF44 is adopted for the classifiers  As a default, all the weights in the model are initialized following BIBREF45 ( BIBREF45 ) and the biases are set to 0  The total norm of the gradients of the parameters is clipped not to be over 5 during training 
Our best models for each dataset were chosen by validation accuracy in cases where a validation set was provided as a part of the dataset  Otherwise, we perform a grid search on probable hyper-parameter settings, or run 10-fold cross-validation in cases where even a test set does not exist 
Ablation Study
In this section, we design an ablation study on the core modules of our model to explore their effectiveness  The dataset used in this experiment is SST-2  To conduct the experiment, we only replace the target module with other candidates while maintaining the other settings  To be specific, we focus on two modules, the leaf-LSTM and structure-aware tag embeddings (tag-level tree-LSTM)  In the first case, the leaf-LSTM is replaced with a fully-connected layer with a $\tanh $ activation or Bi-LSTM  In the second case, we replace the structure-aware tag embeddings with naive tag embeddings or do not employ them at all 
The experimental results are depicted in Figure 3   As the chart shows, our model outperforms all the other options we have considered  In detail, the left part of the chart shows that the leaf-LSTM is the most effective option compared to its competitors  Note that the sequential leaf-LSTM is somewhat superior or competitive than the bidirectional leaf-LSTM when both have a comparable number of parameters  We conjecture this may because a backward LSTM does not add additional useful knowledge when the structure of a sentence is already known  In conclusion, we use the uni-directional LSTM as a leaf module because of its simplicity and remarkable performance 
Meanwhile, the right part of the figure demonstrates that our newly introduced structure-aware embeddings have a real impact on improving the model performance  Interestingly, employing the naive tag embeddings made no difference in terms of the test accuracy, even though the absolute validation accuracy increased (not reported in the figure)  This result supports our assumption that tag information should be considered in the structure 
Qualitative Analysis
In previous sections, we have numerically demonstrated that our model is effective in encouraging useful composition of semantic units  Here, we directly investigate the computed representations for each node of a tree, showing that the remarkable performance of our model is mainly due to the gradual and recursive composition of the intermediate representations on the syntactic structure 
To observe the phrase-level embeddings at a glance, we draw a scatter plot in which a point represents the corresponding intermediate representation  We utilize PCA (Principal Component Analysis) to project the representations into a two-dimensional vector space  As a target parse tree, we reuse the one seen in Figure 1   The result is shown in Figure 4  
From this figure, we confirm that the intermediate representations have a hierarchy in the semantic space, which is very similar to that of the parse tree  In other words, as many tree-structured models pursue, we can see the tendency of constructing the representations from the low-level (the bottom of the figure) to the high-level (the top-left and top-right of the figure), integrating the meaning of the constituents recursively  An interesting thing to note is that the final sentence representation is near that of the phrase `, the stories are quietly moving ' rather than that of `Despite the film's shortcomings', catching the main meaning of the sentence 
Conclusion
We have proposed a novel RvNN architecture to fully utilize linguistic priors  A newly introduced tag-level tree-LSTM demonstrates that it can effectively control the composition function of the corresponding word-level tree-LSTM  In addition, the proper contextualization of the input word vectors results in significant performance improvements on several sentence-level tasks  For future work, we plan to explore a new way of exploiting dependency trees effectively, similar to the case of constituency trees 
Acknowledgments
We thank anonymous reviewers for their constructive and fruitful comments  This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF2016M3C4A7952587) 
Context 3: Introduction
The concept of message passing over graphs has been around for many years BIBREF0, BIBREF1, as well as that of graph neural networks (GNNs) BIBREF2, BIBREF3  However, GNNs have only recently started to be closely investigated, following the advent of deep learning  Some notable examples include BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12  These approaches are known as spectral  Their similarity with message passing (MP) was observed by BIBREF9 and formalized by BIBREF13 and BIBREF14 
The MP framework is based on the core idea of recursive neighborhood aggregation  That is, at every iteration, the representation of each vertex is updated based on messages received from its neighbors  All spectral GNNs can be described in terms of the MP framework 
GNNs have been applied with great success to bioinformatics and social network data, for node classification, link prediction, and graph classification  However, a few studies only have focused on the application of the MP framework to representation learning on text  This paper proposes one such application  More precisely, we represent documents as word co-occurrence networks, and develop an expressive MP GNN tailored to document understanding, the Message Passing Attention network for Document understanding (MPAD)  We also propose several hierarchical variants of MPAD  Evaluation on 10 document classification datasets show that our architectures learn representations that are competitive with the state-of-the-art  Furthermore, ablation experiments shed light on the impact of various architectural choices 
In what follows, we first provide some background about the MP framework (in sec  SECREF2), thoroughly describe and explain MPAD (sec  SECREF3), present our experimental framework (sec  SECREF4), report and interpret our results (sec  SECREF5), and provide a review of the relevant literature (sec  SECREF6) 
Message Passing Neural Networks
BIBREF13 proposed a MP framework under which many of the recently introduced GNNs can be reformulated  MP consists in an aggregation phase followed by a combination phase BIBREF14  More precisely, let $G(V,E)$ be a graph, and let us consider $v \in V$  At time $t+1$, a message vector $\mathbf {m}_v^{t+1}$ is computed from the representations of the neighbors $\mathcal {N}(v)$ of $v$:
The new representation $\mathbf {h}^{t+1}_v$ of $v$ is then computed by combining its current feature vector $\mathbf {h}^{t}_v$ with the message vector $\mathbf {m}_v^{t+1}$:
Messages are passed for $T$ time steps  Each step is implemented by a different layer of the MP network  Hence, iterations correspond to network depth  The final feature vector $\mathbf {h}_v^T$ of $v$ is based on messages propagated from all the nodes in the subtree of height $T$ rooted at $v$  It captures both the topology of the neighborhood of $v$ and the distribution of the vertex representations in it 
If a graph-level feature vector is needed, e g , for classification or regression, a READOUT pooling function, that must be invariant to permutations, is applied:
Next, we present the MP network we developed for document understanding 
Message Passing Attention network for Document understanding (MPAD) ::: Word co-occurrence networks
We represent a document as a statistical word co-occurrence network BIBREF18, BIBREF19 with a sliding window of size 2 overspanning sentences  Let us denote that graph $G(V,E)$  Each unique word in the preprocessed document is represented by a node in $G$, and an edge is added between two nodes if they are found together in at least one instantiation of the window  $G$ is directed and weighted: edge directions and weights respectively capture text flow and co-occurrence counts 
$G$ is a compact representation of its document  In $G$, immediate neighbors are consecutive words in the same sentence  That is, paths of length 2 correspond to bigrams  Paths of length more than 2 can correspond either to traditional $n$-grams or to relaxed $n$-grams, that is, words that never appear in the same sentence but co-occur with the same word(s)  Such nodes are linked through common neighbors 
Master node  Inspired by BIBREF3, our $G$ also includes a special document node, linked to all other nodes via unit weight bi-directional edges  In what follows, let us denote by $n$ the number of nodes in $G$, including the master node 
Message Passing Attention network for Document understanding (MPAD) ::: Message passing
We formulate our AGGREGATE function as:
where $\mathbf {H}^t \in \mathbb {R}^{n \times d}$ contains node features ($d$ is a hyperparameter), and $\mathbf {A} \in \mathbb {R}^{n \times n}$ is the adjacency matrix of $G$  Since $G$ is directed, $\mathbf {A}$ is asymmetric  Also, $\mathbf {A}$ has zero diagonal as we choose not to consider the feature of the node itself, only that of its incoming neighbors, when updating its representation  Since $G$ is weighted, the $i^{th}$ row of $A$ contains the weights of the edges incoming on node $v_i$  $\mathbf {D} \in \mathbb {R}^{n \times n}$ is the diagonal in-degree matrix of $G$  MLP denotes a multi-layer perceptron, and $\mathbf {M}^{t+1} \in \mathbb {R}^{n \times d}$ is the message matrix 
The use of a MLP was motivated by the observation that for graph classification, MP neural nets with 1-layer perceptrons are inferior to their MLP counterparts BIBREF14  Indeed, 1-layer perceptrons are not universal approximators of multiset functions  Note that like in BIBREF14, we use a different MLP at each layer 
Renormalization  The rows of $\mathbf {D}^{-1}\mathbf {A}$ sum to 1  This is equivalent to the renormalization trick of BIBREF9, but using only the in-degrees  That is, instead of computing a weighted sum of the incoming neighbors' feature vectors, we compute a weighted average of them  The coefficients are proportional to the strength of co-occurrence between words  One should note that by averaging, we lose the ability to distinguish between different neighborhood structures in some special cases, that is, we lose injectivity  Such cases include neighborhoods in which all nodes have the same representations, and neighborhoods of different sizes containing various representations in equal proportions BIBREF14  As suggested by the results of an ablation experiment, averaging is better than summing in our application (see subsection SECREF30)  Note that instead of simply summing/averaging, we also tried using GAT-like attention BIBREF11 in early experiments, without obtaining better results 
As far as our COMBINE function, we use the Gated Recurrent Unit BIBREF20, BIBREF21:
Omitting biases for readability, we have:
where the $\mathbf {W}$ and $\mathbf {U}$ matrices are trainable weight matrices not shared across time steps, $\sigma (\mathbf {x}) = 1/(1+\exp (-\mathbf {x}))$ is the sigmoid function, and $\mathbf {R}$ and $\mathbf {Z}$ are the parameters of the reset and update gates  The reset gate controls the amount of information from the previous time step (in $\mathbf {H}^t$) that should propagate to the candidate representations, $\tilde{\mathbf {H}}^{t+1}$  The new representations $\mathbf {H}^{t+1}$ are finally obtained by linearly interpolating between the previous and the candidate ones, using the coefficients returned by the update gate 
Interpretation  Updating node representations through a GRU should in principle allow nodes to encode a combination of local and global signals (low and high values of $t$, resp ), by allowing them to remember about past iterations  In addition, we also explicitly consider node representations at all iterations when reading out (see Eq  DISPLAY_FORM18) 
Message Passing Attention network for Document understanding (MPAD) ::: Readout
After passing messages and performing updates for $T$ iterations, we obtain a matrix $\mathbf {H}^T \in \mathbb {R}^{n \times d}$ containing the final vertex representations  Let $\hat{G}$ be graph $G$ without the special document node, and matrix $\mathbf {\hat{H}}^T \in \mathbb {R}^{(n-1) \times d}$ be the corresponding representation matrix (i e , $\mathbf {H}^T$ without the row of the document node) 
We use as our READOUT function the concatenation of self-attention applied to $\mathbf {\hat{H}}^T$ with the final document node representation  More precisely, we apply a global self-attention mechanism BIBREF22 to the rows of $\mathbf {\hat{H}}^T$  As shown in Eq  DISPLAY_FORM17, $\mathbf {\hat{H}}^T$ is first passed to a dense layer parameterized by matrix $\mathbf {W}_A^T \in \mathbb {R}^{d \times d}$  An alignment vector $\mathbf {a}$ is then derived by comparing, via dot products, the rows of the output of the dense layer $\mathbf {Y}^T \in \mathbb {R}^{(n-1) \times d}$ with a trainable vector $\mathbf {v}^T \in \mathbb {R}^d$ (initialized randomly) and normalizing with a softmax  The normalized alignment coefficients are finally used to compute the attentional vector $\mathbf {u}^T \in \mathbb {R}^d$ as a weighted sum of the final representations $\mathbf {\hat{H}}^T$ 
Note that we tried with multiple context vectors, i e , with a matrix $\mathbf {V}^T$ instead of a vector $\mathbf {v}^T$, like in BIBREF22, but results were not convincing, even when adding a regularization term to the loss to favor diversity among the rows of $\mathbf {V}^T$ 
Master node skip connection  $\mathbf {h}_G^T \in \mathbb {R}^{2d}$ is obtained by concatenating $\mathbf {u}^T$ and the final master node representation  That is, the master node vector bypasses the attention mechanism  This is equivalent to a skip or shortcut connection BIBREF23  The reason behind this choice is that we expect the special document node to learn a high-level summary about the document, such as its size, vocabulary, etc  (more details are given in subsection SECREF30)  Therefore, by making the master node bypass the attention layer, we directly inject global information about the document into its final representation 
Multi-readout  BIBREF14, inspired by Jumping Knowledge Networks BIBREF12, recommend to not only use the final representations when performing readout, but also that of the earlier steps  Indeed, as one iterates, node features capture more and more global information  However, retaining more local, intermediary information might be useful too  Thus, instead of applying the readout function only to $t=T$, we apply it to all time steps and concatenate the results, finally obtaining $\mathbf {h}_G \in \mathbb {R}^{T \times 2d}$ :
In effect, with this modification, we take into account features based on information aggregated from subtrees of different heights (from 1 to $T$), corresponding to local and global features 
Message Passing Attention network for Document understanding (MPAD) ::: Hierarchical variants of MPAD
Through the successive MP iterations, it could be argued that MPAD implicitly captures some soft notion of the hierarchical structure of documents (words $\rightarrow $ bigrams $\rightarrow $ compositions of bigrams, etc )  However, it might be beneficial to explicitly capture document hierarchy  Hierarchical architectures have brought significant improvements to many NLP tasks, such as language modeling and generation BIBREF24, BIBREF25, sentiment and topic classification BIBREF26, BIBREF27, and spoken language understanding BIBREF28, BIBREF29  Inspired by this line of research, we propose several hierarchical variants of MPAD, detailed in what follows  In all of them, we represent each sentence in the document as a word co-occurrence network, and obtain an embedding for it by applying MPAD as previously described 
MPAD-sentence-att  Here, the sentence embeddings are simply combined through self-attention 
MPAD-clique  In this variant, we build a complete graph where each node represents a sentence  We then feed that graph to MPAD, where the feature vectors of the nodes are initialized with the sentence embeddings previously obtained 
MPAD-path  This variant is similar to the clique one, except that instead of a complete graph, we build a path according to the natural flow of the text  That is, two nodes are linked by a directed edge if the two sentences they represent follow each other in the document 
Experiments ::: Datasets
We evaluate the quality of the document embeddings learned by MPAD on 10 document classification datasets, covering the topic identification, coarse and fine sentiment analysis and opinion mining, and subjectivity detection tasks  We briefly introduce the datasets next  Their statistics are reported in Table TABREF21 
(1) Reuters  This dataset contains stories collected from the Reuters news agency in 1987  Following common practice, we used the ModApte split and considered only the 10 classes with the highest number of positive training examples  We also removed documents belonging to more than one class and then classes left with no document (2 classes) 
(2) BBCSport BIBREF30 contains documents from the BBC Sport website corresponding to 2004-2005 sports news articles 
(3) Polarity BIBREF31 features positive and negative labeled snippets from Rotten Tomatoes 
(4) Subjectivity BIBREF32 contains movie review snippets from Rotten Tomatoes (subjective sentences), and Internet Movie Database plot summaries (objective sentences) 
(5) MPQA BIBREF33 is made of positive and negative phrases, annotated as part of the summer 2002 NRRC Workshop on Multi-Perspective Question Answering 
(6) IMDB BIBREF34 is a collection of highly polarized movie reviews from IMDB (positive and negative)  There are at most 30 reviews for each movie 
(7) TREC BIBREF35 consists of questions that are classified into 6 different categories 
(8) SST-1 BIBREF36 contains the same snippets as Polarity  The authors used the Stanford Parser to parse the snippets and split them into multiple sentences  They then used Amazon Mechanical Turk to annotate the resulting phrases according to their polarity (very negative, negative, neutral, positive, very positive) 
(9) SST-2 BIBREF36 is the same as SST-1 but with neutral reviews removed and snippets classified as positive or negative 
(10) Yelp2013 BIBREF26 features reviews obtained from the 2013 Yelp Dataset Challenge 
Experiments ::: Baselines
We evaluate MPAD against multiple state-of-the-art baseline models, including hierarchical ones, to enable fair comparison with the hierarchical MPAD variants 
doc2vec BIBREF37  Doc2vec (or paragraph vector) is an extension of word2vec that learns vectors for documents in a fully unsupervised manner  Document embeddings are then fed to a logistic regression classifier 
CNN BIBREF38  The convolutional neural network architecture, well-known in computer vision, is applied to text  There is one spatial dimension and the word embeddings are used as channels (depth dimensions) 
DAN BIBREF39  The Deep Averaging Network passes the unweighted average of the embeddings of the input words through multiple dense layers and a final softmax 
Tree-LSTM BIBREF40 is a generalization of the standard LSTM architecture to constituency and dependency parse trees 
DRNN BIBREF41  Recursive neural networks are stacked and applied to parse trees 
LSTMN BIBREF42 is an extension of the LSTM model where the memory cell is replaced by a memory network which stores word representations 
C-LSTM BIBREF43 combines convolutional and recurrent neural networks  The region embeddings provided by a CNN are fed to a LSTM 
SPGK BIBREF44 also models documents as word co-occurrence networks  It computes a graph kernel that compares shortest paths extracted from the word co-occurrence networks and then uses a SVM to categorize documents 
WMD BIBREF45 is an application of the well-known Earth Mover's Distance to text  A k-nearest neighbor classifier is used 
S-WMD BIBREF46 is a supervised extension of the Word Mover's Distance 
Semantic-CNN BIBREF47  Here, a CNN is applied to semantic units obtained by clustering words in the embedding space 
LSTM-GRNN BIBREF26 is a hierarchical model where sentence embeddings are obtained with a CNN and a GRU-RNN is fed the sentence representations to obtain a document vector 
HN-ATT BIBREF27 is another hierarchical model, where the same encoder architecture (a bidirectional GRU-RNN) is used for both sentences and documents, with different parameters  A self-attention mechanism is applied to the RNN annotations at each level 
Experiments ::: Model configuration and training
We preprocess all datasets using the code of BIBREF38  On Yelp2013, we also replace all tokens appearing strictly less than 6 times with a special UNK token, like in BIBREF27  We then build a directed word co-occurrence network from each document, with a window of size 2 
We use two MP iterations ($T$=2) for the basic MPAD, and two MP iterations at each level, for the hierarchical variants  We set $d$ to 64, except on IMDB and Yelp on which $d=128$, and use a two-layer MLP  The final graph representations are passed through a softmax for classification  We train MPAD in an end-to-end fashion by minimizing the cross-entropy loss function with the Adam optimizer BIBREF48 and an initial learning rate of 0 001 
To regulate potential differences in magnitude, we apply batch normalization after concatenating the feature vector of the master node with the self-attentional vector, that is, after the skip connection (see subsection SECREF16)  To prevent overfitting, we use dropout BIBREF49 with a rate of 0 5  We select the best epoch, capped at 200, based on the validation accuracy  When cross-validation is used (see 3rd column of Table TABREF21), we construct a validation set by randomly sampling 10% of the training set of each fold 
On all datasets except Yelp2013, we use the publicly available 300-dimensional pre-trained Google News vectors ($D$=300) BIBREF50 to initialize the node representations $\mathbf {H}^0$  On Yelp2013, we follow BIBREF27 and learn our own word vectors from the training and validation sets with the gensim implementation of word2vec BIBREF51 
MPAD was implemented in Python 3 6 using the PyTorch library BIBREF52  All experiments were run on a single machine consisting of a 3 4 GHz Intel Core i7 CPU with 16 GB of RAM and an NVidia GeForce Titan Xp GPU 
Results and ablations ::: Results
Experimental results are shown in Table TABREF28  For the baselines, the best scores reported in each original paper are shown  MPAD reaches best performance on 7 out of 10 datasets, and is close second elsewhere  Moreover, the 7 datasets on which MPAD ranks first widely differ in training set size, number of categories, and prediction task (topic, sentiment, subjectivity), which indicates that MPAD can perform well in different settings 
MPAD vs  hierarchical variants  On 9 datasets out of 10, one or more of the hierarchical variants outperform the vanilla MPAD architecture, highlighting the benefit of explicitly modeling the hierarchical nature of documents 
However, on Subjectivity, standard MPAD outperforms all hierarchical variants  On TREC, it reaches the same accuracy  We hypothesize that in some cases, using a different graph to separately encode each sentence might be worse than using one single graph to directly encode the document  Indeed, in the single document graph, some words that never appear in the same sentence can be connected through common neighbors, as was explained in subsection SECREF7  So, this way, some notion of cross-sentence context is captured while learning representations of words, bigrams, etc  at each MP iteration  This creates better informed representations, resulting in a better document embedding  With the hierarchical variants, on the other hand, each sentence vector is produced in isolation, without any contextual information about the other sentences in the document  Therefore, the final sentence embeddings might be of lower quality, and as a group might also contain redundant/repeated information  When the sentence vectors are finally combined into a document representation, it is too late to take context into account 
Results and ablations ::: Ablation studies
To understand the impact of some hyperparameters on performance, we conducted additional experiments on the Reuters, Polarity, and IMDB datasets, with the non-hierarchical version of MPAD  Results are shown in Table TABREF29 
Number of MP iterations  First, we varied the number of message passing iterations from 1 to 4  We can clearly see in Table TABREF29 that having more iterations improves performance  We attribute this to the fact that we are reading out at each iteration from 1 to $T$ (see Eq  DISPLAY_FORM18), which enables the final graph representation to encode a mixture of low-level and high-level features  Indeed, in initial experiments involving readout at $t$=$T$ only, setting $T\ge 2$ was always decreasing performance, despite the GRU-based updates (Eq  DISPLAY_FORM14)  These results were consistent with that of BIBREF53 and BIBREF9, who both are reading out only at $t$=$T$ too  We hypothesize that node features at $T\ge 2$ are too diffuse to be entirely relied upon during readout  More precisely, initially at $t$=0, node representations capture information about words, at $t$=1, about their 1-hop neighborhood (bigrams), at $t$=2, about compositions of bigrams, etc  Thus, pretty quickly, node features become general and diffuse  In such cases, considering also the lower-level, more precise features of the earlier iterations when reading out may be necessary 
Undirected edges  On Reuters, using an undirected graph leads to better performance, while on Polarity and IMDB, it is the opposite  This can be explained by the fact that Reuters is a topic classification task, for which the presence or absence of some patterns is important, but not necessarily the order in which they appear, while Polarity and IMDB are sentiment analysis tasks  To capture sentiment, modeling word order is crucial, e g , in detecting negation 
No master node  Removing the master node deteriorates performance across all datasets, clearly showing the value of having such a node  We hypothesize that since the special document node is connected to all other nodes, it is able to encode during message passing a summary of the document 
No renormalization  Here, we do not use the renormalization trick of BIBREF9 during MP (see subsection SECREF10)  That is, Eq  DISPLAY_FORM11 becomes $\mathbf {M}^{t+1} = \textsc {MLP}^{t+1}\big (\mathbf {A}\mathbf {H}^{t}\big )$  In other words, instead of computing a weighted average of the incoming neighbors' feature vectors, we compute a weighted sum of them  Unlike the mean, which captures distributions, the sum captures structural information BIBREF14  As shown in Table TABREF29, using sum instead of mean decreases performance everywhere, suggesting that in our application, capturing the distribution of neighbor representations is more important that capturing their structure  We hypothesize that this is the case because statistical word co-occurrence networks tend to have similar structural properties, regardless of the topic, polarity, sentiment, etc  of the corresponding documents 
Neighbors-only  In this experiment, we replaced the GRU combine function (see Eq  DISPLAY_FORM14) with the identity function  That is, we simply have $\mathbf {H}^{t+1}$=$\mathbf {M}^{t+1}$  Since $\mathbf {A}$ has zero diagonal, by doing so, we completely ignore the previous feature of the node itself when updating its representation  That is, the update is based entirely on its neighbors  Except on Reuters (almost no change), performance always suffers, stressing the need to take into account the root node during updates, not only its neighborhood 
Related work
In what follows, we offer a brief review of relevant studies, ranked by increasing order of similarity with our work 
BIBREF9, BIBREF54, BIBREF11, BIBREF10 conduct some node classification experiments on citation networks, where nodes are scientific papers, i e , textual data  However, text is only used to derive node feature vectors  The external graph structure, which plays a central role in determining node labels, is completely unrelated to text 
On the other hand, BIBREF55, BIBREF7 experiment on traditional document classification tasks  They both build $k$-nearest neighbor similarity graphs based on the Gaussian diffusion kernel  More precisely, BIBREF55 build one single graph where nodes are documents and distance is computed in the BoW space  Node features are then used for classification  Closer to our work, BIBREF7 represent each document as a graph  All document graphs are derived from the same underlying structure  Only node features, corresponding to the entries of the documents' BoW vectors, vary  The underlying, shared structure is that of a $k$-NN graph where nodes are vocabulary terms and similarity is the cosine of the word embedding vectors  BIBREF7 then perform graph classification  However they found performance to be lower than that of a naive Bayes classifier 
BIBREF56 use a GNN for hierarchical classification into a large taxonomy of topics  This task differs from traditional document classification  The authors represent documents as unweighted, undirected word co-occurrence networks with word embeddings as node features  They then use the spatial GNN of BIBREF15 to perform graph classification 
The work closest to ours is probably that of BIBREF53  The authors adopt the semi-supervised node classification approach of BIBREF9  They build one single undirected graph from the entire dataset, with both word and document nodes  Document-word edges are weighted by TF-IDF and word-word edges are weighted by pointwise mutual information derived from co-occurrence within a sliding window  There are no document-document edges  The GNN is trained based on the cross-entropy loss computed only for the labeled nodes, that is, the documents in the training set  When the final node representations are obtained, one can use that of the test documents to classify them and evaluate prediction performance 
There are significant differences between BIBREF53 and our work  First, our approach is inductive, not transductive  Indeed, while the node classification approach of BIBREF53 requires all test documents at training time, our graph classification model is able to perform inference on new, never-seen documents  The downside of representing documents as separate graphs, however, is that we lose the ability to capture corpus-level dependencies  Also, our directed graphs capture word ordering, which is ignored by BIBREF53  Finally, the approach of BIBREF53 requires computing the PMI for every word pair in the vocabulary, which may be prohibitive on datasets with very large vocabularies  On the other hand, the complexity of MPAD does not depend on vocabulary size 
Conclusion
We have proposed an application of the message passing framework to NLP, the Message Passing Attention network for Document understanding (MPAD)  Experiments conducted on 10 standard text classification datasets show that our architecture is competitive with the state-of-the-art  By processing weighted, directed word co-occurrence networks, MPAD is sensitive to word order and word-word relationship strength  To explicitly capture the hierarchical structure of documents, we also propose three hierarchical variants of MPAD, that we show bring improvements over the vanilla architecture 
Acknowledgments
We thank the NVidia corporation for the donation of a GPU as part of their GPU grant program 

ground_truth: Various tree structured neural networks including variants of Tree-LSTM, Tree-based CNN, RNTN, and non-tree models including variants of LSTMs, CNNs, residual, and self-attention based networks

Sentence classification baselines: RNTN (Socher et al. 2013), AdaMC-RNTN (Dong et al. 2014), TE-RNTN (Qian et al. 2015), TBCNN (Mou et al. 2015), Tree-LSTM (Tai, Socher, and Manning 2015), AdaHT-LSTM-CM (Liu, Qiu, and Huang 2017), DC-TreeLSTM (Liu, Qiu, and Huang 2017), TE-LSTM (Huang, Qian, and Zhu 2017), BiConTree (Teng and Zhang 2017), Gumbel Tree-LSTM (Choi, Yoo, and Lee 2018), TreeNet (Cheng et al. 2018), CNN (Kim 2014), AdaSent (Zhao, Lu, and Poupart 2015), LSTM-CNN (Zhou et al. 2016), byte-mLSTM (Radford, Jozefowicz, and Sutskever 2017), BCN + Char + CoVe (McCann et al. 2017), BCN + Char + ELMo (Peters et al. 2018). 
Stanford Natural Language Inference baselines: Latent Syntax Tree-LSTM (Yogatama et al. 2017), Tree-based CNN (Mou et al. 2016), Gumbel Tree-LSTM (Choi, Yoo, and Lee 2018), NSE (Munkhdalai and Yu 2017), Reinforced Self- Attention Network (Shen et al. 2018), Residual stacked encoders: (Nie and Bansal 2017), BiLSTM with generalized pooling (Chen, Ling, and Zhu 2018).

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which component is the least impactful?
Context 4:  To construct the nbow representation, given the word embeddings dictionary where each word is associated with a vector, we apply the average compositional function that averages the embeddings of the words that compose a tweet  Simple compositional functions like average were shown to be robust and efficient in previous work BIBREF17   Instead of training embeddings from scratch, we use the pre-trained on tweets GloVe embeddings of BIBREF10   In terms of resources required, using only nbow is efficient as it does not require any domain knowledge  However, previous research on sentiment analysis showed that using extra resources, like sentiment lexicons, can benefit significantly the performance BIBREF3 , BIBREF2   To validate this and examine at which extent neural networks and multitask learning benefit from such features we evaluate the models using an augmented version of nbow, dubbed nbow+  The feature space of the latter, is augmented using 1,368 extra features consisting mostly of counts of punctuation symbols ('!?#@'), emoticons, elongated words and word membership features in several sentiment lexicons  Due to space limitations, for a complete presentation of these features, we refer the interested reader to BIBREF2 , whose open implementation we used to extract them 
Evaluation measure To reproduce the setting of the SemEval challenges BIBREF16 , we optimize our systems using as primary measure the macro-averaged Mean Absolute Error ( INLINEFORM0 ) given by: INLINEFORM1
where INLINEFORM0 is the number of categories, INLINEFORM1 is the set of instances whose true class is INLINEFORM2 , INLINEFORM3 is the true label of the instance INLINEFORM4 and INLINEFORM5 the predicted label  The measure penalizes decisions far from the true ones and is macro-averaged to account for the fact that the data are unbalanced  Complementary to INLINEFORM6 , we report the performance achieved on the micro-averaged INLINEFORM7 measure, which is a commonly used measure for classification 
The models To evaluate the multitask learning approach, we compared it with several other models  Support Vector Machines (SVMs) are maximum margin classification algorithms that have been shown to achieve competitive performance in several text classification problems BIBREF16   SVM INLINEFORM0 stands for an SVM with linear kernel and an one-vs-rest approach for the multi-class problem  Also, SVM INLINEFORM1 is an SVM with linear kernel that employs the crammer-singer strategy BIBREF18 for the multi-class problem  Logistic regression (LR) is another type of linear classification method, with probabilistic motivation  Again, we use two types of Logistic Regression depending on the multi-class strategy: LR INLINEFORM2 that uses an one-vs-rest approach and multinomial Logistic Regression also known as the MaxEnt classifier that uses a multinomial criterion 
Both SVMs and LRs as discussed above treat the problem as a multi-class one, without considering the ordering of the classes  For these four models, we tuned the hyper-parameter INLINEFORM0 that controls the importance of the L INLINEFORM1 regularization part in the optimization problem with grid-search over INLINEFORM2 using 10-fold cross-validation in the union of the training and development data and then retrained the models with the selected values  Also, to account for the unbalanced classification problem we used class weights to penalize more the errors made on the rare classes  These weights were inversely proportional to the frequency of each class  For the four models we used the implementations of Scikit-learn BIBREF19  
For multitask learning we use the architecture shown in Figure FIGREF2 , which we implemented with Keras BIBREF20   The embeddings are initialized with the 50-dimensional GloVe embeddings while the output of the biLSTM network is set to dimension 50  The activation function of the hidden layers is the hyperbolic tangent  The weights of the layers were initialized from a uniform distribution, scaled as described in BIBREF21   We used the Root Mean Square Propagation optimization method  We used dropout for regularizing the network  We trained the network using batches of 128 examples as follows: before selecting the batch, we perform a Bernoulli trial with probability INLINEFORM0 to select the task to train for  With probability INLINEFORM1 we pick a batch for the fine-grained sentiment classification problem, while with probability INLINEFORM2 we pick a batch for the ternary problem  As shown in Figure FIGREF2 , the error is backpropagated until the embeddings, that we fine-tune during the learning process  Notice also that the weights of the network until the layer INLINEFORM3 are shared and therefore affected by both tasks 
To tune the neural network hyper-parameters we used 5-fold cross validation  We tuned the probability INLINEFORM0 of dropout after the hidden layers INLINEFORM1 and for the biLSTM for INLINEFORM2 , the size of the hidden layer INLINEFORM3 and the probability INLINEFORM4 of the Bernoulli trials from INLINEFORM5   During training, we monitor the network's performance on the development set and apply early stopping if the performance on the validation set does not improve for 5 consecutive epochs 
Experimental results Table TABREF9 illustrates the performance of the models for the different data representations  The upper part of the Table summarizes the performance of the baselines  The entry “Balikas et al ” stands for the winning system of the 2016 edition of the challenge BIBREF2 , which to the best of our knowledge holds the state-of-the-art  Due to the stochasticity of training the biLSTM models, we repeat the experiment 10 times and report the average and the standard deviation of the performance achieved 
Several observations can be made from the table  First notice that, overall, the best performance is achieved by the neural network architecture that uses multitask learning  This entails that the system makes use of the available resources efficiently and improves the state-of-the-art performance  In conjunction with the fact that we found the optimal probability INLINEFORM0 , this highlights the benefits of multitask learning over single task learning  Furthermore, as described above, the neural network-based models have only access to the training data as the development are hold for early stopping  On the other hand, the baseline systems were retrained on the union of the train and development sets  Hence, even with fewer resources available for training on the fine-grained problem, the neural networks outperform the baselines  We also highlight the positive effect of the additional features that previous research proposed  Adding the features both in the baselines and in the biLSTM-based architectures improves the INLINEFORM1 scores by several points 
Lastly, we compare the performance of the baseline systems with the performance of the state-of-the-art system of BIBREF2   While BIBREF2 uses n-grams (and character-grams) with INLINEFORM0 , the baseline systems (SVMs, LRs) used in this work use the nbow+ representation, that relies on unigrams  Although they perform on par, the competitive performance of nbow highlights the potential of distributed representations for short-text classification  Further, incorporating structure and distributed representations leads to the gains of the biLSTM network, in the multitask and single task setting 
Similar observations can be drawn from Figure FIGREF10 that presents the INLINEFORM0 scores  Again, the biLSTM network with multitask learning achieves the best performance  It is also to be noted that although the two evaluation measures are correlated in the sense that the ranking of the models is the same, small differences in the INLINEFORM1 have large effect on the scores of the INLINEFORM2 measure 
Conclusion
In this paper, we showed that by jointly learning the tasks of ternary and fine-grained classification with a multitask learning model, one can greatly improve the performance on the second  This opens several avenues for future research  Since sentiment is expressed in different textual types like tweets and paragraph-sized reviews, in different languages (English, German,   ) and in different granularity levels (binary, ternary,  ) one can imagine multitask approaches that could benefit from combining such resources  Also, while we opted for biLSTM networks here, one could use convolutional neural networks or even try to combine different types of networks and tasks to investigate the performance effect of multitask learning  Lastly, while our approach mainly relied on the foundations of BIBREF4 , the internal mechanisms and the theoretical guarantees of multitask learning remain to be better understood 
Acknowledgements
This work is partially supported by the CIFRE N 28/2015 

ground_truth: Based on table results provided changing directed to undirected edges had least impact - max abs difference of 0.33 points on all three datasets.