prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which baselines did they compare against?
Context 1: Introduction
One of the most fundamental topics in natural language processing is how best to derive high-level representations from constituent parts, as natural language meanings are a function of their constituent parts  How best to construct a sentence representation from distributed word embeddings is an example domain of this larger issue  Even though sequential neural models such as recurrent neural networks (RNN) BIBREF0 and their variants including Long Short-Term Memory (LSTM) BIBREF1 and Gated Recurrent Unit (GRU) BIBREF2 have become the de-facto standard for condensing sentence-level information from a sequence of words into a fixed vector, there have been many lines of research towards better sentence representation using other neural architectures, e g  convolutional neural networks (CNN) BIBREF3 or self-attention based models BIBREF4  
From a linguistic point of view, the underlying tree structure—as expressed by its constituency and dependency trees—of a sentence is an integral part of its meaning  Inspired by this fact, some recursive neural network (RvNN) models are designed to reflect the syntactic tree structure, achieving impressive results on several sentence-level tasks such as sentiment analysis BIBREF5 , BIBREF6 , machine translation BIBREF7 , natural language inference BIBREF8 , and discourse relation classification BIBREF9  
However, some recent works have BIBREF10 , BIBREF11 proposed latent tree models, which learn to construct task-specific tree structures without explicit supervision, bringing into question the value of linguistically-motivated recursive neural models  Witnessing the surprising performance of the latent tree models on some sentence-level tasks, there arises a natural question: Are linguistic tree structures the optimal way of composing sentence representations for NLP tasks?
In this paper, we demonstrate that linguistic priors are in fact useful for devising effective neural models for sentence representations, showing that our novel architecture based on constituency trees and their tag information obtains superior performance on several sentence-level tasks, including sentiment analysis and natural language inference 
A chief novelty of our approach is that we introduce a small separate tag-level tree-LSTM to control the composition function of the existing word-level tree-LSTM, which is in charge of extracting helpful syntactic signals for meaningful semantic composition of constituents by considering both the structures and linguistic tags of constituency trees simultaneously  In addition, we demonstrate that applying a typical LSTM to preprocess the leaf nodes of a tree-LSTM greatly improves the performance of the tree models  Moreover, we propose a clustered tag set to replace the existing tags on the assumption that the original syntactic tags are too fined-grained to be useful in neural models 
In short, our contributions in this work are as follows:
Related Work
Recursive neural networks (RvNN) are a kind of neural architecture which model sentences by exploiting syntactic structure  While earlier RvNN models proposed utilizing diverse composition functions, including feed-forward neural networks BIBREF12 , matrix-vector multiplication BIBREF5 , and tensor computation BIBREF6 , tree-LSTMs BIBREF13 remain the standard for several sentence-level tasks 
Even though classic RvNNs have demonstrated superior performance on a variety of tasks, their inflexibility, i e  their inability to handle dynamic compositionality for different syntactic configurations, is a considerable weakness  For instance, it would be desirable if our model could distinguish e g  adjective-noun composition from that of verb-noun or preposition-noun composition, as models failing to make such a distinction ignore real-world syntactic considerations such as `-arity' of function words (i e  types), and the adjunct/argument distinction 
To enable dynamic compositionality in recursive neural networks, many previous works BIBREF14 , BIBREF15 , BIBREF16 , BIBREF9 , BIBREF17 , BIBREF18 , BIBREF19 have proposed various methods 
One main direction of research leverages tag information, which is produced as a by-product of parsing  In detail, BIBREF16 ( BIBREF16 ) suggested TG-RNN, a model employing different composition functions according to POS tags, and TE-RNN/TE-RNTN, models which leverage tag embeddings as additional inputs for the existing tree-structured models  Despite the novelty of utilizing tag information, the explosion of the number of parameters (in case of the TG-RNN) and the limited performance of the original models (in case of the TE-RNN/TE-RNTN) have prevented these models from being widely adopted  Meanwhile, BIBREF9 ( BIBREF9 ) and BIBREF18 ( BIBREF18 ) proposed models based on a tree-LSTM which also uses the tag vectors to control the gate functions of the tree-LSTM  In spite of their impressive results, there is a limitation that the trained tag embeddings are too simple to reflect the rich information which tags provide in different syntactic structures  To alleviate this problem, we introduce structure-aware tag representations in the next section 
Another way of building dynamic compositionality into RvNNs is to take advantage of a meta-network (or hyper-network)  Inspired by recent works on dynamic parameter prediction, DC-TreeLSTMs BIBREF17 dynamically create the parameters for compositional functions in a tree-LSTM  Specifically, the model has two separate tree-LSTM networks whose architectures are similar, but the smaller of the two is utilized to calculate the weights of the bigger one  A possible problem for this model is that it may be easy to be trained such that the role of each tree-LSTM is ambiguous, as they share the same input, i e  word information  Therefore, we design two disentangled tree-LSTMs in our model so that one focuses on extracting useful features from only syntactic information while the other composes semantic units with the aid of the features  Furthermore, our model reduces the complexity of computation by utilizing typical tree-LSTM frameworks instead of computing the weights for each example 
Finally, some recent works BIBREF10 , BIBREF11 have proposed latent tree-structured models that learn how to formulate tree structures from only sequences of tokens, without the aid of syntactic trees or linguistic information  The latent tree models have the advantage of being able to find the optimized task-specific order of composition rather than a sequential or syntactic one  In experiments, we compare our model with not only syntactic tree-based models but also latent tree models, demonstrating that modeling with explicit linguistic knowledge can be an attractive option 
Model
In this section, we introduce a novel RvNN architecture, called SATA Tree-LSTM (Structure-Aware Tag Augmented Tree-LSTM)  This model is similar to typical Tree-LSTMs, but provides dynamic compositionality by augmenting a separate tag-level tree-LSTM which produces structure-aware tag representations for each node in a tree  In other words, our model has two independent tree-structured modules based on the same constituency tree, one of which (word-level tree-LSTM) is responsible for constructing sentence representations given a sequence of words as usual, while the other (tag-level tree-LSTM) provides supplementary syntactic information to the former 
In section 3 1, we first review tree-LSTM architectures  Then in section 3 2, we introduce a tag-level tree-LSTM and structure-aware tag representations  In section 3 3, we discuss an additional technique to boost the performance of tree-structured models, and in section 3 4, we describe the entire architecture of our model in detail 
Tree-LSTM
The LSTM BIBREF1 architecture was first introduced as an extension of the RNN architecture to mitigate the vanishing and exploding gradient problems  In addition, several works have discovered that applying the LSTM cell into tree structures can be an effective means of modeling sentence representations 
To be formal, the composition function of the cell in a tree-LSTM can be formulated as follows:
$$  \begin{bmatrix} \mathbf {i} \\ \mathbf {f}_l \\ \mathbf {f}_r \\ \mathbf {o} \\ \mathbf {g} \end{bmatrix} = \begin{bmatrix} \sigma \\ \sigma \\ \sigma \\ \sigma \\ \tanh \end{bmatrix} \Bigg ( \mathbf {W} \begin{bmatrix} \mathbf {h}_l \\ \mathbf {h}_r \\ \end{bmatrix} + \mathbf {b} \Bigg )$$   (Eq  8)
$$  \mathbf {c} = \mathbf {f}_l \odot \mathbf {c}_l + \mathbf {f}_r \odot \mathbf {c}_r + \mathbf {i} \odot \mathbf {g}\\$$   (Eq  9)
where $\mathbf {h}, \mathbf {c} \in \mathbb {R}^{d}$ indicate the hidden state and cell state of the LSTM cell, and $\mathbf {h}_l, \mathbf {h}_r, \mathbf {c}_l, \mathbf {c}_r \in \mathbb {R}^{d}$ the hidden states and cell states of a left and right child  $\mathbf {g} \in \mathbb {R}^{d}$ is the newly composed input for the cell and $\mathbf {i}, \mathbf {f}_{l}, \mathbf {f}_{r}, \mathbf {o} \in \mathbb {R}^{d}$ represent an input gate, two forget gates (left, right), and an output gate respectively  $\mathbf {W} \in \mathbb {R}^{5d\times 2d}$ and $\mathbf {b} \in \mathbb {R}^{5d}$ are trainable parameters  $\sigma $ corresponds to the sigmoid function, $\tanh $ to the hyperbolic tangent, and $\odot $ to element-wise multiplication 
Note the equations assume that there are only two children for each node, i e  binary or binarized trees, following the standard in the literature  While RvNN models can be constructed on any tree structure, in this work we only consider constituency trees as inputs 
In spite of the obvious upside that recursive models have in being so flexible, they are known for being difficult to fully utilize with batch computations as compared to other neural architectures because of the diversity of structure found across sentences  To alleviate this problem, BIBREF8 ( BIBREF8 ) proposed the SPINN model, which brings a shift-reduce algorithm to the tree-LSTM  As SPINN simplifies the process of constructing a tree into only two operations, i e  shift and reduce, it can support more effective parallel computations while enjoying the advantages of tree structures  For efficiency, our model also starts from our own SPINN re-implementation, whose function is exactly the same as that of the tree-LSTM 
Structure-aware Tag Representation
In most previous works using linguistic tag information BIBREF16 , BIBREF9 , BIBREF18 , tags are usually represented as simple low-dimensional dense vectors, similar to word embeddings  This approach seems reasonable in the case of POS tags that are attached to the corresponding words, but phrase-level constituent tags (e g  NP, VP, ADJP) vary greatly in size and shape, making them less amenable to uniform treatment  For instance, even the same phrase tags within different syntactic contexts can vary greatly in size and internal structure, as the case of NP tags in Figure 1 shows  Here, the NP consisting of DT[the]-NN[stories] has a different internal structure than the NP consisting of NP[the film 's]-NNS[shortcomings] 
One way of deriving structure-aware tag representations from the original tag embeddings is to introduce a separate tag-level tree-LSTM which accepts the typical tag embeddings at each node of a tree and outputs the computed structure-aware tag representations for the nodes  Note that the module concentrates on extracting useful syntactic features by considering only the tags and structures of the trees, excluding word information 
Formally, we denote a tag embedding for the tag attached to each node in a tree as $\textbf {e} \in \mathbb {R}^{d_\text{T}}$   Then, the function of each cell in the tag tree-LSTM is defined in the following way  Leaf nodes are defined by the following:
$$  \begin{bmatrix} \hat{\mathbf {c}} \\ \hat{\mathbf {h}} \\ \end{bmatrix} = \tanh {\left(\mathbf {U}_\text{T} \mathbf {e} + \mathbf {a}_\text{T}\right)}$$   (Eq  13)
while non-leaf nodes are defined by the following:
$$  \begin{bmatrix} \hat{\mathbf {i}} \\ \hat{\mathbf {f}}_l \\ \hat{\mathbf {f}}_r \\ \hat{\mathbf {o}} \\ \hat{\mathbf {g}} \end{bmatrix} = \begin{bmatrix} \sigma \\ \sigma \\ \sigma \\ \sigma \\ \tanh \end{bmatrix} \Bigg ( \mathbf {W_\text{T}} \begin{bmatrix} \hat{\mathbf {h}}_l \\ \hat{\mathbf {h}}_r \\ \mathbf {e} \\ \end{bmatrix} + \mathbf {b}_\text{T} \Bigg )$$   (Eq  14)
$$  \hat{\mathbf {c}} = \hat{\mathbf {f}}_l \odot \hat{\mathbf {c}}_l + \hat{\mathbf {f}}_r \odot \hat{\mathbf {c}}_r + \hat{\mathbf {i}} \odot \hat{\mathbf {g}}\\$$   (Eq  15)
where $\hat{\mathbf {h}}, \hat{\mathbf {c}} \in \mathbb {R}^{d_\text{T}}$ represent the hidden state and cell state of each node in the tag tree-LSTM  We regard the hidden state ( $\hat{\mathbf {h}}$ ) as a structure-aware tag representation for the node  $ \mathbf {U}_\text{T} \in \mathbb {R}^{2d_\text{T} \times d_\text{T}}, \textbf {a}_\text{T} \in \mathbb {R}^{2d_\text{T}}, \mathbf {W}_\text{T} \in \mathbb {R}^{5d_\text{T} \times 3d_\text{T}}$ , and $\mathbf {b}_\text{T} \in \mathbb {R}^{5d_\text{T}}$ are trainable parameters  The rest of the notation follows equations 8 , 9 , and 10   In case of leaf nodes, the states are computed by a simple non-linear transformation  Meanwhile, the composition function in a non-leaf node absorbs the tag embedding ( $\mathbf {e}$ ) as an additional input as well as the hidden states of the two children nodes  The benefit of revising tag representations according to the internal structure is that the derived embedding is a function of the corresponding makeup of the node, rather than a monolithic, categorical tag 
With regard to the tags themselves, we conjecture that the taxonomy of the tags currently in use in many NLP systems is too complex to be utilized effectively in deep neural models, considering the specificity of many tag sets and the limited amount of data with which to train  Thus, we cluster POS (word-level) tags into 12 groups following the universal POS tagset BIBREF20 and phrase-level tags into 11 groups according to criteria analogous to the case of words, resulting in 23 tag categories in total  In this work, we use the revised coarse-grained tags instead of the original ones  For more details, we refer readers to the supplemental materials 
Leaf-LSTM
An inherent shortcoming of RvNNs relative to sequential models is that each intermediate representation in a tree is unaware of its external context until all the information is gathered together at the root node  In other words, each composition process is prone to be locally optimized rather than globally optimized 
To mitigate this problem, we propose using a leaf-LSTM following the convention of some previous works BIBREF21 , BIBREF7 , BIBREF11 , which is a typical LSTM that accepts a sequence of words in order  Instead of leveraging word embeddings directly, we can use each hidden state and cell state of the leaf-LSTM as input tokens for leaf nodes in a tree-LSTM, anticipating the proper contextualization of the input sequence 
Formally, we denote a sequence of words in an input sentence as $w_{1:n}$ ( $n$ : the length of the sentence), and the corresponding word embeddings as $\mathbf {x}_{1:n}$   Then, the operation of the leaf-LSTM at time $t$ can be formulated as,
$$  \begin{bmatrix} \tilde{\mathbf {i}} \\ \tilde{\mathbf {f}} \\ \tilde{\mathbf {o}} \\ \tilde{\mathbf {g}} \end{bmatrix} = \begin{bmatrix} \sigma \\ \sigma \\ \sigma \\ \tanh \end{bmatrix} \Bigg ( \mathbf {W}_\text{L} \begin{bmatrix} \tilde{\mathbf {h}}_{t-1} \\ \mathbf {x}_t \\ \end{bmatrix} + \mathbf {b}_\text{L} \Bigg )$$   (Eq  18)
$$  \tilde{\mathbf {c}}_t = \tilde{\mathbf {f}} \odot \tilde{\mathbf {c}}_{t-1} + \tilde{\mathbf {i}} \odot \tilde{\mathbf {g}}\\$$   (Eq  19)
where $\mathbf {x}_t \in \mathbb {R}^{d_w}$ indicates an input word vector and $\tilde{\mathbf {h}}_t$ , $\tilde{\mathbf {c}}_t \in \mathbb {R}^{d_h}$ represent the hidden and cell state of the LSTM at time $t$ ( $\tilde{\mathbf {h}}_{t-1}$ corresponds to the hidden state at time $t$ -1)  $\mathbf {W}_\text{L}$ and $\mathbf {b}_\text{L} $ are learnable parameters  The remaining notation follows that of the tree-LSTM above 
In experiments, we demonstrate that introducing a leaf-LSTM fares better at processing the input words of a tree-LSTM compared to using a feed-forward neural network  We also explore the possibility of its bidirectional setting in ablation study 
SATA Tree-LSTM
In this section, we define SATA Tree-LSTM (Structure-Aware Tag Augmented Tree-LSTM, see Figure 2 ) which joins a tag-level tree-LSTM (section 3 2), a leaf-LSTM (section 3 3), and the original word tree-LSTM together 
As above we denote a sequence of words in an input sentence as $w_{1:n}$ and the corresponding word embeddings as $\mathbf {x}_{1:n}$   In addition, a tag embedding for the tag attached to each node in a tree is denoted by $\textbf {e} \in \mathbb {R}^{d_\text{T}}$   Then, we derive the final sentence representation for the input sentence with our model in two steps 
First, we compute structure-aware tag representations ( $\hat{\mathbf {h}}$ ) for each node of a tree using the tag tree-LSTM (the right side of Figure 2 ) as follows:
$$  \begin{bmatrix} \hat{\mathbf {c}} \\ \hat{\mathbf {h}} \\ \end{bmatrix} = {\left\lbrace \begin{array}{ll} \text{Tag-Tree-LSTM}(\mathbf {e}) & \text{if a leaf node} \\ \text{Tag-Tree-LSTM}(\hat{\mathbf {h}}_l, \hat{\mathbf {h}}_r, \mathbf {e}) & \text{otherwise} \end{array}\right }$$   (Eq  23)
where Tag-Tree-LSTM indicates the module we described in section 3 2 
Second, we combine semantic units recursively on the word tree-LSTM in a bottom-up fashion  For leaf nodes, we leverage the Leaf-LSTM (the bottom-left of Figure 2 , explained in section 3 3) to compute $\tilde{\mathbf {c}}_{t}$ and $\tilde{\mathbf {h}}_{t}$ in sequential order, with the corresponding input $\mathbf {x}_t$  
$$  \begin{bmatrix} \tilde{\mathbf {c}}_{t} \\ \tilde{\mathbf {h}}_{t} \\ \end{bmatrix} = \text{Leaf-LSTM}(\tilde{\textbf {h}}_{t-1}, \textbf {x}_t)$$   (Eq  24)
Then, the $\tilde{\mathbf {c}}_{t}$ and $\tilde{\mathbf {h}}_{t}$ can be utilized as input tokens to the word tree-LSTM, with the left (right) child of the target node corresponding to the $t$ th word in the input sentence 
$$  \begin{bmatrix} \check{\textbf {c}}_{\lbrace l, r\rbrace } \\ \check{\textbf {h}}_{\lbrace l, r\rbrace } \end{bmatrix} = \begin{bmatrix} \tilde{\textbf {c}}_{t} \\ \tilde{\textbf {h}}_{t} \end{bmatrix}$$   (Eq  25)
In the non-leaf node case, we calculate phrase representations for each node in the word tree-LSTM (the upper-left of Figure 2 ) recursively as follows:
$$  \check{\mathbf {g}} = \tanh {\left( \mathbf {U}_\text{w} \begin{bmatrix} \check{\mathbf {h}}_l \\ \check{\mathbf {h}}_r \\ \end{bmatrix} + \mathbf {a}_\text{w} \right)}$$   (Eq  26)
$$  \begin{bmatrix} \check{\mathbf {i}} \\ \check{\mathbf {f}}_l \\ \check{\mathbf {f}}_r \\ \check{\mathbf {o}} \end{bmatrix} = \begin{bmatrix} \sigma \\ \sigma \\ \sigma \\ \sigma \end{bmatrix} \Bigg ( \mathbf {W_\text{w}} \begin{bmatrix} \check{\mathbf {h}}_l \\ \check{\mathbf {h}}_r \\ \hat{\mathbf {h}} \\ \end{bmatrix} + \mathbf {b}_\text{w} \Bigg )$$   (Eq  27)
where $\check{\mathbf {h}}$ , $\check{\mathbf {c}} \in \mathbb {R}^{d_h}$ represent the hidden and cell state of each node in the word tree-LSTM  $\mathbf {U}_\text{w} \in \mathbb {R}^{d_h \times 2d_h}$ , $\mathbf {W}_\text{w} \in \mathbb {R}^{4d_h \times \left(2d_h+d_\text{T}\right)}$ , $\mathbf {a}_\text{w} \in \mathbb {R}^{d_h}$ , $\mathbf {b}_\text{w} \in \mathbb {R}^{4d_h}$ are learned parameters  The remaining notation follows those of the previous sections  Note that the structure-aware tag representations ( $\hat{\mathbf {h}}$ ) are only utilized to control the gate functions of the word tree-LSTM in the form of additional inputs, and are not involved in the semantic composition ( $\check{\mathbf {g}}$ ) directly 
Finally, the hidden state of the root node ( $\check{\mathbf {h}}_\text{root}$ ) in the word-level tree-LSTM becomes the final sentence representation of the input sentence 
Quantitative Analysis
One of the most basic approaches to evaluate a sentence encoder is to measure the classification performance with the sentence representations made by the encoder  Thus, we conduct experiments on the following five datasets  (Summary statistics for the datasets are reported in the supplemental materials )
MR: A group of movie reviews with binary (positive / negative) classes  BIBREF22
SST-2: Stanford Sentiment Treebank BIBREF6   Similar to MR, but each review is provided in the form of a binary parse tree whose nodes are annotated with numeric sentiment values  For SST-2, we only consider binary (positive / negative) classes 
SST-5: Identical to SST-2, but the reviews are grouped into fine-grained (very negative, negative, neutral, positive, very positive) classes 
SUBJ: Sentences grouped as being either subjective or objective (binary classes)  BIBREF23
TREC: A dataset which groups questions into six different question types (classes)  BIBREF24
As a preprocessing step, we construct parse trees for the sentences in the datasets using the Stanford PCFG parser BIBREF25   Because syntactic tags are by-products of constituency parsing, we do not need further preprocessing 
To classify the sentence given our sentence representation ( $\check{\mathbf {h}}_\text{root}$ ), we use one fully-connected layer with a ReLU activation, followed by a softmax classifier  The final predicted probability distribution of the class $y$ given the sentence $w_{1:n}$ is defined as follows,
$$\mathbf {s} = \text{ReLU}(\mathbf {W}_\text{s} \check{\mathbf {h}}_\text{root}+ \mathbf {b}_\text{s})$$   (Eq  37)
$$p(y|w_{1:n}) = \text{softmax}(\mathbf {W}_\text{c}\mathbf {s} + \mathbf {b}_\text{c})$$   (Eq  38)
where $\textbf {s} \in \mathbb {R}^{d_\text{s}}$ is the computed task-specific sentence representation for the classifier, and $\textbf {W}_\text{s} \in \mathbb {R}^{d_\text{s} \times d_h}$ , $\textbf {W}_\text{c} \in \mathbb {R}^{d_\text{c} \times d_s}$ , $\textbf {b}_\text{s} \in \mathbb {R}^{d_s}$ , $\textbf {b}_\text{c} \in \mathbb {R}^{d_c}$ are trainable parameters  As an objective function, we use the cross entropy of the predicted and true class distributions 
The results of the experiments on the five datasets are shown in table 1   In this table, we report the test accuracy of our model and various other models on each dataset in terms of percentage  To consider the effects of random initialization, we report the best numbers obtained from each several runs with hyper-parameters fixed 
Compared with the previous syntactic tree-based models as well as other neural models, our SATA Tree-LSTM shows superior or competitive performance on all tasks  Specifically, our model achieves new state-of-the-art results within the tree-structured model class on 4 out of 5 sentence classification tasks—SST-2, SST-5, MR, and TREC  The model shows its strength, in particular, when the datasets provide phrase-level supervision to facilitate tree structure learning (i e  SST-2, SST-5)  Moreover, the numbers we report for SST-5 and TREC are competitive to the existing state-of-the-art results including ones from structurally pre-trained models such as ELMo BIBREF26 , proving our model's superiority  Note that the SATA Tree-LSTM also outperforms the recent latent tree-based model, indicating that modeling a neural model with explicit linguistic knowledge can be an attractive option 
On the other hand, a remaining concern is that our SATA Tree-LSTM is not robust to random seeds when the size of a dataset is relatively small, as tag embeddings are randomly initialized rather than relying on pre-trained ones in contrast with the case of words  From this observation, we could find out there needs a direction of research towards pre-trained tag embeddings 
To estimate the performance of our model beyond the tasks requiring only one sentence at a time, we conduct an experiment on the Stanford Natural Language Inference BIBREF34 dataset, each example of which consists of two sentences, the premise and the hypothesis  Our objective given the data is to predict the correct relationship between the two sentences among three options— contradiction, neutral, or entailment 
We use the siamese architecture to encode both the premise ( $p_{1:m}$ ) and hypothesis ( $h_{1:n}$ ) following the standard of sentence-encoding models in the literature  (Specifically, $p_{1:m}$ is encoded as $\check{\mathbf {h}}_\text{root}^p \in \mathbb {R}^{d_h}$ and $h_{1:n}$ is encoded as $\check{\mathbf {h}}_\text{root}^h \in \mathbb {R}^{d_h}$ with the same encoder ) Then, we leverage some heuristics BIBREF35 , followed by one fully-connected layer with a ReLU activation and a softmax classifier  Specifically,
$$\mathbf {z} = \left[ \check{\mathbf {h}}_\text{root}^p; \check{\mathbf {h}}_\text{root}^h; | \check{\mathbf {h}}_\text{root}^p - \check{\mathbf {h}}_\text{root}^h |; \check{\mathbf {h}}_\text{root}^p \odot \check{\mathbf {h}}_\text{root}^h \right]$$   (Eq  41)
$$\mathbf {s} = \text{ReLU}(\mathbf {W}_\text{s} \mathbf {z} + \mathbf {b}_\text{s})$$   (Eq  42)
where $\textbf {z} \in \mathbb {R}^{4d_h}$ , $\textbf {s} \in \mathbb {R}^{d_s}$ are intermediate features for the classifier and $\textbf {W}_\text{s} \in \mathbb {R}^{d_\text{s} \times 4d_h}$ , $\textbf {W}_\text{c} \in \mathbb {R}^{d_\text{c} \times d_s}$ , $\textbf {b}_\text{s} \in \mathbb {R}^{d_s}$ , $\textbf {b}_\text{c} \in \mathbb {R}^{d_c}$ are again trainable parameters 
Our experimental results on the SNLI dataset are shown in table 2   In this table, we report the test accuracy and number of trainable parameters for each model  Our SATA-LSTM again demonstrates its decent performance compared against the neural models built on both syntactic trees and latent trees, as well as the non-tree models  (Latent Syntax Tree-LSTM: BIBREF10 ( BIBREF10 ), Tree-based CNN: BIBREF35 ( BIBREF35 ), Gumbel Tree-LSTM: BIBREF11 ( BIBREF11 ), NSE: BIBREF36 ( BIBREF36 ), Reinforced Self-Attention Network: BIBREF4 ( BIBREF4 ), Residual stacked encoders: BIBREF37 ( BIBREF37 ), BiLSTM with generalized pooling: BIBREF38 ( BIBREF38 ) ) Note that the number of learned parameters in our model is also comparable to other sophisticated models, showing the efficiency of our model 
Even though our model has proven its mettle, the effect of tag information seems relatively weak in the case of SNLI, which contains a large amount of data compared to the others  One possible explanation is that neural models may learn some syntactic rules from large amounts of text when the text size is large enough, reducing the necessity of external linguistic knowledge  We leave the exploration of the effectiveness of tags relative to data size for future work 
Here we go over the settings common across our models during experimentation  For more task-specific details, refer to the supplemental materials 
For our input embeddings, we used 300 dimensional 840B GloVe BIBREF39 as pre-trained word embeddings, and tag representations were randomly sampled from the uniform distribution [-0 005, 0 005]  Tag vectors are revised during training while the fine-tuning of the word embedding depends on the task  Our models were trained using the Adam BIBREF40 or Adadelta BIBREF41 optimizer, depending on task  For regularization, weight decay is added to the loss function except for SNLI following BIBREF42 ( BIBREF42 ) and Dropout BIBREF43 is also applied for the word embeddings and task-specific classifiers  Moreover, batch normalization BIBREF44 is adopted for the classifiers  As a default, all the weights in the model are initialized following BIBREF45 ( BIBREF45 ) and the biases are set to 0  The total norm of the gradients of the parameters is clipped not to be over 5 during training 
Our best models for each dataset were chosen by validation accuracy in cases where a validation set was provided as a part of the dataset  Otherwise, we perform a grid search on probable hyper-parameter settings, or run 10-fold cross-validation in cases where even a test set does not exist 
Ablation Study
In this section, we design an ablation study on the core modules of our model to explore their effectiveness  The dataset used in this experiment is SST-2  To conduct the experiment, we only replace the target module with other candidates while maintaining the other settings  To be specific, we focus on two modules, the leaf-LSTM and structure-aware tag embeddings (tag-level tree-LSTM)  In the first case, the leaf-LSTM is replaced with a fully-connected layer with a $\tanh $ activation or Bi-LSTM  In the second case, we replace the structure-aware tag embeddings with naive tag embeddings or do not employ them at all 
The experimental results are depicted in Figure 3   As the chart shows, our model outperforms all the other options we have considered  In detail, the left part of the chart shows that the leaf-LSTM is the most effective option compared to its competitors  Note that the sequential leaf-LSTM is somewhat superior or competitive than the bidirectional leaf-LSTM when both have a comparable number of parameters  We conjecture this may because a backward LSTM does not add additional useful knowledge when the structure of a sentence is already known  In conclusion, we use the uni-directional LSTM as a leaf module because of its simplicity and remarkable performance 
Meanwhile, the right part of the figure demonstrates that our newly introduced structure-aware embeddings have a real impact on improving the model performance  Interestingly, employing the naive tag embeddings made no difference in terms of the test accuracy, even though the absolute validation accuracy increased (not reported in the figure)  This result supports our assumption that tag information should be considered in the structure 
Qualitative Analysis
In previous sections, we have numerically demonstrated that our model is effective in encouraging useful composition of semantic units  Here, we directly investigate the computed representations for each node of a tree, showing that the remarkable performance of our model is mainly due to the gradual and recursive composition of the intermediate representations on the syntactic structure 
To observe the phrase-level embeddings at a glance, we draw a scatter plot in which a point represents the corresponding intermediate representation  We utilize PCA (Principal Component Analysis) to project the representations into a two-dimensional vector space  As a target parse tree, we reuse the one seen in Figure 1   The result is shown in Figure 4  
From this figure, we confirm that the intermediate representations have a hierarchy in the semantic space, which is very similar to that of the parse tree  In other words, as many tree-structured models pursue, we can see the tendency of constructing the representations from the low-level (the bottom of the figure) to the high-level (the top-left and top-right of the figure), integrating the meaning of the constituents recursively  An interesting thing to note is that the final sentence representation is near that of the phrase `, the stories are quietly moving ' rather than that of `Despite the film's shortcomings', catching the main meaning of the sentence 
Conclusion
We have proposed a novel RvNN architecture to fully utilize linguistic priors  A newly introduced tag-level tree-LSTM demonstrates that it can effectively control the composition function of the corresponding word-level tree-LSTM  In addition, the proper contextualization of the input word vectors results in significant performance improvements on several sentence-level tasks  For future work, we plan to explore a new way of exploiting dependency trees effectively, similar to the case of constituency trees 
Acknowledgments
We thank anonymous reviewers for their constructive and fruitful comments  This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF2016M3C4A7952587) 
Context 2: Introduction
Long short term memory (LSTM) units BIBREF1 are popular for many sequence modeling tasks and are used extensively in language modeling  A key to their success is their articulated gating structure, which allows for more control over the information passed along the recurrence  However, despite the sophistication of the gating mechanisms employed in LSTMs and similar recurrent units, the input and context vectors are treated with simple linear transformations prior to gating  Non-linear transformations such as convolutions BIBREF2 have been used, but these have not achieved the performance of well regularized LSTMs for language modeling BIBREF3  
A natural way to improve the expressiveness of linear transformations is to increase the number of dimensions of the input and context vectors, but this comes with a significant increase in the number of parameters which may limit generalizability  An example is shown in Figure FIGREF1 , where LSTMs performance decreases with the increase in dimensions of the input and context vectors  Moreover, the semantics of the input and context vectors are different, suggesting that each may benefit from specialized treatment 
Guided by these insights, we introduce a new recurrent unit, the Pyramidal Recurrent Unit (PRU), which is based on the LSTM gating structure  Figure FIGREF2 provides an overview of the PRU  At the heart of the PRU is the pyramidal transformation (PT), which uses subsampling to effect multiple views of the input vector  The subsampled representations are combined in a pyramidal fusion structure, resulting in richer interactions between the individual dimensions of the input vector than is possible with a linear transformation  Context vectors, which have already undergone this transformation in the previous cell, are modified with a grouped linear transformation (GLT) which allows the network to learn latent representations in high dimensional space with fewer parameters and better generalizability (see Figure FIGREF1 ) 
We show that PRUs can better model contextual information and demonstrate performance gains on the task of language modeling  The PRU improves the perplexity of the current state-of-the-art language model BIBREF0 by up to 1 3 points, reaching perplexities of 56 56 and 64 53 on the Penn Treebank and WikiText2 datasets while learning 15-20% fewer parameters  Replacing an LSTM with a PRU results in improvements in perplexity across a variety of experimental settings  We provide detailed ablations which motivate the design of the PRU architecture, as well as detailed analysis of the effect of the PRU on other components of the language model 
Related work
Multiple methods, including a variety of gating structures and transformations, have been proposed to improve the performance of recurrent neural networks (RNNs)  We first describe these approaches and then provide an overview of recent work in language modeling 
Pyramidal Recurrent Units
We introduce Pyramidal Recurrent Units (PRUs), a new RNN architecture which improves modeling of context by allowing for higher dimensional vector representations while learning fewer parameters  Figure FIGREF2 provides an overview of PRU  We first elaborate on the details of the pyramidal transformation and the grouped linear transformation  We then describe our recurrent unit, PRU 
Pyramidal transformation for input
The basic transformation in many recurrent units is a linear transformation INLINEFORM0 defined as: DISPLAYFORM0
where INLINEFORM0 are learned weights that linearly map INLINEFORM1 to INLINEFORM2   To simplify notation, we omit the biases 
Motivated by successful applications of sub-sampling in computer vision (e g , BIBREF22 , BIBREF23 , BIBREF9 , BIBREF24 ), we subsample input vector INLINEFORM0 into INLINEFORM1 pyramidal levels to achieve representation of the input vector at multiple scales  This sub-sampling operation produces INLINEFORM2 vectors, represented as INLINEFORM3 , where INLINEFORM4 is the sampling rate and INLINEFORM5   We learn scale-specific transformations INLINEFORM6 for each INLINEFORM7   The transformed subsamples are concatenated to produce the pyramidal analog to INLINEFORM8 , here denoted as INLINEFORM9 : DISPLAYFORM0
where INLINEFORM0 indicates concatenation  We note that pyramidal transformation with INLINEFORM1 is the same as the linear transformation 
To improve gradient flow inside the recurrent unit, we combine the input and output using an element-wise sum (when dimension matches) to produce residual analog of pyramidal transformation, as shown in Figure FIGREF2 BIBREF25  
We sub-sample the input vector INLINEFORM0 into INLINEFORM1 pyramidal levels using the kernel-based approach BIBREF8 , BIBREF9   Let us assume that we have a kernel INLINEFORM2 with INLINEFORM3 elements  Then, the input vector INLINEFORM4 can be sub-sampled as: DISPLAYFORM0
where INLINEFORM0 represents the stride and INLINEFORM1  
The number of parameters learned by the linear transformation and the pyramidal transformation with INLINEFORM0 pyramidal levels to map INLINEFORM1 to INLINEFORM2 are INLINEFORM3 and INLINEFORM4 respectively  Thus, pyramidal transformation reduces the parameters of a linear transformation by a factor of INLINEFORM5   For example, the pyramidal transformation (with INLINEFORM6 and INLINEFORM7 ) learns INLINEFORM8 fewer parameters than the linear transformation 
Grouped linear transformation for context
Many RNN architectures apply linear transformations to both the input and context vector  However, this may not be ideal due to the differing semantics of each vector  In many NLP applications including language modeling, the input vector is a dense word embedding which is shared across all contexts for a given word in a dataset  In contrast, the context vector is highly contextualized by the current sequence  The differences between the input and context vector motivate their separate treatment in the PRU architecture 
The weights learned using the linear transformation (Eq  EQREF9 ) are reused over multiple time steps, which makes them prone to over-fitting BIBREF26   To combat over-fitting, various methods, such as variational dropout BIBREF26 and weight dropout BIBREF0 , have been proposed to regularize these recurrent connections  To further improve generalization abilities while simultaneously enabling the recurrent unit to learn representations at very high dimensional space, we propose to use grouped linear transformation (GLT) instead of standard linear transformation for recurrent connections BIBREF27   While pyramidal and linear transformations can be applied to transform context vectors, our experimental results in Section SECREF39 suggests that GLTs are more effective 
The linear transformation INLINEFORM0 maps INLINEFORM1 linearly to INLINEFORM2   Grouped linear transformations break the linear interactions by factoring the linear transformation into two steps  First, a GLT splits the input vector INLINEFORM3 into INLINEFORM4 smaller groups such that INLINEFORM5   Second, a linear transformation INLINEFORM6 is applied to map INLINEFORM7 linearly to INLINEFORM8 , for each INLINEFORM9   The INLINEFORM10 resultant output vectors INLINEFORM11 are concatenated to produce the final output vector INLINEFORM12   DISPLAYFORM0
GLTs learn representations at low dimensionality  Therefore, a GLT requires INLINEFORM0 fewer parameters than the linear transformation  We note that GLTs are subset of linear transformations  In a linear transformation, each neuron receives an input from each element in the input vector while in a GLT, each neuron receives an input from a subset of the input vector  Therefore, GLT is the same as a linear transformation when INLINEFORM1  
Pyramidal Recurrent Unit
We extend the basic gating architecture of LSTM with the pyramidal and grouped linear transformations outlined above to produce the Pyramidal Recurrent Unit (PRU), whose improved sequence modeling capacity is evidenced in Section SECREF4  
At time INLINEFORM0 , the PRU combines the input vector INLINEFORM1 and the previous context vector (or previous hidden state vector) INLINEFORM2 using the following transformation function as: DISPLAYFORM0
where INLINEFORM0 indexes the various gates in the LSTM model, and INLINEFORM1 and INLINEFORM2 represents the pyramidal and grouped linear transformations defined in Eqns  EQREF10 and EQREF15 , respectively 
We will now incorporate INLINEFORM0 into LSTM gating architecture to produce PRU  At time INLINEFORM1 , a PRU cell takes INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 as inputs to produce forget INLINEFORM5 , input INLINEFORM6 , output INLINEFORM7 , and content INLINEFORM8 gate signals  The inputs are combined with these gate signals to produce context vector INLINEFORM9 and cell state INLINEFORM10   Mathematically, the PRU with the LSTM gating architecture can be defined as: DISPLAYFORM0
where INLINEFORM0 represents the element-wise multiplication operation, and INLINEFORM1 and INLINEFORM2 are the sigmoid and hyperbolic tangent activation functions  We note that LSTM is a special case of PRU when INLINEFORM3 = INLINEFORM4 =1 
Experiments
To showcase the effectiveness of the PRU, we evaluate the performance on two standard datasets for word-level language modeling and compare with state-of-the-art methods  Additionally, we provide a detailed examination of the PRU and its behavior on the language modeling tasks 
Set-up
Following recent works, we compare on two widely used datasets, the Penn Treebank (PTB) BIBREF28 as prepared by BIBREF29 and WikiText2 (WT-2) BIBREF20   For both datasets, we follow the same training, validation, and test splits as in BIBREF0  
We extend the language model, AWD-LSTM BIBREF0 , by replacing LSTM layers with PRU  Our model uses 3-layers of PRU with an embedding size of 400  The number of parameters learned by state-of-the-art methods vary from 18M to 66M with majority of the methods learning about 22M to 24M parameters on the PTB dataset  For a fair comparison with state-of-the-art methods, we fix the model size to 19M and vary the value of INLINEFORM0 and hidden layer sizes so that total number of learned parameters is similar across different configurations  We use 1000, 1200, and 1400 as hidden layer sizes for values of INLINEFORM1 =1,2, and 4, respectively  We use the same settings for the WT-2 dataset  We set the number of pyramidal levels INLINEFORM2 to two in our experiments and use average pooling for sub-sampling  These values are selected based on our ablation experiments on the validation set (Section SECREF39 )  We measure the performance of our models in terms of word-level perplexity  We follow the same training strategy as in BIBREF0  
To understand the effect of regularization methods on the performance of PRUs, we perform experiments under two different settings: (1) Standard dropout: We use a standard dropout BIBREF12 with probability of 0 5 after embedding layer, the output between LSTM layers, and the output of final LSTM layer  (2) Advanced dropout: We use the same dropout techniques with the same dropout values as in BIBREF0   We call this model as AWD-PRU 
Results
Table TABREF23 compares the performance of the PRU with state-of-the-art methods  We can see that the PRU achieves the best performance with fewer parameters 
PRUs achieve either the same or better performance than LSTMs  In particular, the performance of PRUs improves with the increasing value of INLINEFORM0   At INLINEFORM1 , PRUs outperform LSTMs by about 4 points on the PTB dataset and by about 3 points on the WT-2 dataset  This is explained in part by the regularization effect of the grouped linear transformation (Figure FIGREF1 )  With grouped linear and pyramidal transformations, PRUs learn rich representations at very high dimensional space while learning fewer parameters  On the other hand, LSTMs overfit to the training data at such high dimensions and learn INLINEFORM2 to INLINEFORM3 more parameters than PRUs 
With the advanced dropouts, the performance of PRUs improves by about 4 points on the PTB dataset and 7 points on the WT-2 dataset  This further improves with finetuning on the PTB (about 2 points) and WT-2 (about 1 point) datasets 
For similar number of parameters, the PRU with standard dropout outperforms most of the state-of-the-art methods by large margin on the PTB dataset (e g  RAN BIBREF7 by 16 points with 4M less parameters, QRNN BIBREF33 by 16 points with 1M more parameters, and NAS BIBREF31 by 1 58 points with 6M less parameters)  With advanced dropouts, the PRU delivers the best performance  On both datasets, the PRU improves the perplexity by about 1 point while learning 15-20% fewer parameters 
PRU is a drop-in replacement for LSTM, therefore, it can improve language models with modern inference techniques such as dynamic evaluation BIBREF21   When we evaluate PRU-based language models (only with standard dropout) with dynamic evaluation on the PTB test set, the perplexity of PRU ( INLINEFORM0 ) improves from 62 42 to 55 23 while the perplexity of an LSTM ( INLINEFORM1 ) with similar settings improves from 66 29 to 58 79; suggesting that modern inference techniques are equally applicable to PRU-based language models 
Analysis
It is shown above that the PRU can learn representations at higher dimensionality with more generalization power, resulting in performance gains for language modeling  A closer analysis of the impact of the PRU in a language modeling system reveals several factors that help explain how the PRU achieves these gains 
As exemplified in Table TABREF34 , the PRU tends toward more confident decisions, placing more of the probability mass on the top next-word prediction than the LSTM  To quantify this effect, we calculate the entropy of the next-token distribution for both the PRU and the LSTM using 3687 contexts from the PTB validation set  Figure FIGREF32 shows a histogram of the entropies of the distribution, where bins of size 0 23 are used to effect categories  We see that the PRU more often produces lower entropy distributions corresponding to higher confidences for next-token choices  This is evidenced by the mass of the red PRU curve lying in the lower entropy ranges compared to the blue LSTM's curve  The PRU can produce confident decisions in part because more information is encoded in the higher dimensional context vectors 
The PRU has the ability to model individual words at different resolutions through the pyramidal transform; which provides multiple paths for the gradient to the embedding layer (similar to multi-task learning) and improves the flow of information  When considering the embeddings by part of speech, we find that the pyramid level 1 embeddings exhibit higher variance than the LSTM across all POS categories (Figure FIGREF33 ), and that pyramid level 2 embeddings show extremely low variance  We hypothesize that the LSTM must encode both coarse group similarities and individual word differences into the same vector space, reducing the space between individual words of the same category  The PRU can rely on the subsampled embeddings to account for coarse-grained group similarities, allowing for finer individual word distinctions in the embedding layer  This hypothesis is strengthened by the entropy results described above: a model which can make finer distinctions between individual words can more confidently assign probability mass  A model that cannot make these distinctions, such as the LSTM, must spread its probability mass across a larger class of similar words 
Saliency analysis using gradients help identify relevant words in a test sequence that contribute to the prediction BIBREF34 , BIBREF35 , BIBREF36   These approaches compute the relevance as the squared norm of the gradients obtained through back-propagation  Table TABREF34 visualizes the heatmaps for different sequences  PRUs, in general, give more relevance to contextual words than LSTMs, such as southeast (sample 1), cost (sample 2), face (sample 4), and introduced (sample 5), which help in making more confident decisions  Furthermore, when gradients during back-propagation are visualized BIBREF37 (Table TABREF34 ), we find that PRUs have better gradient coverage than LSTMs, suggesting PRUs use more features than LSTMs that contributes to the decision  This also suggests that PRUs update more parameters at each iteration which results in faster training  Language model in BIBREF0 takes 500 and 750 epochs to converge with PRU and LSTM as a recurrent unit, respectively 
Ablation studies
In this section, we provide a systematic analysis of our design choices  Our training methodology is the same as described in Section SECREF19 with the standard dropouts  For a thorough understanding of our design choices, we use a language model with a single layer of PRU and fix the size of embedding and hidden layers to 600  The word-level perplexities are reported on the validation sets of the PTB and the WT-2 datasets 
The two hyper-parameters that control the trade-off between performance and number of parameters in PRUs are the number of pyramidal levels INLINEFORM0 and groups INLINEFORM1   Figure FIGREF35 provides a trade-off between perplexity and recurrent unit (RU) parameters 
Variable INLINEFORM0 and fixed INLINEFORM1 : When we increase the number of pyramidal levels INLINEFORM2 at a fixed value of INLINEFORM3 , the performance of the PRU drops by about 1 to 4 points while reducing the total number of recurrent unit parameters by up to 15%  We note that the PRU with INLINEFORM4 at INLINEFORM5 delivers similar performance as the LSTM while learning about 15% fewer recurrent unit parameters 
Fixed INLINEFORM0 and variable INLINEFORM1 : When we vary the value of INLINEFORM2 at fixed number of pyramidal levels INLINEFORM3 , the total number of recurrent unit parameters decreases significantly with a minimal impact on the perplexity  For example, PRUs with INLINEFORM4 and INLINEFORM5 learns 77% fewer recurrent unit parameters while its perplexity (lower is better) increases by about 12% in comparison to LSTMs  Moreover, the decrease in number of parameters at higher value of INLINEFORM6 enables PRUs to learn the representations in high dimensional space with better generalizability (Table TABREF23 ) 
Table TABREF43 shows the impact of different transformations of the input vector INLINEFORM0 and the context vector INLINEFORM1   We make following observations: (1) Using the pyramidal transformation for the input vectors improves the perplexity by about 1 point on both the PTB and WT-2 datasets while reducing the number of recurrent unit parameters by about 14% (see R1 and R4)  We note that the performance of the PRU drops by up to 1 point when residual connections are not used (R4 and R6)  (2) Using the grouped linear transformation for context vectors reduces the total number of recurrent unit parameters by about 75% while the performance drops by about 11% (see R3 and R4)  When we use the pyramidal transformation instead of the linear transformation, the performance drops by up to 2% while there is no significant drop in the number of parameters (R4 and R5) 
We set sub-sampling kernel INLINEFORM0 (Eq  EQREF12 ) with stride INLINEFORM1 and size of 3 ( INLINEFORM2 ) in four different ways: (1) Skip: We skip every other element in the input vector  (2) Convolution: We initialize the elements of INLINEFORM3 randomly from normal distribution and learn them during training the model  We limit the output values between -1 and 1 using INLINEFORM4 activation function to make training stable  (3) Avg  pool: We initialize the elements of INLINEFORM5 to INLINEFORM6   (4) Max pool: We select the maximum value in the kernel window INLINEFORM7  
Table TABREF45 compares the performance of the PRU with different sampling methods  Average pooling performs the best while skipping give comparable performance  Both of these methods enable the network to learn richer word representations while representing the input vector in different forms, thus delivering higher performance  Surprisingly, a convolution-based sub-sampling method does not perform as well as the averaging method  The INLINEFORM0 function used after convolution limits the range of output values which are further limited by the LSTM gating structure, thereby impeding in the flow of information inside the cell  Max pooling forces the network to learn representations from high magnitude elements, thus distinguishing features between elements vanishes, resulting in poor performance 
Conclusion
We introduce the Pyramidal Recurrent Unit, which better model contextual information by admitting higher dimensional representations with good generalizability  When applied to the task of language modeling, PRUs improve perplexity across several settings, including recent state-of-the-art systems  Our analysis shows that the PRU improves the flow of gradient and expand the word embedding subspace, resulting in more confident decisions  Here we have shown improvements for language modeling  In future, we plan to study the performance of PRUs on different tasks, including machine translation and question answering  In addition, we will study the performance of the PRU on language modeling with more recent inference techniques, such as dynamic evaluation and mixture of softmax 
Acknowledgments
This research was supported by NSF (IIS 1616112, III 1703166), Allen Distinguished Investigator Award, and gifts from Allen Institute for AI, Google, Amazon, and Bloomberg  We are grateful to Aaron Jaech, Hannah Rashkin, Mandar Joshi, Aniruddha Kembhavi, and anonymous reviewers for their helpful comments 
Context 3: Introduction
The use of RNNs in the field of Statistical Machine Translation (SMT) has revolutionised the approaches to automated translation  As opposed to traditional shallow SMT models, which require a lot of memory to run, these neural translation models require only a small fraction of memory used, about 5% BIBREF0   Also, neural translation models are optimized such that every module is trained to jointly improve translation quality  With that being said, one of the main downsides of neural translation models is the heavy corpus requirement in order to ensure learning of deeper contexts  This is where the application of these encoder decoder architectures in translation to and/or from morphologically rich languages takes a severe hit 
For any language pair, the efficiency of an MT system depends on two major factors: the availability and size of parallel corpus used for training and the syntactic divergence between the two languages i e morphological richness, word order differences, grammatical structure etc  BIBREF0   The main differences between the languages stem from the fact that languages similar to English are predominantly fusional languages whereas many of the morphologically rich languages are agglutinative in nature  The nature of morphologically rich languages being structurally and semantically discordant from languages like English adds to the difficulty of SMT involving such languages 
In morphologically rich languages, any suffix can be added to any verb or noun to simply mean one specific thing about that particular word that the suffix commonly represents (agglutination)  This means that there exists a lot of inflectional forms of the same noun and verb base words, conveying similar notions  For example, in Tamil, there are at least 30,000 inflectional forms of any given verb and about 5,000 forms of inflectional forms for any noun  The merged words carry information about part of speech (POS) tags, tense, plurality and so forth that are important for analyzing text for Machine Translation (MT)  Not only are these hidden meanings not captured, the corresponding root words are trained as different units, thereby increasing the complexity of developing such MT systems BIBREF1  
To add to the complexities of being a morphologically rich language, there are several factors unique to Tamil that make translation very difficult  The availability of parallel corpus for Tamil is very scarce  Most of the other models in the field of English–Tamil MT have made use of their own translation corpora that were manually created for the purposes of research  Most of these corpora are not available online for use 
Another issue specific to Tamil is the addition of suffix characters included to the words in the language for smoothness in pronunciation  These characters are of so many different types; there is a unique suffix for each and every consonant in the language  These suffixes degrade performance of MT because the same words with different such pronounciation-based suffixes will be taken as different words in training 
Also to take into consideration is the existence of two different forms of the language being used  Traditionally defined Tamil and its pronunciations aren't acoustically pleasing to use  There's no linguistic flow between syllables and its usage in verbal communication is time consuming  Therefore, there exists two forms of the language, the written form, rigid in structure and syntax, and the spoken form, in which the flow and pace of the language is given priority over syntax and correctness of spelling  This divide leads to the corpus having 2 different versions of the language that increase the vocabulary even with the same words  This can be evidently seen in the corpus between the sentences used in the Bible, which is in traditional Tamil and sentences from movie subtitles, being in spoken Tamil format 
To account for such difficulties, a trade-off between domain specificity and size of the corpus is integral in building an English–Tamil neural MT system 
Corpus
The corpus selected for this experiment was a combination of different corpora from various domains  The major part of the corpus was made up by the EnTam v2 corpus BIBREF2   This corpus contained sentences taken from parallel news articles, English and Tamil bible corpus and movie subtitles  It also comprised of a tourism corpus that was obtained from TDIL (Technology Development for Indian Languages) and a corpus created from Tamil novels and short stories from AU-KBC, Anna university  The complete corpus consisted of 197,792 sentences  Fig  FIGREF20 shows the skinny shift and heatmap representations of the relativity between the sentences in terms of their sentence lengths 
An extra monolingual Tamil corpus, collated from various online sources was used for the word2vec embedding of the Tamil target language to enhance the richness of context of the word vectors  It was also used to create the language model for the phrase-based SMT model  This corpus contained 567,772 sentences and was self-collected by combining hundreds of ancient Tamil scriptures, novels and poems by accessing the websites of popular online ebook libraries in Python using the urllib package  Since the sources had Tamil text in different encodings, the encoding scheme was standardized to be UTF-8 for the entirety of the monolingual and parallel corpora using the chardet package  The corpora were cleaned for any stray special characters, unnecessary html tags and website URLs 
Word2Vec
The word embeddings of the source and target language sentences are used as initial vectors of the model to improve contextualization  The skip gram model of the word2vec algorithm optimizes the vectors by accounting for the average log probability of context words given a source word  DISPLAYFORM0
where k is the context window taken for the vectorization, INLINEFORM0 refers to the INLINEFORM1 word of the corpus and INLINEFORM2 is the size of the training corpus in terms of the number of words  Here, the probabily INLINEFORM3 is computed as a hierarchical softmax of the product of the transpose of the output vector of INLINEFORM4 and the input vector of INLINEFORM5 for each and every pair over the entire vocabulary  The processes of negative sampling and subsampling of frequent words that were used in the original model aren't used in this experiment BIBREF3  
For the process of creating semantically meaningful word embeddings, a monolingual corpus of 569,772 Tamil sentences was used  This gave the vectors more contextual richness due to the increased size of the corpus as opposed to using just the bilingual corpus' target side sentences BIBREF3  
In the experiment, the word2vec model was trained using a vector size of 100 to ensure that the bulk of the limited memory of the GPU will be used for the neural attention translation model  It has been shown that any size over that of 150 used for word vectorization gives similar results and that a size of 100 performs close to the model with 150-sized word vectors BIBREF7   A standard size of 5 was used as window size and the model was trained over 7 worker threads simultaneously  A batch size of 50 words was used for training  The negative sampling was set at 1 as it is the nature of morphologically rich languages to have a lot of important words that don't occur more than once in the corpus  The gensim word2vec toolkit was used to implement this word embedding process BIBREF8  
Neural Translation Model
The model used for translation is the one implemented by Bahdanau et al  Bahdanau2014  A bidirectional LSTM encoder first takes the source sentence and encodes it into a context vector which acts as input for the decoder  The decoder is attention-based where the hidden states of the decoder get as input the weighted sum of all the hidden layer outputs of the encoder alongwith the output of the previous hidden layer and the previously decoded word  This provides a contextual reference into the source language sentence BIBREF4  
Neural Machine Translation models directly compute the probability of the target language sentence given the source language sentence, word by word for every time step  The model with a basic decoder without the attention module computes the log probability of target sentence given source sentence as the sum of log probabilities of every word given every word before that  The attention-based model, on the other hand, calculates: DISPLAYFORM0
where INLINEFORM0 is the number of words in the target sentence, INLINEFORM1 is the target sentence, INLINEFORM2 is the source sentence, INLINEFORM3 is the fixed length output vector of the encoder and INLINEFORM4 is the weighted sum of all the hidden layer outputs of the encoder at every time step  Both the encoder's output context vector and the weighted sum (known as attention vector) help to improve the quality of translation by enabling selective source sentence lookup 
The decoder LSTM computes: DISPLAYFORM0
where the probability is computed as a function of the decoder's output in the previous time step INLINEFORM0 , the hidden layer vector of the decoder in the current timestep INLINEFORM1 and the context vector from the attention mechanism INLINEFORM2   The context vector INLINEFORM3 for time step INLINEFORM4 is computed as a weighted sum of the output of the entire sentence using a weight parameter INLINEFORM5 : DISPLAYFORM0
where INLINEFORM0 is the number of tokens in the source sentence, INLINEFORM1 refers to the value of the hidden layer of the encoder at time step INLINEFORM2 , and INLINEFORM3 is the alignment parameter  This parameter is calculated by means of a feed forward neural network to ensure that the alignment model is free from the difficulties of contextualization of long sentences into a single vector  The feed forward network is trained along with the neural translation model to jointly improve the performance of the translation  Mathematically, DISPLAYFORM0 DISPLAYFORM1
where INLINEFORM0 is the softmax output of the result of the feedforward network, INLINEFORM1 is the hidden state value of the decoder at timestep INLINEFORM2 and INLINEFORM3 is the encoder's hidden layer annotation at timestep INLINEFORM4   A concatenation of the forward and the reverse hidden layer parameters of the encoder is used at each step to compute the weights INLINEFORM5 for the attention mechanism  This is done to enable an overall context of the sentence, as opposed to a context of only all the previous words of the sentence for every word in consideration  Fig  FIGREF12 is the general architecture of the neural translation model without the Bidirectional LSTM encoder 
A global attention mechanism is preferred over local attention because the differences in the structures of the languages cannot be mapped efficiently to enable lookup into the right parts of the source sentence  Using local attention mechanism with a monotonic context lookup, where the region around INLINEFORM0 source word is looked up for the prediction of the INLINEFORM1 target word, is impractical because of the structural discordance between the English and Tamil sentences (see Figs  FIGREF37 and FIGREF44 )  The use of gaussian and other such distributions to facilitate local attention would also be inefficient because the existence of various forms of translations for the same source sentence involving morphological and structural variations that don't stay uniform through the entire corpus BIBREF5  
The No Peepholes (NP) variant of the LSTM cell, formulated in Greff et al  greff2015lstm is used in this experiment as it proved to give the best results amongst all the variants of an LSTM cell  It is specified by means of a gated mechanism designed to ensure that the vanishing gradient problem is prevented  LSTM maintains its hidden layer in two components, the cell vector INLINEFORM0 and the actual hidden layer output vector INLINEFORM1   The cell vector is ensured to never reach zero by means of a weighted sum of the previous layer's cell vector INLINEFORM2 regulated by the forget gate INLINEFORM3 and an activation of the weighted sum of the input INLINEFORM4 in the current timestep INLINEFORM5 and the previous timestep's hidden layer output vector INLINEFORM6 

ground_truth: Various tree structured neural networks including variants of Tree-LSTM, Tree-based CNN, RNTN, and non-tree models including variants of LSTMs, CNNs, residual, and self-attention based networks

Sentence classification baselines: RNTN (Socher et al. 2013), AdaMC-RNTN (Dong et al. 2014), TE-RNTN (Qian et al. 2015), TBCNN (Mou et al. 2015), Tree-LSTM (Tai, Socher, and Manning 2015), AdaHT-LSTM-CM (Liu, Qiu, and Huang 2017), DC-TreeLSTM (Liu, Qiu, and Huang 2017), TE-LSTM (Huang, Qian, and Zhu 2017), BiConTree (Teng and Zhang 2017), Gumbel Tree-LSTM (Choi, Yoo, and Lee 2018), TreeNet (Cheng et al. 2018), CNN (Kim 2014), AdaSent (Zhao, Lu, and Poupart 2015), LSTM-CNN (Zhou et al. 2016), byte-mLSTM (Radford, Jozefowicz, and Sutskever 2017), BCN + Char + CoVe (McCann et al. 2017), BCN + Char + ELMo (Peters et al. 2018). 
Stanford Natural Language Inference baselines: Latent Syntax Tree-LSTM (Yogatama et al. 2017), Tree-based CNN (Mou et al. 2016), Gumbel Tree-LSTM (Choi, Yoo, and Lee 2018), NSE (Munkhdalai and Yu 2017), Reinforced Self- Attention Network (Shen et al. 2018), Residual stacked encoders: (Nie and Bansal 2017), BiLSTM with generalized pooling (Chen, Ling, and Zhu 2018).

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what previous RNN models do they compare with?
Context 3:  The combination is similarly regulated by the input gate INLINEFORM7   The hidden layer output is determined as an activation of the cell gate, regulated by the output gate INLINEFORM8   The interplay between these two vectors ( INLINEFORM9 and INLINEFORM10 ) at every timestep ensures that the problem of vanishing gradients doesn't occur  The three gates are also formed as a sigmoid of the weighted sum of the previous hidden layer output INLINEFORM11 and the input in the current timestep INLINEFORM12   The output generated out of the LSTM's hidden layer is specified as a weighted softmax over the hidden layer output INLINEFORM13   The learnable parameters of an LSTM cell are all the weights INLINEFORM14 and the biases INLINEFORM15   DISPLAYFORM0
The LSTM specified by equations 7 through 11 is the one used for the decoder of the model  The encoder uses a bidirectional RNN LSTM cell in which there are two hidden layer components INLINEFORM0 and INLINEFORM1 that contribute to the output INLINEFORM2 of each time step INLINEFORM3   Both the components have their own sets of LSTM equations in such a way that INLINEFORM4 for every timestep is computed from the first timestep till the INLINEFORM5 token is reached and INLINEFORM6 is computed from the INLINEFORM7 timestep backwards until the first token is reached  All the five vectors of the two components are all exactly the same as the LSTM equations specified with one variation in the computation of the result  DISPLAYFORM0
Morphological Segmentation
The morphological segmentation used is a semi-supervised extension to the generative probabilistic model of maximizing the probability of a INLINEFORM0 prefix,root,postfix INLINEFORM1 recursive split up of words based on an exhaustive combination of all possible morphemes  The details of this model are specified and extensively studied in Kohonen et al  kohonen2010semi  The model parameters INLINEFORM2 include the morph type count, morph token count of training data, the morph strings and their counts  The model is trained by maximizing the Maximum A Posteriori (MAP) probability using Bayes' rule: DISPLAYFORM0
where INLINEFORM0 refers to every word in the training lexicon  The prior INLINEFORM1 is estimated using the Minimum Description Length(MDL) principle  The likelihood INLINEFORM2 is estimated as: DISPLAYFORM0
where INLINEFORM0 refers to the intermediate analyses and INLINEFORM1 refers to the INLINEFORM2 morpheme of word INLINEFORM3  
An extension to the Viterbi algorithm is used for the decoding step based on exhaustive mapping of morphemes  To account for over-segmentation and under-segmentation issues associated with unsupervised morphological segmentation, extra parameters ( INLINEFORM0 ) and ( INLINEFORM1 ) are used with the cost function INLINEFORM2 DISPLAYFORM0
where INLINEFORM0 is the likelihood of the cost function, INLINEFORM1 describes the likelihood of contribution of the annotated dataset to the cost function and INLINEFORM2 is the likelihood of the labeled data  A decrease in the value of INLINEFORM3 will cause smaller segments and vice versa  INLINEFORM4 takes care of size discrepancies due to reduced availability of annotated corpus as compared to the training corpus BIBREF2 , BIBREF6  
The Python extension to the morphological segmentation tool morfessor 2 0 was used for this experiment to perform the segmentation  The annotation data for Tamil language collated and released by Anoop Kunchukkutan in the Indic NLP Library was used as the semi-supervised input to the model BIBREF9 , BIBREF6  
Experiment
The complexities of neural machine translation of morphologically rich languages were studied with respect to English to Tamil machine translation using the RNN LSTM Bi-directional encoder attention decoder architecture  To compare with a baseline system, a phrase based SMT system was implemented using the same corpus  The Factored SMT model with source-side preprocessing by Kumar et al  kumar2014improving was used as a reference for the translation between these language pairs  Also, an additional 569,772 monolingual Tamil sentences were used for the language model of the SMT system  The model used could be split up into various modules as expanded in Fig  FIGREF17  
Bucketing
The input source and target language sentences used for training were taken and divided into bucketed pairs of sentences of a fixed number of sizes  This relationship was determined by examining the distribution of words in the corpus primarily to minimize the number of PAD tokens in the sentence  The heat map of the number of words in the English–Tamil sentence pairs of the corpus revealed that the distribution is centered around the 10–20 words region  Therefore, more buckets in that region were applied as there would be enough number of examples in each of these bucket pairs for the model to learn about the sentences in each and every bucket  The exact scheme used for the RNNSearch models is specified by Fig  FIGREF21   The bucketing scheme for the RNNMorph model, involving morphs instead of words, was a simple shifted scheme of the one used in Fig  FIGREF21 , where every target sentence bucket count was increased uniformly by 5 
Model Details
Due to various computational constraints and lack of availability of comprehensive corpora, the vocabularies for English and Tamil languages for the RNNSearch model were restricted to 60,000 out of 67,768 and 150,000 out of 340,325 respectively  The vocabulary of the languages for the RNNMorph didn't have to be restricted and the actual number of words in the corpus i e  67,768 words for English and 41,906 words for Tamil could be accommodated into the training  Words not in the vocabulary from the test set input and output were replaced with the universal INLINEFORM0 UNK INLINEFORM1 token, symbolizing an unknown word  The LSTM hidden layer size, the training batch size, and the vocabulary sizes of the languages, together, acted as a bottleneck  The model was run on a 2GB NVIDIA GeForce GT 650M card with 384 cores and the memory allotment was constrained to the limits of the GPU  Therefore, after repeated experimentation, it was determined that with a batch size of 16, the maximum hidden layer size possible was 500, which was the size used  Attempts to reduce the batch size resulted in poor convergence, and so the parameters were set to center around the batch size of 16  The models used were of 4 layers of LSTM hidden units in the bidirectional encoder and attention decoder 
The model used a Stochastic Gradient Descent (SGD) optimization algorithm with a sampled softmax loss of 512 per sample to handle large vocabulary size of the target language BIBREF10   The model was trained with a learning rate 1 0 and a decay of rate 0 5 enforced manually  Gradient clipping based on the global norm of 5 0 was carried out to prevent gradients exploding and going to unrecoverable values tending towards infinity  The model described is the one used in the Tensorflow BIBREF11 seq2seq library 
Results and Discussion
The BLEU metric parameters (modified 1-gram, 2-gram, 3-gram and 4-gram precision values) and human evaluation metrics of adequacy, fluency and relative ranking values were used to evaluate the performance of the models 
BLEU Evaluation
The BLEU scores obtained using the various models used in the experiment are tabulated in Table TABREF25  
The BLEU metric computes the BLEU unigram, bigram, trigram and BLEU-4 modified precision values, each micro-averaged over the test set sentences BIBREF7   It was observed, as expected, that the performance of the phrase-based SMT model was inferior to that of the RNNSearch model  The baseline RNNSearch system was further refined by using word2vec vectors to embed semantic understanding, as observed with the slight increase in the BLEU scores  Fig  FIGREF26 plots the BLEU scores as a line graph for visualization of the improvement in performance  Also, the 4-gram BLEU scores for the various models were plotted as a bar graph in Fig  FIGREF26
Due to the agglutinative and morphologically rich nature of the target language i e  Tamil, the use of morphological segmentation to split the words into morphemes further improved the BLEU precision values in the RNNMorph model  One of the reasons for the large extent of increase in the BLEU score could be attributed to the overall increase in the number of word units per sentence  Since the BLEU score computes micro-average precision scores, an increase in both the numerator and denominator of the precision scores is apparent with an increase in the number of tokens due to morphological segmentation of the target language  Thus, the numeric extent of the increase of accuracy might not efficiently describe the improvement in performance of the translation 
Human Evaluation
To ensure that the increase in BLEU score correlated to actual increase in performance of translation, human evaluation metrics like adequacy, precision and ranking values (between RNNSearch and RNNMorph outputs) were estimated in Table TABREF30   A group of 50 native people who were well-versed in both English and Tamil languages acted as annotators for the evaluation  A collection of samples of about 100 sentences were taken from the test set results for comparison  This set included a randomized selection of the translation results to ensure the objectivity of evaluation  Fluency and adequacy results for the RNNMorph results are tabulated  Adequacy rating was calculated on a 5-point scale of how much of the meaning is conveyed by the translation (All, Most, Much, Little, None)  The fluency rating was calculated based on grammatical correctness on a 5-point scale of (Flawless, Good, Non-native, Disfluent, Incomprehensive)  For the comparison process, the RNNMorph and the RNNSearch + Word2Vec models’ sentence level translations were individually ranked between each other, permitting the two translations to have ties in the ranking  The intra-annotator values were computed for these metrics and the scores are shown in Table TABREF32 BIBREF12 , BIBREF13  
The human evaluation Kappa co-efficient results are calculated with respect to: DISPLAYFORM0
It was observed that the ranking Kappa co-efficient for intra-annotator ranking of the RNNMorph model was at 0 573, higher that the 0

ground_truth: Variational LSTM, CharCNN, Pointer Sentinel-LSTM, RHN, NAS Cell, SRU, QRNN, RAN, 4-layer skip-connection LSTM, AWD-LSTM, Quantized LSTM

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How were the human judgements assembled?
Context 3: 410 of the RNNSearch+Word2Vec model, implying that the annotators found the RNNMorph model to produce better results when compared to the RNNSearch + Word2Vec model 
Model Parameters
The learning rate decay through the training process of the RNNMorph model is showcased in the graph in Fig  FIGREF34   This process was done manually where the learning rate was decayed after the end of specific epochs based on an observed stagnation in perplexity The RNNMorph model achieved saturation of perplexities much earlier through the epochs than the RNNSearch + Word2Vec model  This conforms to the expected outcome as the morphological segmentation has reduced the vocabulary size of the target language from 340,325 words to a mere 41,906 morphs 
The error function used was the sampled SoftMax loss to ensure a large target vocabulary could be accommodated BIBREF10   A zoomed inset graph (Fig  FIGREF35 ) has been used to visualize the values of the error function for the RNNSearch + Word2Vec and RNNMorph models with 4 hidden layers  It can be seen that the RNNMorph model is consistently better in terms of the perplexity values through the time steps 
Attention Vectors
In order to further demonstrate the quality of the RNNMorph model, the attention vectors of both the RNNSearch with Word2Vec embedding and RNNMorph models are compared for several good translations in Figs  FIGREF37 and FIGREF44   It is observed that the reduction in vocabulary size has improved the source sentence lookup by quite an extent  Each cell in the heatmap displays the magnitude of the attention layer weight INLINEFORM0 for the INLINEFORM1 Tamil word and the INLINEFORM2 English word in the respective sentences  The intensity of black corresponds to the magnitude of the cell INLINEFORM3   Also, the attention vectors of the RNNSearch model with Word2Vec embeddings tend to attend to INLINEFORM4 EOS INLINEFORM5 token in the middle of the sentence leading to incomplete translations  This could be due to the fact that only 44% of the Tamil vocabulary and 74% of the English vocabulary is taken for training in this model, as opposed to 100% of English and Tamil words in the RNNMorph model 
Target vocabulary size
A very large target vocabulary is an inadvertent consequence of the morphological richness of the Tamil language  This creates a potential restriction on the accuracy of the model as many inflectional forms of the same word are trained as independent units  One of the advantages of morphological segmentation of Tamil text is that the target vocabulary size decreased from 340,325 to a mere 41,906  This reduction helps improve the performance of the translation as the occurrence of unknown tokens was reduced compared to the RNNSearch model  This morphologically segmented vocabulary is divided into a collection of morphological roots and inflections as individual units 
Repetitions
Some of the translations of the RNNMorph model have repetitions of the same phrases (Fig  FIGREF53 ), whereas such repetitions occur much less frequently in the RNNSearch predictions  Such translations would make for good results if the repetitions weren't present and all parts of the sentence occur just once  These repetitions might be due to the increase in the general sequence length of the target sentences because of the morphological segmentation  While it is true the target vocabulary size has decreased due to morphological segmentation, the RNNMorph has more input units (morphs) per sentence, which makes it more demanding of the LSTM's memory units and the feed forward network of the attention model  Additionally, this behavior could also be attributed to the errors in the semi-supervised morphological segmentation due to the complexities of the Tamil language and the extent of the corpus 
Model Outputs
The translation outputs of the RNNSearch + Word2Vec and Morph2Vec models for the same input sentences from the test set demonstrate the effectiveness of using a morphological segmentation tool and how the morphemes have changed the sentence to be more grammatically sound  It is also observed (from Fig  FIGREF55 ) that most of the translation sentences of the Morph2Vec model have no INLINEFORM0 UNK INLINEFORM1 tokens  They exist in the predictions mostly only due to a word in the English test sentence not present in the source vocabulary 
Related Work
Professors CN Krishnan, Sobha et al developed a machine-aided-translation (MAT) system similar to the Anusaakara English Hindi MT system, using a small corpus and very few transfer rules, available at AU-KBC website BIBREF14   Balajapally et al  balajapally2006multilingual developed an example based machine translation (EBMT) system with 700000 sentences for English to INLINEFORM0 Tamil, Kannada, Hindi INLINEFORM1 transliterated text BIBREF15 , BIBREF16   Renganathan renganathan2002interactive developed a rule based MT system for English and Tamil using grammar rules for the language pair  Vetrivel et al  vetrivel2010english used HMMs to align and translate English and Tamil parallel sentences to build an SMT system  Irvine et al  irvine2013combining tried to combine parallel and similar corpora to improve the performance of English to Tamil SMT amongst other languages  Kasthuri et al  kasthuri2014rule used a rule based MT system using transfer lexicon and morphological analysis tools  Anglabharathi was developed at IIT Kanpur, a system translating English to a collection of Indian languages including Tamil using CFG like structures to create a pseudo target to convert to Indian languages BIBREF17 , BIBREF18   A variety of hybrid approaches have also been used for English–Tamil MT in combinations of rule based (transfer methods), interlingua representations BIBREF19 , BIBREF20 , BIBREF21   The use of Statistical Machine Translation took over the English–Tamil MT system research because of its desirable properties of language independence, better generalization features and a reduced requirement of linguistic expertise BIBREF1 , BIBREF22 , BIBREF23   Various enhancement techniques external to the MT system have also been proposed to improve the performance of translation using morphological pre and post processing techniques BIBREF24 , BIBREF25 , BIBREF26  
The use of RNN Encoder Decoder models in machine translation has shown good results in languages with similar grammatical structure  Deep MT systems have been performing better than the other shallow SMT models recently, with the availability of computational resources and hardware making it feasible to train such models  The first of these models came in 2014, with Cho et al SecondOneByCho  The model used was the RNN LSTM encoder decoder model with the context vector output of the encoder (run for every word in the sentence) is fed to every decoder unit along with the previous word output until INLINEFORM0 EOS INLINEFORM1 is reached  This model was used to score translation results of another MT system  Sutskever et al  sutskever2014sequence created a similar encoder decoder model with the decoder getting the context vector only for the first word of the target language sentence  After that, only the decoded target outputs act as inputs to the various time steps of the decoder  One major drawback of these models is the size of the context vector of the encoder being static in nature  The same sized vector was expected to to represent sentences of arbitrary length, which was impractical when it came to very long sentences 
The next breakthrough came from Bahdanau et al  Bahdanau2014 where variable length word vectors were used and instead of just the context vector, a weighted sum of the inputs is given for the decoder  This enabled selective lookup to the source sentence during decoding and is known as the attention mechanism BIBREF27   The attention mechanism was further analysed by Luong et al  luong2015effective where they made a distinction between global and local attention by means of AER scores of the attention vectors  A Gaussian distribution and a monotonic lookup were used to facilitate the corresponding local source sentence look-up 
Conclusion
Thus, it is seen that the use of morphological segmentation on a morphologically rich language before translation helps with the performance of the translation in multiple ways  Thus, machine translation involving morphologically rich languages should ideally be carried out only after morphological segmentation  If the translation has to be carried out between two morphologically rich languages, then both the languages' sentences should be individually segmented based on morphology  This is because while it is true that they are both morphologically rich languages, the schemes that the languages use for the process of agglutination might be different, in which case a mapping between the units would be difficult without the segmentation 
One drawback of morphological segmentation is the increase in complexity of the model due to an increase in the average sentence lengths  This cannot be avoided as it is essential to enable a correspondence between the sentences of the two languages when one of them is a simple fusional language  Even with the increase in the average sentence length, the attention models that have been developed to ensure correctness of translation of long sequences can be put to good use when involving morphologically rich languages  Another point to note here is that morphologically rich languages like Tamil generally have lesser number of words per sentence than languages like English due to the inherent property of agglutination 
Future Work
The model implemented in this paper only includes source-side morphological segmentation and does not include a target side morphological agglutination to give back the output in words rather than morphemes  In order to implement an end-to-end translation system for morphologically rich languages, a morphological generator is essential because the output units of the translation cannot be morphemes 
The same model implemented can be further enhanced by means of a better corpus that can generalize over more than just domain specific source sentences  Also, the use of a better GPU would result in a better allocation of the hidden layer sizes and the batch sizes thereby possibly increasing the scope and accuracy of learning of the translation model 
Although not directly related to Machine Translation, the novel encoder– decoder architecture proposed in by Rocktaschel et al  rocktaschel2015reasoning for Natural Language Inference (NLI) can be used for the same  Their model fuses inferences from each and every individual word, summarizing information at each step, thereby linking the hidden state of the encoder with that of the decoder by means of a weighted sum, trained for optimization 
Acknowledgements
I would like to thank Dr  M  Anand Kumar, Assistant Professor, Amrita Vishwa Vidyapeetham for his continuous support and guidance  I would also like to thank Dr  Arvindan, Professor, SSN College Of Engineering for his inputs and suggestions 

ground_truth: 50 human annotators ranked a random sample of 100 translations by Adequacy, Fluency and overall ranking on a 5-point scale.

adequacy, precision and ranking values