Title: WORDS OR CHARACTERS? FINE-GRAINED GATING FOR READING COMPREHENSION

Abstract: Previous work combines word-level and character-level representations using concatenation or scalar weighting, which is suboptimal for high-level tasks like reading comprehension. We present a fine-grained gating mechanism to dynamically combine word-level and character-level representations based on properties of the words. We also extend the idea of fine-grained gating to modeling the interaction between questions and paragraphs for reading comprehension. Experiments show that our approach can improve the performance on reading comprehension tasks, achieving new state-of-the-art results on the Children's Book Test and Who Did What datasets. To demonstrate the generality of our gating mechanism, we also show improved results on a social media tag prediction task. 1

1. INTRODUCTION:

Finding semantically meaningful representations of the words (also called tokens) in a document is necessary for strong performance in Natural Language Processing tasks. In neural networks, tokens are mainly represented in two ways, either using word-level representations or character-level representations. Word-level representations are obtained from a lookup table, where each unique token is represented as a vector. Character-level representations are usually obtained by applying recurrent neural networks (RNNs) or convolutional neural networks (CNNs) on the character sequence of the token, and their hidden states are combined to form the representation. Word-level representations are good at memorizing the semantics of the tokens while character-level representations are more suitable for modeling sub-word morphologies (Ling et al., 2015; Yang et al., 2016a) . For example, considering "cat" and "cats", word-level representations can only learn the similarities between the two tokens by training on a large amount of training data, while character-level representations, by design, can easily capture the similarities. Character-level representations are also used to alleviate the difficulties of modeling out-of-vocabulary (OOV) tokens (Luong & Manning, 2016) .

Hybrid word-character models have been proposed to leverage the advantages of both word-level and character-level representations. The most commonly used method is to concatenate these two representations (Yang et al., 2016a) . However, concatenating word-level and character-level representations is technically problematic. For frequent tokens, the word-level representations are usually accurately estimated during the training process, and thus introducing character-level representations can potentially bias the entire representations. For infrequent tokens, the estimation of wordlevel representations have high variance, which will have negative effects when combined with the character-level representations. To address this issue, recently Miyamoto & Cho (2016) introduced a scalar gate conditioned on the word-level representations to control the ratio of the two representations. However, for the task of reading comprehension, preliminary experiments showed that this method was not able to improve the performance over concatenation. There are two possible reasons. First, word-level representations might not contain sufficient information to support the decisions of selecting between the two representations. Second, using a scalar gate means applying the same ratio for each of the dimensions, which can be suboptimal.

In this work, we present a fine-grained gating mechanism to combine the word-level and characterlevel representations. We compute a vector gate as a linear projection of the token features followed by a sigmoid activation. We then multiplicatively apply the gate to the character-level and wordlevel representations. Each dimension of the gate controls how much information is flowed from the word-level and character-level representations respectively. We use named entity tags, part-ofspeech tags, document frequencies, and word-level representations as the features for token properties which determine the gate. More generally, our fine-grained gating mechanism can be used to model multiple levels of structure in language, including words, characters, phrases, sentences and paragraphs. In this work we focus on studying the effects on word-character gating.

To better tackle the problem of reading comprehension, we also extend the idea of fine-grained gating for modeling the interaction between documents and queries. Previous work has shown the importance of modeling interactions between document and query tokens by introducing various attention architectures for the task (Hermann et al., 2015; Kadlec et al., 2016) . Most of these use an inner product between the two representations to compute the relative importance of document tokens. The Gated-Attention Reader (Dhingra et al., 2016a) showed improved performance by replacing the inner-product with an element-wise product to allow for better matching at the semantic level. However, they use aggregated representations of the query which may lead to loss of information. In this work we use a fine-grained gating mechanism for each token in the paragraph and each token in the query. The fine-grained gating mechanism applies an element-wise multiplication of the two representations.

We show improved performance on reading comprehension datasets, including Children's Book Test (CBT), Who Did What, and SQuAD. On CBT, our approach achieves new state-of-the-art results without using an ensemble. Our model also improves over state-of-the-art results on the Who Did What dataset. To demonstrate the generality of our method, we apply our word-character finegrained gating mechanism to a social media tag prediction task and show improved performance over previous methods.

Our contributions are two-fold. First, we present a fine-grained word-character gating mechanism and show improved performance on a variety of tasks including reading comprehension. Second, to better tackle the reading comprehension tasks, we extend our fine-grained gating approach to modeling the interaction between documents and queries.

2. RELATED WORK:

Hybrid word-character models have been proposed to take advantages of both word-level and character-level representations. Ling et al. (2015) introduce a compositional character to word (C2W) model based on bidirectional LSTMs. Kim et al. (2016) describe a model that employs a convolutional neural network (CNN) and a highway network over characters for language modeling. Miyamoto & Cho (2016) use a gate to adaptively find the optimal mixture of the character-level and word-level inputs. Yang et al. (2016a) employ deep gated recurrent units on both character and word levels to encode morphology and context information. Concurrent to our work, Rei et al. (2016) employed a similar gating idea to combine word-level and character-level representations, but their focus is on low-level sequence tagging tasks and the gate is not conditioned on linguistic features.

The gating mechanism is widely used in sequence modeling. Long short-term memory (LSTM) networks (Hochreiter & Schmidhuber, 1997) are designed to deal with vanishing gradients through the gating mechanism. Similar to LSTM, Gated Recurrent Unit (GRU) was proposed by Cho et al. (2014) , which also uses gating units to modulate the flow of information. The gating mechanism can also be viewed as a form of attention mechanism (Bahdanau et al., 2015; Yang et al., 2016b) over two inputs.

Similar to the idea of gating, multiplicative integration has also been shown to provide a benefit in various settings. Yang et al. (2014) find that multiplicative operations are superior to additive operations in modeling relations. Wu et al. (2016) propose to use Hadamard product to replace sum operation in recurrent networks, which gives a significant performance boost over existing RNN models. Dhingra et al. (2016a) use a multiplicative gating mechanism to achieve state-of-the-art results on question answering benchmarks.

Reading comprehension is a challenging task for machines. A variety of models have been proposed to extract answers from given text (Hill et al., 2016; Kadlec et al., 2016; Trischler et al., 2016; Chen et al., 2016; Sordoni et al., 2016; Cui et al., 2016) . Yu et al. (2016) propose a dynamic chunk reader to extract and rank a set of answer candidates from a given document to answer questions. Wang & Jiang (2016) introduce an end-to-end neural architecture which incorporates match-LSTM and pointer networks (Vinyals et al., 2015) .

3. FINE-GRAINED GATING:

In this section, we will describe our fine-grained gating approach in the context of reading comprehension. We first introduce the settings of reading comprehension tasks and a general neural network architecture. We will then describe our word-character gating and document-query gating approaches respectively.

3.1. READING COMPREHENSION SETTING:

The reading comprehension task involves a document

P = (p 1 , p 2 , • • • , p M ) and a query Q = (q 1 , q 2 , • • • , q N )

, where M and N are the lengths of the document and the query respectively. Each token p i is denoted as (w i , C i ), where w i is a one-hot encoding of the token in the vocabulary and C i is a matrix with each row representing a one-hot encoding of a character. Each token in the query q j is similarly defined. We use i as a subscript for documents and j for queries. The output of the problem is an answer a, which can either be an index or a span of indices in the document. Now we describe a general architecture used in this work, which is a generalization of the gated attention reader (Dhingra et al., 2016a) . For each token in the document and the query, we compute a vector representation using a function f . More specifically, for each token p i in the document, we have

h 0 i = f (w i , C i ).

The same function f is also applied to the tokens in the query. Let H 0 p and H q denote the vector representations computed by f for tokens in documents and queries respectively. In Section 3.2, we will discuss the "word-character" fine-grained gating used to define the function f . Suppose that we have a network of K layers. At the k-th layer, we apply RNNs on H k-1 p and H q to obtain hidden states P k and Q k , where P k is a M × d matrix and Q k is a N × d matrix with d being the number of hidden units in the RNNs. Then we use a function r to compute a new representation for the document H k p = r(P k , Q k ). In Section 3.3, we will introduce the "document-query" finegrained gating used to define the function r.

After going through K layers, we predict the answer index a using a softmax layer over hidden states H k p . For datasets where the answer is a span of text, we use two softmax layers for the start and end indices respectively.

3.2. WORD-CHARACTER FINE-GRAINED GATING:

Given a one-hot encoding w i and a character sequence C i , we now describe how to compute the vector representation h i = f (w i , C i ) for the token. In the rest of the section, we will drop the subscript i for notation simplicity.

We first apply an RNN on C and take the hidden state in the last time step c as the character-level representation (Yang et al., 2016a) . Let E denote the token embedding lookup table. We perform a matrix-vector multiplication Ew to obtain a word-level representation. We assume c and Ew have the same length d e in this work.

Previous methods defined f using the word-level representation Ew (Collobert et al., 2011) , the character-level representation c (Ling et al., 2015) , or the concatenation [Ew; c] (Yang et al., 2016a) . Unlike these methods, we propose to use a gate to dynamically choose between the word-level and character-level representations based on the properties of the token. Let v denote a feature vector that encodes these properties. In this work, we use the concatenation of named entity tags, partof-speech tags, binned document frequency vectors, and the word-level representations to form the feature vector v. Let d v denote the length of v.

The gate is computed as follows: The final representation is computed using a fine-grained gating mechanism,

g = σ(W g v + b g )

h = f (c, w) = g c + (1 -g) (Ew)

where denotes element-wise product between two vectors.

An illustration of our fine-grained gating mechanism is shown in Figure 1 . Intuitively speaking, when the gate g has high values, more information flows from the character-level representation to the final representation; when the gate g has low values, the final representation is dominated by the word-level representation.

Though Miyamoto & Cho (2016) also use a gate to choose between word-level and character-level representations, our method is different in two ways. First, we use a more fine-grained gating mechanism, i.e., vector gates rather than scalar gates. Second, we condition the gate on features that better reflect the properties of the token. For example, for noun phrases and entities, we would expect the gate to bias towards character-level representations because noun phrases and entities are usually less common and display richer morphological structure. Experiments show that these changes are key to the performance improvements for reading comprehension tasks.

Our approach can be further generalized to a setting of multi-level networks so that we can combine multiple levels of representations using fine-grained gating mechanisms, which we leave for future work.

3.3. DOCUMENT-QUERY FINE-GRAINED GATING:

Given the hidden states P k and Q k , we now describe how to compute a representation H k that encodes the interactions between the document and the query. In this section, we drop the superscript k (the layer number) for notation simplicity. Let p i denote the i-th row of P and q j denote the j-row of Q. Let d h denote the lengths of p i and q j .

Attention-over-attention (AoA) (Cui et al., 2016) defines a dot product between each pair of tokens in the document and the query, i.e., p T i q j , followed by row-wise and column-wise softmax nonlinearities. AoA imposes pair-wise interactions between the document and the query, but using a dot product is potentially not expressive enough and hard to generalize to multi-layer networks. The gated attention (GA) reader (Dhingra et al., 2016a) defines an element-wise product as p i g i where g i is a gate computed by attention mechanism on the token p i and the entire query. The intuition for the gate g i is to attend to important information in the document. However, there is no direct pair-wise interaction between each token pair. We present a fine-grained gating method that combines the advantages of the above methods (i.e., both pairwise and element-wise). We compute the pairwise element-wise product between the hidden states in the document and the query, as shown in Figure 2 . More specifically, for p i and q j , we have

I ij = tanh(p i q j )

where q j can be viewed as a gate to filter the information in p i . We then use an attention mechanism over I ij to output hidden states h i as follows

h i = j softmax(u T h I ij + w T i w j b h1 + b h2 )I ij

where u h is a d v -dimensional model parameter, b h1 and b h2 are scalar model parameters, w i and w j are one-hot encodings for p i and q j respectively. We additionally use one-hot encodings in the attention mechanism to reinforce the matching between the same tokens since such information is not fully preserved in I ij when k is large. The softmax nonlinearity is applied over all j's. The final hidden states H are formed by concatenating the h i 's for each token p i .

4. EXPERIMENTS:

We first present experimental results on the Twitter dataset where we can rule out the effects of different choices of network architectures, to demonstrate the effectiveness of our word-character fine-grained gating approach. Later we show experiments on more challenging datasets on reading comprehension to further show that our approach can be used to improve the performance on highlevel NLP tasks as well.

4.1. EVALUATING WORD-CHARACTER GATING ON TWITTER:

We evaluate the effectiveness of our word-character fine-grained gating mechanism on a social media tag prediction task. We use the Twitter dataset and follow the experimental settings in Dhingra et al. (2016b) . We also use the same network architecture upon the token representations, which is an LSTM layer followed by a softmax classification layer (Dhingra et al., 2016b) . The Twitter dataset consists of English tweets with at least one hashtag from Twitter. Hashtags and HTML tags have been removed from the body of the tweet, and user names and URLs are replaced with special tokens. The dataset contains 2 million tweets for training, 10K for validation and 50K for testing, with a total of 2,039 distinct hashtags. The task is to predict the hashtags of each tweet.

We compare several different methods as follows. Word char concat uses the concatenation of word-level and character-level representations as in Yang et al. (2016a) ; word char feat concat concatenates the word-level and character-level representations along with the features described in 

Model:

Precision@1 Recall@10 Mean Rank word (Dhingra et al., 2016b) Miyamoto & Cho (2016) but is conditioned on the features; fine-grained gate is our method described in Section 3.2. We include word char feat concat for a fair comparison because our fine-grained gating approach also uses the token features.

The results are shown in Table 1 . We report three evaluation metrics including precision@1, recall@10, and mean rank. Our method outperforms character-level models used in Dhingra et al. (2016b) by 2.29%, 2.69%, and 2.5 points in terms of precision, recall and mean rank respectively. We can observe that scalar gating approach (Miyamoto & Cho, 2016) can only marginally improve over the baseline methods, while fine-grained gating methods can substantially improve model performance. Note that directly concatenating the token features with the character-level and word-level representations does not boost the performance, but using the token features to compute a gate (as done in fine-grained gating) leads to better results. This indicates that the benefit of fine-grained gating mainly comes from better modeling rather than using additional features.

4.2. PERFORMANCE ON READING COMPREHENSION:

After investigating the effectiveness of the word-character fine-grained gating mechanism on the Twitter dataset, we now move on to a more challenging task, reading comprehension. In this section, we experiment with two datasets, the Children's Book Test dataset (Hill et al., 2016) and the SQuAD dataset (Rajpurkar et al., 2016) .

4.2.1. CLOZE-STYLE QUESTIONS:

We evaluate our model on cloze-style question answering benchmarks. ,000 for validation and 10,000 for testing. We closely follow the setting in Dhingra et al. (2016a) and incrementally add different components to see the changes in performance. For the fine-grained gating approach, we use the same hyper-parameters as in Dhingra et al. (2016a) except that we use a character-level GRU with 100 units to be of the same size as the word lookup table. The word embeddings are updated during training.

In addition to different ways of combining word-level and character-level representations, we also compare two different ways of integrating documents and queries: GA refers to the gated attention reader (Dhingra et al., 2016a) and FG refers to our fine-grained gating described in Section 3.3.

The results are reported in Table 2 . We report the results on common noun (CN) questions and named entity (NE) questions, which are two widely used question categories in CBT. Our finegrained gating approach achieves new state-of-the-art performance on both settings and outperforms the current state-of-the-art results by up to 1.76% without using ensembles. Our method outperforms the baseline GA reader by up to 2.4%, which indicates the effectiveness of the fine-grained gating mechanism. Consistent with the results on the Twitter dataset, using word-character fine-grained gating can substantially improve the performance over concatenation or scalar gating. Furthermore, we can see that document-query fine-grained gating also contributes significantly to the final results.

We also apply our fine-grained gating model to the Who Did What (WDW) dataset (Onishi et al., 2016) . As shown in Table 3 , our model achieves state-of-the-art results compared to strong baselines. We fix the word embeddings during training.

4.2.2. SQUAD:

The Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset collected recently (Rajpurkar et al., 2016) We report our results in Table 4 . "Exact match" computes the ratio of questions that are answered correctly by strict string comparison, and the F1 score is computed on the token level. We can observe that both word-character fine-grained gating and document-query fine-grained gating can substantially improve the performance, leading to state-of-the-art results among published papers. Note that at the time of submission, the best score on the leaderboard is 0.716 in exact match and 0.804 in F1 without published papers. A gap exists because our architecture described in Section 3.1 does not specifically model the answer span structure that is unique to SQuAD. In this work, we focus on this general architecture to study the effectiveness of fine-grained gating mechanisms.

4.3. VISUALIZATION AND ANALYSIS:

We visualize the model parameter W g as described in Section 3.2. For each feature, we average the corresponding weight vector in W g . The results are described in Figure 3 . We can see that named entities like "Organization" and noun phrases (with tags "NNP" or "NNPS") tend to use characterlevel representations, which is consistent with human intuition because those tokens are usually infrequent or display rich morphologies. Also, DOCLEN-4, WH-adverb ("WRB"), and conjunction ("IN" and "CC") tokens tend to use word-level representations because they appear frequently.

We also sample random span of text from the SQuAD dataset, and visualize the average gate values in Figure 4 . The results are consistent with our observations in Figure 3 . Rare tokens, noun phrases, and named entities tend to use character-level representations, while others tend to use word-level representations. To further justify this argument, we also list the tokens with highest and lowest gate values in Table 5 . 

5. CONCLUSIONS:

We present a fine-grained gating mechanism that dynamically combines word-level and characterlevel representations based on word properties. Experiments on the Twitter tag prediction dataset show that fine-grained gating substantially outperforms scalar gating and concatenation. Our method also improves the performance on reading comprehension and achieves new state-of-the-art results on CBT and WDW. In our future work, we plan to to apply the fine-grained gating mechanism for combining other levels of representations, such as phrases and sentences. It will also be intriguing to integrate NER and POS networks and learn the token representation in an end-to-end manner.

:

Code is available at https://github.com/kimiyoung/fg-gating

Figures and Tables:

Figure 1: Word-character fine-grained gating. The two lookup tables are shared. "NER", "POS", "frequency" refer to named entity tags, part-of-speech tags, document frequency features.

Figure 2: Paragraph-question fine-grained gating.

Figure 3: Visualization of the weight matrix Wg. Weights for each features are averaged. Red means high and yellow means low. High weight values favor character-level representations, and low weight values favor wordlevel representations. "Organization", "'Person", "Location", and "O" are named entity tags; "DOCLEN-n" are document frequency features (larger n means higher frequency, n from 0 to 4); others are POS tags.

Figure 4: Visualization of gate values in the text. Red means high and yellow means low. High gate values favor character-level representations, and low gate values favor word-level representations.

Performance on the Twitter dataset. "word" and "char" means using word-level and character-level representations respectively.

Table content: <table/>

Performance on the CBT dataset. The "GA word char concat" results are extracted fromDhingra   et al. (2016a). Our results on fine-grained gating are based on a single model. "CN" and "NE" are two widely used question categories. "dev" means development set, and "test" means test set.

Table content: <table><tr><td/><td>0.241</td><td>0.428</td><td>133</td><td/></tr><tr><td>char (Dhingra et al., 2016b)</td><td>0.284</td><td>0.485</td><td>104</td><td/></tr><tr><td>word char concat</td><td>0.2961</td><td>0.4959</td><td colspan="2">105.8</td></tr><tr><td>word char feat concat</td><td>0.2951</td><td>0.4974</td><td colspan="2">106.2</td></tr><tr><td>scalar gate</td><td>0.2974</td><td>0.4982</td><td colspan="2">104.2</td></tr><tr><td>fine-grained gate</td><td>0.3069</td><td>0.5119</td><td colspan="2">101.5</td></tr><tr><td>Model</td><td colspan="4">CN dev CN test NE dev NE test</td></tr><tr><td>GA word char concat</td><td>0.731</td><td>0.696</td><td>0.768</td><td>0.725</td></tr><tr><td>GA word char feat concat</td><td>0.7250</td><td colspan="3">0.6928 0.7815 0.7256</td></tr><tr><td>GA scalar gate</td><td>0.7240</td><td colspan="3">0.6908 0.7810 0.7260</td></tr><tr><td>GA fine-grained gate</td><td>0.7425</td><td colspan="3">0.7084 0.7890 0.7464</td></tr><tr><td>FG fine-grained gate</td><td>0.7530</td><td colspan="3">0.7204 0.7910 0.7496</td></tr><tr><td>Sordoni et al. (2016)</td><td>0.721</td><td>0.692</td><td>0.752</td><td>0.686</td></tr><tr><td>Trischler et al. (2016)</td><td>0.715</td><td>0.674</td><td>0.753</td><td>0.697</td></tr><tr><td>Cui et al. (2016)</td><td>0.722</td><td>0.694</td><td>0.778</td><td>0.720</td></tr><tr><td>Munkhdalai &amp; Yu (2016)</td><td>0.743</td><td>0.719</td><td>0.782</td><td>0.732</td></tr><tr><td>Kadlec et al. (2016) ensemble</td><td>0.711</td><td>0.689</td><td>0.762</td><td>0.710</td></tr><tr><td>Sordoni et al. (2016) ensemble</td><td>0.741</td><td>0.710</td><td>0.769</td><td>0.720</td></tr><tr><td colspan="2">Trischler et al. (2016) ensemble 0.736</td><td>0.706</td><td>0.766</td><td>0.718</td></tr><tr><td colspan="2">Section 3.2; scalar gate uses a scalar gate similar to</td><td/><td/><td/></tr></table>

Performance on the Who Did What dataset. "dev" means development set, and "test" means test set. "WDW-R" is the relaxed version of WDW.

Table content: <table><tr><td>Model</td><td colspan="4">WDW dev WDW test WDW-R dev WDW-R test</td></tr><tr><td>Kadlec et al. (2016)</td><td>-</td><td>0.570</td><td>-</td><td>0.590</td></tr><tr><td>Chen et al. (2016)</td><td>-</td><td>0.640</td><td>-</td><td>0.650</td></tr><tr><td colspan="2">Munkhdalai &amp; Yu (2016) 0.665</td><td>0.662</td><td>0.670</td><td>0.667</td></tr><tr><td>Dhingra et al. (2016a)</td><td>0.716</td><td>0.712</td><td>0.726</td><td>0.726</td></tr><tr><td>this paper</td><td>0.723</td><td>0.717</td><td>0.731</td><td>0.726</td></tr></table>

Performance on the SQuAD dev set. Test set results are included in the brackets.

Table content: <table><tr><td>Model</td><td>F1</td><td>Exact Match</td></tr><tr><td>GA word</td><td>0.6695</td><td>0.5492</td></tr><tr><td>GA word char concat</td><td>0.6857</td><td>0.5639</td></tr><tr><td>GA word char feat concat</td><td>0.6904</td><td>0.5711</td></tr><tr><td>GA scalar gate</td><td>0.6850</td><td>0.5620</td></tr><tr><td>GA fine-grained gate</td><td>0.6983</td><td>0.5804</td></tr><tr><td>FG fine-grained gate</td><td>0.7125</td><td>0.5995</td></tr><tr><td colspan="3">FG fine-grained gate + ensemble 0.7341 (0.733) 0.6238 (0.625)</td></tr><tr><td>Yu et al. (2016)</td><td colspan="2">0.712 (0.710) 0.625 (0.625)</td></tr><tr><td>Wang &amp; Jiang (2016)</td><td colspan="2">0.700 (0.703) 0.591 (0.595)</td></tr><tr><td colspan="3">The Children's Book Test (CBT) dataset is built from children's books. The whole dataset has</td></tr><tr><td>669,343 questions for training, 8</td><td/><td/></tr></table>

Word tokens with highest and lowest gate values. High gate values favor character-level representations, and low gate values favor word-level representations. Gate values Word tokens Lowest or but But These these However however among Among that when When although Although because Because until many Many than though Though this This Since since date where Where have That and And Such such number so which by By how before Before with With between Between even Even if Highest Sweetgum Untersee Jianlong Floresta Chlorella Obersee PhT Doctorin Jumonville WFTS WTSP Boven Pharm Nederrijn Otrar Rhin Magicicada WBKB Tanzler KMBC WPLG Mainau Merwede RMJM Kleitman Scheur Bodensee Kromme Horenbout Vorderrhein Chlamydomonas Scantlebury Qingshui Funchess

Table content: <table/>

Peer review (iclr2017):
I think the problem here is well motivated, the approach is insightful and intuitive, and the results are convincing of the approach (although lacking in variety of applications). I like the fact that the authors use POS and NER in terms of an intermediate signal for the decision. Also they compare against a sufficient range of baselines to show the effectiveness of the proposed model. I am also convinced by the authors* answers to my question, I think there is sufficient evidence provided in the results to show the effectiveness of the inductive bias introduced by the fine-grained gating model.