Introduction
Recurrent neural networks (RNNs) live at the heart of many sequence modeling problems. In particular, the incorporation of gated additive recurrent connections is extremely powerful, leading to the pervasive adoption of models such as Gated Recurrent Units (GRU) BIBREF0 or Long Short-Term Memory (LSTM) BIBREF1 across many NLP applications BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . In these models, the key idea is that the gating functions control information flow and compositionality over time, deciding how much information to read/write across time steps. This not only serves as a protection against vanishing/exploding gradients but also enables greater relative ease in modeling long-range dependencies.
There are two common ways to increase the representation capability of RNNs. Firstly, the number of hidden dimensions could be increased. Secondly, recurrent layers could be stacked on top of each other in a hierarchical fashion BIBREF6 , with each layer's input being the output of the previous, enabling hierarchical features to be captured. Notably, the wide adoption of stacked architectures across many applications BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 signify the need for designing complex and expressive encoders. Unfortunately, these strategies may face limitations. For example, the former might run a risk of overfitting and/or hitting a wall in performance. On the other hand, the latter might be faced with the inherent difficulties of going deep such as vanishing gradients or difficulty in feature propagation across deep RNN layers BIBREF11 .
This paper proposes Recurrently Controlled Recurrent Networks (RCRN), a new recurrent architecture and a general purpose neural building block for sequence modeling. RCRNs are characterized by its usage of two key components - a recurrent controller cell and a listener cell. The controller cell controls the information flow and compositionality of the listener RNN. The key motivation behind RCRN is to provide expressive and powerful sequence encoding. However, unlike stacked architectures, all RNN layers operate jointly on the same hierarchical level, effectively avoiding the need to go deeper. Therefore, RCRNs provide a new alternate way of utilizing multiple RNN layers in conjunction by allowing one RNN to control another RNN. As such, our key aim in this work is to show that our proposed controller-listener architecture is a viable replacement for the widely adopted stacked recurrent architecture.
To demonstrate the effectiveness of our proposed RCRN model, we conduct extensive experiments on a plethora of diverse NLP tasks where sequence encoders such as LSTMs/GRUs are highly essential. These tasks include sentiment analysis (SST, IMDb, Amazon Reviews), question classification (TREC), entailment classification (SNLI, SciTail), answer selection (WikiQA, TrecQA) and reading comprehension (NarrativeQA). Experimental results show that RCRN outperforms BiLSTMs and multi-layered/stacked BiLSTMs on all 26 datasets, suggesting that RCRNs are viable replacements for the widely adopted stacked recurrent architectures. Additionally, RCRN achieves close to state-of-the-art performance on several datasets.
Related Work
RNN variants such as LSTMs and GRUs are ubiquitous and indispensible building blocks in many NLP applications such as question answering BIBREF12 , BIBREF9 , machine translation BIBREF2 , entailment classification BIBREF13 and sentiment analysis BIBREF14 , BIBREF15 . In recent years, many RNN variants have been proposed, ranging from multi-scale models BIBREF16 , BIBREF17 , BIBREF18 to tree-structured encoders BIBREF19 , BIBREF20 . Models that are targetted at improving the internals of the RNN cell have also been proposed BIBREF21 , BIBREF22 . Given the importance of sequence encoding in NLP, the design of effective RNN units for this purpose remains an active area of research.
Stacking RNN layers is the most common way to improve representation power. This has been used in many highly performant models ranging from speech recognition BIBREF7 to machine reading BIBREF9 . The BCN model BIBREF5 similarly uses multiple BiLSTM layers within their architecture. Models that use shortcut/residual connections in conjunctin with stacked RNN layers are also notable BIBREF11 , BIBREF14 , BIBREF10 , BIBREF23 .
Notably, a recent emerging trend is to model sequences without recurrence. This is primarily motivated by the fact that recurrence is an inherent prohibitor of parallelism. To this end, many works have explored the possibility of using attention as a replacement for recurrence. In particular, self-attention BIBREF24 has been a popular choice. This has sparked many innovations, including general purpose encoders such as DiSAN BIBREF25 and Block Bi-DiSAN BIBREF26 . The key idea in these works is to use multi-headed self-attention and positional encodings to model temporal information.
While attention-only models may come close in performance, some domains may still require the complex and expressive recurrent encoders. Moreover, we note that in BIBREF25 , BIBREF26 , the scores on multiple benchmarks (e.g., SST, TREC, SNLI, MultiNLI) do not outperform (or even approach) the state-of-the-art, most of which are models that still heavily rely on bidirectional LSTMs BIBREF27 , BIBREF20 , BIBREF5 , BIBREF10 . While self-attentive RNN-less encoders have recently been popular, our work moves in an orthogonal and possibly complementary direction, advocating a stronger RNN unit for sequence encoding instead. Nevertheless, it is also good to note that our RCRN model outperforms DiSAN in all our experiments.
Another line of work is also concerned with eliminating recurrence. SRUs (Simple Recurrent Units) BIBREF28 are recently proposed networks that remove the sequential dependencies in RNNs. SRUs can be considered a special case of Quasi-RNNs BIBREF29 , which performs incremental pooling using pre-learned convolutional gates. A recent work, Multi-range Reasoning Units (MRU) BIBREF30 follows the same paradigm, trading convolutional gates with features learned via expressive multi-granular reasoning. BIBREF31 proposed sentence-state LSTMs (S-LSTM) that exchanges incremental reading for a single global state.
Our work proposes a new way of enhancing the representation capability of RNNs without going deep. For the first time, we propose a controller-listener architecture that uses one recurrent unit to control another recurrent unit. Our proposed RCRN consistently outperforms stacked BiLSTMs and achieves state-of-the-art results on several datasets. We outperform above-mentioned competitors such as DiSAN, SRUs, stacked BiLSTMs and sentence-state LSTMs.
Recurrently Controlled Recurrent Networks (RCRN)
This section formally introduces the RCRN architecture. Our model is split into two main components - a controller cell and a listener cell. Figure FIGREF1 illustrates the model architecture.
Controller Cell
The goal of the controller cell is to learn gating functions in order to influence the target cell. In order to control the target cell, the controller cell constructs a forget gate and an output gate which are then used to influence the information flow of the listener cell. For each gate (output and forget), we use a separate RNN cell. As such, the controller cell comprises two cell states and an additional set of parameters. The equations of the controller cell are defined as follows: i1t = s(W1ixt + U1ih1t-1 + b1i) and i2t = s(W2ixt + U2ih2t-1 + b2i)
f1t = s(W1fxt + U1fh1t-1 + b1f) and f2t = s(W2fxt + U2fh2t-1 + b2f)
o1t = s(W1oxt + U1oh1t-1 + b1o) and o2t = s(W2oxt + U2oh2t-1 + b2o)
c1t = f1t c1t-1 + i1t (W1cxt + U1ch1t-1 + b1c)
c2t = f2t c2t-1 + i2t (W2cxt + U2ch2t-1 + b2c)
h1t = o1t (c1t) and h2t = o2t (c2t) where INLINEFORM0 is the input to the model at time step INLINEFORM1 . INLINEFORM2 are the parameters of the model where INLINEFORM3 and INLINEFORM4 . INLINEFORM5 is the sigmoid function and INLINEFORM6 is the tanh nonlinearity. INLINEFORM7 is the Hadamard product. The controller RNN has two cell states denoted as INLINEFORM8 and INLINEFORM9 respectively. INLINEFORM10 are the outputs of the unidirectional controller cell at time step INLINEFORM11 . Next, we consider a bidirectional adaptation of the controller cell. Let Equations ( SECREF2 - SECREF2 ) be represented by the function INLINEFORM12 , the bidirectional adaptation is represented as: h1t,h2t = CT(h1t-1, h2t-1, xt) t=1,
h1t,h2t = CT(h1t+1, h2t+1, xt) t=M, 1
h1t = [h1t; h1t] and h2t = [h2t; h2t] The outputs of the bidirectional controller cell are INLINEFORM0 for time step INLINEFORM1 . These hidden outputs act as gates for the listener cell.
Listener Cell
The listener cell is another recurrent cell. The final output of the RCRN is generated by the listener cell which is being influenced by the controller cell. First, the listener cell uses a base recurrent model to process the sequence input. The equations of this base recurrent model are defined as follows: i3t = s(W3ixt + U3ih3t-1 + b3i)
f3t = s(W3fxt + U3fh3t-1 + b3f)
o3t = s(W3oxt + U3oh3t-1 + b3o)
c3t = f3t c3t-1 + i3t (W3cxt + U3ch3t-1 + b3c)
h3t = o3t (c3t) Similarly, a bidirectional adaptation is used, obtaining INLINEFORM0 . Next, using INLINEFORM1 (outputs of the controller cell), we define another recurrent operation as follows: c4t = s(h1t) c4t-1 + (1-s(h1t)) h3t
h4t = h2t c3t where INLINEFORM0 and INLINEFORM1 are the cell and hidden states at time step INLINEFORM2 . INLINEFORM3 are the parameters of the listener cell where INLINEFORM4 . Note that INLINEFORM5 and INLINEFORM6 are the outputs of the controller cell. In this formulation, INLINEFORM7 acts as the forget gate for the listener cell. Likewise INLINEFORM8 acts as the output gate for the listener.
Overall RCRN Architecture, Variants and Implementation
Intuitively, the overall architecture of the RCRN model can be explained as follows: Firstly, the controller cell can be thought of as two BiRNN models which hidden states are used as the forget and output gates for another recurrent model, i.e., the listener. The listener uses a single BiRNN model for sequence encoding and then allows this representation to be altered by listening to the controller. An alternative interpretation to our model architecture is that it is essentially a `recurrent-over-recurrent' model. Clearly, the formulation we have used above uses BiLSTMs as the atomic building block for RCRN. Hence, we note that it is also possible to have a simplified variant of RCRN that uses GRUs as the atomic block which we found to have performed slightly better on certain datasets.
For efficiency purposes, we use the cuDNN optimized version of the base recurrent unit (LSTMs/GRUs). Additionally, note that the final recurrent cell (Equation ( SECREF3 )) can be subject to cuda-level optimization following simple recurrent units (SRU) BIBREF28 . The key idea is that this operation can be performed along the dimension axis, enabling greater parallelization on the GPU. For the sake of brevity, we refer interested readers to BIBREF28 . Note that this form of cuda-level optimization was also performed in the Quasi-RNN model BIBREF29 , which effectively subsumes the SRU model.
Note that a single RCRN model is equivalent to a stacked BiLSTM of 3 layers. This is clear when we consider how two controller BiRNNs are used to control a single listener BiRNN. As such, for our experiments, when considering only the encoder and keeping all other components constant, 3L-BiLSTM has equal parameters to RCRN while RCRN and 3L-BiLSTM are approximately three times larger than BiLSTM.
Experiments
This section discusses the overall empirical evaluation of our proposed RCRN model.
Tasks and Datasets
In order to verify the effectiveness of our proposed RCRN architecture, we conduct extensive experiments across several tasks in the NLP domain.
Sentiment analysis is a text classification problem in which the goal is to determine the polarity of a given sentence/document. We conduct experiments on both sentence and document level. More concretely, we use 16 Amazon review datasets from BIBREF32 , the well-established Stanford Sentiment TreeBank (SST-5/SST-2) BIBREF33 and the IMDb Sentiment dataset BIBREF34 . All tasks are binary classification tasks with the exception of SST-5. The metric is the accuracy score.
The goal of this task is to classify questions into fine-grained categories such as number or location. We use the TREC question classification dataset BIBREF35 . The metric is the accuracy score.
This is a well-established and popular task in the field of natural language understanding and inference. Given two sentences INLINEFORM0 and INLINEFORM1 , the goal is to determine if INLINEFORM2 entails or contradicts INLINEFORM3 . We use two popular benchmark datasets, i.e., the Stanford Natural Language Inference (SNLI) corpus BIBREF36 , and SciTail (Science Entailment) BIBREF37 datasets. This is a pairwise classsification problem in which the metric is also the accuracy score.
This is a standard problem in information retrieval and learning-to-rank. Given a question, the task at hand is to rank candidate answers. We use the popular WikiQA BIBREF38 and TrecQA BIBREF39 datasets. For TrecQA, we use the cleaned setting as denoted by BIBREF40 . The evaluation metrics are the MAP (Mean Average Precision) and Mean Reciprocal Rank (MRR) ranking metrics.
This task involves reading documents and answering questions about these documents. We use the recent NarrativeQA BIBREF41 dataset which involves reasoning and answering questions over story summaries. We follow the original paper and report scores on BLEU-1, BLEU-4, Meteor and Rouge-L.
Task-Specific Model Architectures and Implementation Details
In this section, we describe the task-specific model architectures for each task.
This architecture is used for all text classification tasks (sentiment analysis and question classification datasets). We use 300D GloVe BIBREF42 vectors with 600D CoVe BIBREF5 vectors as pretrained embedding vectors. An optional character-level word representation is also added (constructed with a standard BiGRU model). The output of the embedding layer is passed into the RCRN model directly without using any projection layer. Word embeddings are not updated during training. Given the hidden output states of the INLINEFORM0 dimensional RCRN cell, we take the concatenation of the max, mean and min pooling of all hidden states to form the final feature vector. This feature vector is passed into a single dense layer with ReLU activations of INLINEFORM1 dimensions. The output of this layer is then passed into a softmax layer for classification. This model optimizes the cross entropy loss. We train this model using Adam BIBREF43 and learning rate is tuned amongst INLINEFORM2 .
This architecture is used for entailment tasks. This is a pairwise classification models with two input sequences. Similar to the singleton classsification model, we utilize the identical input encoder (GloVe, CoVE and character RNN) but include an additional part-of-speech (POS tag) embedding. We pass the input representation into a two layer highway network BIBREF44 of 300 hidden dimensions before passing into the RCRN encoder. The feature representation of INLINEFORM0 and INLINEFORM1 is the concatentation of the max and mean pooling of the RCRN hidden outputs. To compare INLINEFORM2 and INLINEFORM3 , we pass INLINEFORM4 into a two layer highway network. This output is then passed into a softmax layer for classification. We train this model using Adam and learning rate is tuned amongst INLINEFORM5 . We mainly focus on the encoder-only setting which does not allow cross sentence attention. This is a commonly tested setting on the SNLI dataset.
This architecture is used for the ranking tasks (i.e., answer selection). We use the model architecture from Attentive Pooling BiLSTMs (AP-BiLSTM) BIBREF45 as our base and swap the RNN encoder with our RCRN encoder. The dimensionality is set to 200. The similarity scoring function is the cosine similarity and the objective function is the pairwise hinge loss with a margin of INLINEFORM0 . We use negative sampling of INLINEFORM1 to train our model. We train our model using Adadelta BIBREF46 with a learning rate of INLINEFORM2 .
We use R-NET BIBREF9 as the base model. Since R-NET uses three Bidirectional GRU layers as the encoder, we replaced this stacked BiGRU layer with RCRN. For fairness, we use the GRU variant of RCRN instead. The dimensionality of the encoder is set to 75. We train both models using Adam with a learning rate of INLINEFORM0 .
For all datasets, we include an additional ablative baselines, swapping the RCRN with (1) a standard BiLSTM model and (2) a stacked BiLSTM of 3 layers (3L-BiLSTM). This is to fairly observe the impact of different encoder models based on the same overall model framework.
Overall Results
This section discusses the overall results of our experiments.
On the 16 review datasets (Table TABREF22 ) from BIBREF32 , BIBREF31 , our proposed RCRN architecture achieves the highest score on all 16 datasets, outperforming the existing state-of-the-art model - sentence state LSTMs (SLSTM) BIBREF31 . The macro average performance gain over BiLSTMs ( INLINEFORM0 ) and Stacked (2 X BiLSTM) ( INLINEFORM1 ) is also notable. On the same architecture, our RCRN outperforms ablative baselines BiLSTM by INLINEFORM2 and 3L-BiLSTM by INLINEFORM3 on average across 16 datasets.
Results on SST-5 (Table TABREF22 ) and SST-2 (Table TABREF22 ) are also promising. More concretely, our RCRN architecture achieves state-of-the-art results on SST-5 and SST-2. RCRN also outperforms many strong baselines such as DiSAN BIBREF25 , a self-attentive model and Bi-Attentive classification network (BCN) BIBREF5 that also use CoVe vectors. On SST-2, strong baselines such as Neural Semantic Encoders BIBREF53 and similarly the BCN model are also outperformed by our RCRN model.
Finally, on the IMDb sentiment classification dataset (Table TABREF25 ), RCRN achieved INLINEFORM0 accuracy. Our proposed RCRN outperforms Residual BiLSTMs BIBREF14 , 4-layered Quasi Recurrent Neural Networks (QRNN) BIBREF29 and the BCN model which can be considered to be very competitive baselines. RCRN also outperforms ablative baselines BiLSTM ( INLINEFORM1 ) and 3L-BiLSTM ( INLINEFORM2 ).
Our results on the TREC question classification dataset (Table TABREF25 ) is also promising. RCRN achieved a state-of-the-art score of INLINEFORM0 on this dataset. A notable baseline is the Densely Connected BiLSTM BIBREF23 , a deep residual stacked BiLSTM model which RCRN outperforms ( INLINEFORM1 ). Our model also outperforms BCN (+0.4%) and SRU ( INLINEFORM2 ). Our ablative BiLSTM baselines achieve reasonably high score, posssibly due to CoVe Embeddings. However, our RCRN can further increase the performance score.
Results on entailment classification are also optimistic. On SNLI (Table TABREF26 ), RCRN achieves INLINEFORM0 accuracy, which is competitive to Gumbel LSTM. However, RCRN outperforms a wide range of baselines, including self-attention based models as multi-head BIBREF24 and DiSAN BIBREF25 . There is also performance gain of INLINEFORM1 over Bi-SRU even though our model does not use attention at all. RCRN also outperforms shortcut stacked encoders, which use a series of BiLSTM connected by shortcut layers. Post review, as per reviewer request, we experimented with adding cross sentence attention, in particular adding the attention of BIBREF61 on 3L-BiLSTM and RCRN. We found that they performed comparably (both at INLINEFORM2 ). We did not have resources to experiment further even though intuitively incorporating different/newer variants of attention BIBREF65 , BIBREF63 , BIBREF13 and/or ELMo BIBREF50 can definitely raise the score further. However, we hypothesize that cross sentence attention forces less reliance on the encoder. Therefore stacked BiLSTMs and RCRNs perform similarly.
The results on SciTail similarly show that RCRN is more effective than BiLSTM ( INLINEFORM0 ). Moreover, RCRN outperforms several baselines in BIBREF37 including models that use cross sentence attention such as DecompAtt BIBREF61 and ESIM BIBREF13 . However, it still falls short to recent state-of-the-art models such as OpenAI's Generative Pretrained Transformer BIBREF64 .
Results on the answer selection (Table TABREF26 ) task show that RCRN leads to considerable improvements on both WikiQA and TrecQA datasets. We investigate two settings. The first, we reimplement AP-BiLSTM and swap the BiLSTM for RCRN encoders. Secondly, we completely remove all attention layers from both models to test the ability of the standalone encoder. Without attention, RCRN gives an improvement of INLINEFORM0 on both datasets. With attentive pooling, RCRN maintains a INLINEFORM1 improvement in terms of MAP score. However, the gains on MRR are greater ( INLINEFORM2 ). Notably, AP-RCRN model outperforms the official results reported in BIBREF45 . Overall, we observe that RCRN is much stronger than BiLSTMs and 3L-BiLSTMs on this task.
Results (Table TABREF26 ) show that enhancing R-NET with RCRN can lead to considerable improvements. This leads to an improvement of INLINEFORM0 on all four metrics. Note that our model only uses a single layered RCRN while R-NET uses 3 layered BiGRUs. This empirical evidence might suggest that RCRN is a better way to utilize multiple recurrent layers.
Across all 26 datasets, RCRN outperforms not only standard BiLSTMs but also 3L-BiLSTMs which have approximately equal parameterization. 3L-BiLSTMs were overall better than BiLSTMs but lose out on a minority of datasets. RCRN outperforms a wide range of competitive baselines such as DiSAN, Bi-SRUs, BCN and LSTM-CNN, etc. We achieve (close to) state-of-the-art performance on SST, TREC question classification and 16 Amazon review datasets.
Runtime Analysis
This section aims to get a benchmark on model performance with respect to model efficiency. In order to do that, we benchmark RCRN along with BiLSTMs and 3 layered BiLSTMs (with and without cuDNN optimization) on different sequence lengths (i.e., INLINEFORM0 ). We use the IMDb sentiment task. We use the same standard hardware (a single Nvidia GTX1070 card) and an identical overarching model architecture. The dimensionality of the model is set to 200 with a fixed batch size of 32. Finally, we also benchmark a CUDA optimized adaptation of RCRN which has been described earlier (Section SECREF4 ).
Table TABREF32 reports training/inference times of all benchmarked models. The fastest model is naturally the 1 layer BiLSTM (cuDNN). Intuitively, the speed of RCRN should be roughly equivalent to using 3 BiLSTMs. Surprisingly, we found that the cuda optimized RCRN performs consistently slightly faster than the 3 layer BiLSTM (cuDNN). At the very least, RCRN provides comparable efficiency to using stacked BiLSTM and empirically we show that there is nothing to lose in this aspect. However, we note that cuda-level optimizations have to be performed. Finally, the non-cuDNN optimized BiLSTM and stacked BiLSTMs are also provided for reference.
Conclusion and Future Directions
We proposed Recurrently Controlled Recurrent Networks (RCRN), a new recurrent architecture and encoder for a myriad of NLP tasks. RCRN operates in a novel controller-listener architecture which uses RNNs to learn the gating functions of another RNN. We apply RCRN to a potpourri of NLP tasks and achieve promising/highly competitive results on all tasks and 26 benchmark datasets. Overall findings suggest that our controller-listener architecture is more effective than stacking RNN layers. Moreover, RCRN remains equally (or slightly more) efficient compared to stacked RNNs of approximately equal parameterization. There are several potential interesting directions for further investigating RCRNs. Firstly, investigating RCRNs controlling other RCRNs and secondly, investigating RCRNs in other domains where recurrent models are also prevalent for sequence modeling. The source code of our model can be found at https://github.com/vanzytay/NIPS2018_RCRN.
Acknowledgements
We thank the anonymous reviewers and area chair from NIPS 2018 for their constructive and high quality feedback.

Question: By how much do they outperform BiLSTMs in Sentiment Analysis?
Answer: Proposed RCRN outperforms ablative baselines BiLSTM by +2.9% and 3L-BiLSTM by +1.1% on average across 16 datasets.

Question: Does their model have more parameters than other models?
Answer: approximately equal parameterization