Gated Recurrent Neural Network with Tensor ProductAndros Tjandra∗1∗1

Sakriani Sakti∗1
Ruli Manurung∗2
Satoshi Nakamura∗1∗2

Nara Institute of Science and Technology, JapanMirna Adriani∗2

Universitas Indonesia, Indonesia

In the machine learning fields, Recurrent Neural Network (RNNs) has become a primary choice for modelingsequential data such as text, speech, etc. To deal with long-term dependency in the long sequence, RNN utlizes
gating mechanism to improve the gradient flow between multiple time-steps and avoid exploding/vanishing gradientproblem. In the other hand, we would like to improve the representation power from RNN by using more expressive
operation compared to standard matrix multiplication and summation. In this paper, we proposed a new RNN
architecture with gating mechanism and tensor product between an input layer, a previous hidden layer, and a 3-rd
rank tensor weight and we called it as gated recurrent neural tensor network (GRURNTN).

1.

Introduction

where σ(·) is a sigmoid activation function, rt , zt are resetand update gates, h˜t is the candidate hidden layer values
and ht is the hidden layer values at time-t. In spite of having
one fewer gating layer, the GRU can match LSTM’s performance and its convergence speed convergence sometimes
outperformed LSTM.
In our proposed architecture, the tensor product operation is applied between the current input and previous
hidden layer multiplied by the reset gates for calculating
the current candidate hidden layer values. The calculation is parameterized by tensor weight. To construct aGRURNTN, we defined the formulation as:

Gating mechanism helps recurrent neural network (RNN)
to model long-term dependency between each input timestep. By using gating mechanism, RNN model able to avoidvanishing or exploding gradient [1]. On the other side, tohelp RNN model learn more complex dataset or task, weneed to improve the expressiveness from our model. To get
more powerful representation on hidden layer, Pascanu etal. [3] modified RNNs with an additional nonlinear layerfrom input to the hidden. Socher et al. [2] proposes another approach to model direct interaction between two input layers with tensor products on recursive neural network(RecNN). By using tensor product, we increase our RNN
model expressiveness by using second-degree polynomial interactions, compared to first-degree polynomial interactions
on standard dot product followed by addition in common
RNNs architecture. In this paper we proposed a new RNN
architecture that combine the gating mechanism and tensor
product concepts to incorporate both advantages in a single architecture, denotes as gated recurrent neural tensor
network (GRURNTN).

2.

h˜tσ(xt Wxr + ht−1 Whr + br )(1)zt
h˜t=σ(xt Wxz + ht−1 Whz + br )(2)

=f (xt Wxh + (rt(3)ht=(1 − zt )(4)(ri[1:d]ht−1 ) Wtsr

"xt(r ht−1 )ht−1 )Whh + bh )

#(5)

[1:d]

A gated recurrent unit (GRU) is a gated RNN with similar properties to a long short term memory (LSTM). However, there are several differences: a GRU does not haveseparated memory cells and instead of three gating layers,
it only has two gating layers: reset gates and update gates.
The GRU hidden layer at time t is defined by the following
equations:=hxt

where Wtsr ∈ R(i+d)×(i+d)×d is a tensor weight for mapping the tensor product between the input-hidden layer, iis the input layer size, and d is the hidden layer size. Alternatively, in this paper we use a simpler bilinear form forcalculating h˜t :h˜t

rtf+xt Wxh + (rtProposed Approachht−1 )Whh + bh )ht−1 + zt h˜t==fhh i[1:d]xt Wtsr (rt+xt Wxh + (rt[i:d]ht−1 )i|ht−1 )Whh + bh )

(6)
[i]

where Wtsr ∈ Ri×d×d is a tensor weight. Each slice Wtsr isa matrix Ri×d . The advantage of this asymmetric version
is that we can still maintain the interaction between theinput and hidden layers through a bilinear form. We reduce
the number of parameters from the original neural tensor
network formulation by using this asymmetric version. Fig.
1 visualizes the h˜t calculation in more detail.

3.

Experiment

We used a PennTreeBank (PTB) corpus, which is a standard benchmark corpus for statistical language modeling.
A PTB corpus is a subset of the Wall Street Journal (WSJ)
corpus. In this experiment, we followed the standard preprocessing step that was done by previous research [4].

Contact: Andros Tjandra, Nara Institute of Science and Technology,
8916-5 Takayama-cho,Ikoma, Nara 630-0192, JAPAN, 0743-72-5265,
andros.tjandra.ai6@is.naist.jp

1

Table 1: PennTreeBank test set BPC
ModelTest BPC
HF-MRNN [4]1.41
sRNN [3]1.41
DOT(S)-RNN [3]1.39
GRURNN (our baseline)
1.39
GRURNTN (proposed)
1.33

Table 2: PennTreeBank test set PPL
Model
Test PPL
N-Gram [4]141
RNNLM [4]124.7sRNN [3]110.0
DOT(S)-RNN [3]107.5
GRURNN (our baseline)
97.78
GRURNTN (proposed)
87.38

Figure 1: Calculating candidate hidden layer h˜t from current input xt and previous hidden layer multiplied by resetgate r · ht1 based on Eq. 6.
We used the preprocessed PTB corpus from the RNNLMtoolkit website ∗1 .
We did two different language modeling tasks. First,
we experimented on a word-level language model and weused perplexity (PPL) to measure our RNN performancefor word-level language modeling
1P P L = 2− N

PN

i=1 log2 P (Xi |X1..i−1 ).

outperformed both baseline models. GRURNTN reduced
the BPC from 1.39 to 1.33 (0.06 absolute / 4.32% relativeBPC) from the baseline GRURNN. Overall, GRURNTN
outperformed our baseline model and other published models on the character-level language modeling task.
Table 2 shows the PPL on PTB test set performed byour baseline model, proposed model, and several publishedresults. GRURNTN reduced the perplexity from 97.78 to87.38 (10.4 absolute / 10.63% relative PPL) over the baseline GRURNN. Overall, GRURNN outperformed our baseline models as well as the other models significantly.

(7)

Second, we experimented on a character-level language
model and we used the average number of bits-per-character(BPC) to measure our RNN performance for character-level
language modeling1
BP C = −N

3.1

NX!
log2 p(Xi |X1..i−1 ) .

(8)

5.

i=1Conclusion

We presented a new RNN architecture by combining thegating mechanism and tensor product concepts. From ourexperiment on the PTB corpus, our proposed models outperformed the baseline models with a similar number ofparameters in character-level language modeling and wordlevel language modeling tasks. In a character-level language
modeling task, GRURNTN obtained 0.06 absolute BPC reduction over GRURNN (4.32% relative). In a word-level
language modeling task, GRURNTN obtained 10.4 absolute PPL (10.63% relative) reduction over GRURNN.

Experiment Models

In this experiment, we compared the performancefrom our baseline models GRURNN with our proposedGRURNTN. For the word-level language modeling task,
we used 256 hidden units for GRURNTN and 860 for
GRURNN. All of these models use 128 dimensions forword embedding. The total number of free parameters
for GRURNN and GRURNTN were about 12 million. For
the character-level language modeling task, we used 256
hidden units for GRURNTN and 820 for GRURNN. Allof these models used 32 dimensions for character embedding. The total number of free parameters for GRURNN
and GRURNTN was about 2.2 million. We constrained ourbaseline GRURNN to have a similar number of parameters
as the GRURNTN model for a fair comparison.

6.

Acknowledgment

Part of this work was supported by JSPS KAKENHI Grant Numbers JP17H06101 and JP17K00237

References
4.

Result[1] Hochreiter, Sepp et al. Gradient flow in recurrent nets: thedifficulty of learning long-term dependencies (2001).

In this section, we report our experiment results on PTB
character-level and word-level language modeling using ourbaseline models GRURNN and our proposed GRURNTN.
Table 1 shows BPC on PTB test set performed by our baseline model, our proposed model and several published results. Our proposed model GRURNTN and LSTMRNTN[2] Socher, Richard et al. Reasoning with neural tensor networks
for knowledge base completion. Advances in Neural Information
Processing Systems (2013): 926-934[3] Pascanu, Razvan et al. How to construct deep recurrent neuralnetworks. arXiv preprint arXiv:1312.6026 (2013)[4] Mikolov, Tom. Statistical language models based on neural networks (2012)∗1 http://www.rnnlm.org/2