Interpretable Network Structure for Modeling Contextual Dependency
Authors: Xindian Ma
Peng Zhang
Xiaoliu Mao
Yehua Zhang
Nan Duan
Yuexian Hou
Ming Zhou.
Venue:  (2020)
Keywords: Language Model
Recurrent Neural Network
Separation Rank

Abstract: Neural language models have achieved great success in many NLP tasks, to a large extent, due to the ability to capture contextual dependencies among terms in a text. While many efforts have been devoted to empirically explain the connection between the network hyperparameters and the ability to represent the contextual dependency, the theoretical analysis is relatively insufficient. Inspired by the recent research on the use of tensor space to explain the neural network architecture, we explore the interpretable mechanism for neural language models. Specifically, we define the concept of separation rank in the language modeling process, in order to theoretically measure the degree of contextual dependencies in a sentence. Then, we show that the lower bound of such a separation rank can reveal the quantitative relation between the network structure (e.g. depth/width) and the modeling ability for the contextual dependency. Especially, increasing the depth of the neural network can be more effective to improve the ability of modeling contextual dependency. Therefore, it is important to design an adaptive network to compute the adaptive depth in a task. Inspired by Adaptive Computation Time (ACT), we design an adaptive recurrent network based on the separation rank to model contextual dependency. Experiments on various NLP tasks have verified the proposed theoretical analysis. We also test our adaptive recurrent neural network in the sentence classification task, and the experiments show that it can achieve better results than the traditional bidirectional LSTM.

Under review as a conference paper at ICLR 2020
INTERPRETABLE NETWORK STRUCTURE FOR MODEL- ING CONTEXTUAL DEPENDENCY
Anonymous authors Paper under double-blind review
ABSTRACT
Neural language models have achieved great success in many NLP tasks, to a large extent, due to the ability to capture contextual dependencies among terms tex- t. While many efforts have been devoted to empirically explain the connection between the network hyperparameters and the ability represent the contextual dependency, the theoretical analysis is relatively insufficient. Inspired by the re- cent research on the use tensor space to explain the neural network architecture, we explore the interpretable mechanism for neural language models. Specifical- ly, we define the concept of separation rank in the language modeling process, in order theoretically measure the degree of contextual dependencies in sen- tence. Then, we show that the lower bound of such a separation rank can reveal the quantitative relation between the network structure (e.g. depth/width) and the modeling ability for the contextual dependency. Especially, increasing the depth of the neural network can be more effective improve the ability of modeling contextual dependency. Therefore, it is important to design an adaptive network to compute the adaptive depth in a task. Inspired by Adaptive Computation Time (ACT), we design an adaptive recurrent network based the separation rank to model contextual dependency. Experiments on various NLP tasks have veri- fied the proposed theoretical analysis. We also test our adaptive recurrent neural network in the sentence classification task, and the experiments show that can achieve better results than the traditional bidirectional LSTM.
INTRODUCTION
Language modeling is fundamental topic in natural language processing (NLP), and neural net- work based language models have achieved great success in many NLP tasks (Cheng et al. 2016 Ko al. 2017; Peters et al. 201 8a b). For instance, ELMo (Peters et al. 2018b), as a contextualized word representation model, significantly improves the results in a broad range of NLP problems. One important reason is that the recurrent network and its variants can capture the long- ranggrrng//shortt range contextual dependencies among terms in a text (Graves) 2013) Yang et al. [017 Peters et al.) trying empirically explain the relation between network hyperparameters (such as the number of layers) and the ability to represent contextual dependencies (Karpathy et al.,
201 Researchers
2015 Peters al. 2018a). However, the theoretical explanations are relatively insufficient.
Recently, Peters et al. (2018a) investigated the effect of network depth on the range of contextual dependeneies represented in neural language models. Empirical evidences show that the bottom layer can capture short-range dependencies, while deep layers are more capable to represent long- range dependencies.
observed that different NLP tasks require different layers of traited embeddings in (Peters et al. 201 8a). Specifically, for the coreference resolution task requiring global syntax structures, long-range dependencies among words should be modeled, and thus embeddings vectors in deep layers are more useful than those in bottom layers. On the other hand, for the named entity recognition task, it requires relatively local syntax to be modeled, and then the embeddings in bottom layers can work well (Peters et al. 2018a). However, while the empirical observations are insightful, it is challenging to theoretically derive the quantitative relation between the neural network structure and the representation ability for contextual dependency.
Our work is inspired by these studies on the relation between neural network and tensor network (Co- hen & Shashua 2016a Levine et al. 2017). These studies have shown that the correlation between

---

Under review as a conference paper at ICLR 2020
imput regions of image be calculated by the separation rank. Based on such measure- ment and its analysis, the structure of convolutional neural network can be explained in a tensor space (Cohen & Shashua, 2016a). Levine et al.] (2017) further utilize the separation rank to mea- sure the long-term memory capacity and analyze its relation to the network depth, by mapping the recurrent neural network to tensor network.
In language modeling tasks, tensors have been used for text representation and language model- ing (Liu et al. 2005; Socher al. 2013 Pennington al. 2014 Qiu & Huang. 2015). Recently, the tensor network (as method to model high-order tensors) has been adopted in language model- ing, and a tensor space language model (TSLM) has been proposed (Zhang et al. 2019). TSLM was proved to be a generalization of the n n-gram and RNN language models. TSLM bridges the gap be- tween the tensor network and language modeling, and then paves the way for the further theoretical analysis of neural language modeling in tensor space.
In this paper, we aim to provide quantitative analyses for explaining the mechanisms of modeling contextual dependencies in recurrent neural network (RNN) and its variants (e.g.. LSTM, EMLo). We first define the concept separation rank in the language model process, and show that this separation rank can measure the dependencies in sentence. We further derive the lower bound of the separation rank in the tensor space language model, which a combination number of network depth and network width, etc.
We then provide a lower bound analysis for the separation rank defined in language modeling. shows that along with the increasing number of network layers L (representing the network depth), the lower-bound can be increased by an exponential level, while increasing the number of hid- den units R (representing the network width) can improve the lower-bound by linear level. The lower-bound is associated with the separation rank, which measures the degree of the contextual dependencies.
The method of Adaptive Computation Time (ACT) (Graves) 2016) allows recurrent neural networks learn how many computational steps (i.e.. the depth) to takes on an input. However, this method implies an assumption that the states and outputs are approximately linear, and it lacks a theoretical interpretation. The effect of depth on the ability of neural network to model contextual dependencies is more direct, as mentioned above (Levine et al. (217]] Zia & Razzaq (018)). For a natural language processing task, it is a challenge task to get an adaptive neural network structure (i.e., the number of layers). Inspired by this work (Graves) 2016), we design a bidirectional adaptive recurrent neural network based on the guidance of the separation rank in language model.
In order to verify our thcoretical analysis, we carry out various NLP tasks that reflect different degrees/ranges of contextual dependencies. The experimental results can explain the effect of net- work depth (represented by network layers) on the contextual dependencies modeled in the neural language models. In addition to the network depth, our experiments show that increasing the net- work width (i.e., the number of the hidden units) can also increase the modeling ability for both long-range and short-range contextual dependencies. Moreover, increasing the network depth can be more effective than increasing the network width, the tasks requiring long-range dependency. Furthermore, in order to search a minimum layers and effective modeling contextual dependency, we train the new adaptive neural network on the sentence classification task. The results show that our model achieves better results than fixed depth recurrent neural network (i.e., BiLSTM).
2 PRELIMINARIES
We first briefly introduce the basic language model, as well as the recently developed tensor space language model (TSLM) (Zhang et al. 2019), which is proved to be generalized approach to both n n--rrm and recurrent language models.
2.1 BASIC LANGUAGE MODEL
Given sentence of words denoted as (W1, W2,..., w,), language model computes the joint probability of a sentence. Its joint probability can be written as:
p(w() p(w1...., w.).
2

---

Under review as a conference paper at ICLR 2020
N N-gram language model computes the joint probability of the sentence by calculating the probability of a word w, given the history (Wi...., W]
2.2 TENSOR SPACE LANGUAGE MODEL
Recently, a Tensor Space Language Model (TSLM) (Zhang al. 2019) was proposed, which can consider all the combinatorial dependencies among words through the tensor product. Tensor prod- uct is fundamental operator in tensor analysis, denoted by &. which can map two low-order tensors to high-order tensor. It is proved that TSLM is a generalization of both n-gram (when the word vectors are represented with one-hot vectors) and a recurrent language modeling process (when the word vectors are represented with low-dimensional dense word vector).
In TSLM, for a sentence {ww..... ...;. Wz with length n, each word w, can be represented by m-dimension word vector w; (i E (1,....,m}) m)). The basic formulation of TSLM can be written as:
p(wf)=
Tdi..dd Ad
where Ad,....d are the entries from tensor A which is essentially rank-one tensor, i.e., A w1 @ ... & Wn. Td.....d. are entries from tensor T, which is a higher rank tensor. can en- code the network parameters, c.g.. hidden layers matrix and states matrix, after TSLM is reduced to a recurrent language modeling architecture.
SEPARATION RANK AS MEASUREMENT FOR CONTEXTUAL DEPENDENCY
In this section, our aim is define the separation rank in TSLM. In Sec. 3.1 we first describe the basic ideas for the contextual dependency and expose the variables which are related to the contextual dependencies. Then, we define the separation rank tensor space language model in For better readability, we put detailed proofs in supplementary appendices.
Sec 3.2
3.1
BASIC IDEAS FOR CONTEXTUAL DEPENDENCY
The separation rank measures how far a function is from being separable (Cohen & Shashua 2016b). p(w")) in Eq. I can be considered as the probability density function, given sentence [....., Wr), where is the number of words. Without loss of generality, we can split sentence into disjoint context subsets, Si (Wi,...y w:) and S2 I,..,,..., W,). where i is a position for separating a sentence to S₁ and Sz. If S1 and S₂ is independent. The function of joint probability Eq. can be written as:
p(w() ((w)))()))))
For considering the situation when S₁ and S2 are not independent, we give the function as follows:
p(w)) Σ plu, )(w By (,)((w;+1l)))
where yj is the conditional variable, y4 E and (= yK When K equal Eq. can be reduced to Eq. 3 since p(y1) 1.
In language modeling, K is equal to then p(wF)=p((())) (p(())))( (((w.,) which means that there is no dependency between two context pieces S₁ and Sz. The higher K is, the further two context pieces (S, and S2) are from being separable, meaning that S and S2 has stronger dependency.
3.2 SEPARATION RANK IN TSLM
In this section, we define the separation rank based on K in TSLM. First, consider p(wT) ) in
Eq. 2 as probability function. Then, the separation rank is defined on the factorization of p(w()) ) Such factorization is carried on the matricization of weight tensor T in Eq. 2 The matricization process is to put the entries from a tensor into a matrix. It turns out that the factorization of p(wf)) is reduced to the factorization of of the matrix form of the tensor T, shown in the following claim.
3

---

Under review as a conference paper at ICLR 2020
Claim The factorization of p(w?) in Eq. can be obtained by the Singular Value Decomposition (SVD) on the matrix (81)5))))) ( After the decomposition. the matrix [T]( is written as follows:
K
(T) S2)
(5)
where is number of non-zero singular values.
Proof Our proof can be found in Supplementary Appendices A.I submitted.
In Eq. each different position i corresponds to different partitions (2) and consequent ma- tricizations of tensor T. Thus there could be different decomposition and different numbers of non-zero singular values. order to find measurement for contextual dependency, we define the separation rank as the minimum value of K in TSLM. The claim is given as follows.
Claim Suppose a sentence (=(wi...., wn) can be split into two disjoint subset Si=(wi,... w;) and Sz=(w:+1,.. Wn), where E (1...., 1} is any position. The separation rank of function in Eq. in tensor space language model is formulated as:
sep(s).s)(p(u)))) minfK E E p((pp(w))))))))
Σ ply.))(wwll
(6)
Proof 2 Our detailed proof can be found in Supplementary Appendices A.2 submitted.
In gen separation rank in TSLM se)( have been defined in Eq.
4 LOWER BOUND ANALYSIS OF SEPARATION RANK FOR RECURRENT NEURAL NETWORK STRUCTURE
In this section, by providing the lower bound of the separation rank in TSLM, we investigate the interpretable mechanism in recurrent neural network language modeling. Specifically, we analyze the relation between the separation rank (reflecting the contextual complexity) and the network architecture (c.g., network depth and width). Recall that tensor space language model (TSLM) derive recurrent language modeling process from via the tensor decomposition of high-order tensors (Zhang et al.) 2019). Based on the above analysis, we will establish the relation between separation rank in TSLM (see 3.2) and parameters of recurrent neural language models.
Different partition position in Eq. 4 corresponds to different partition (S1,S2) in Eq. 6 If the partitions (S,, are different, the values of K will be different. Without loss of generality, we set an equal partition, i.c., i = [ When a sentence is partitioned equally, K value in Eq, 6 is larger than the actual separation rank since it the minimal K.
Then, based on the equal partition criteria, step by step, we propose our theoretical results regarding the lower bound of the separation rank. Such lower bound is expected reflect the relation between the ability for modeling contextual dependencies and the model structure (e.g., network depth and width) the language modeling process. Specifically, if a sentence is modeled by recurrent network architecture, the lower bound of the separation rank in TSLM can be computed by some parameters (i.e., hidden units, the number of layers and the number of words in a sentence) in recurrent network architecture. In the following, number of layers is one (i.e., L= = 1), and then we provide our findings when L >
first show such quantitative result when the
Claim In tensor space language model (TSLM), when the number of layer is one (i.e., L=1,) the separation rank in TSLM can be given as follows:
sep(s,.Sgi (+)) 2R R. for -=
where the number of hidden units, (S1, S2) is the case of equal partition.
Proof The detailed proof can be found in Supplementary Appendices A.3 submitted.
'The separation rank not be accuracy computed in experiments.
4

---

Under review as a conference paper at ICLR 2020
Ag-5
hg
Forward
Backward
Figure I: The diagram about a bidirectional adaptive RNN. a) is single direction adaptive RNN, b) is schematic diagram of bidirectional RNN.
In order to show the influence of network depth, we need to further extend the tensor space language model using recurrent network with deep layers, i.e., > 1. We derive the lower bound, i.e., the separation rank in TSLM in case of the L 2. Further details will be expound in Supplementary Appendices A.4. The lower bound will be shown as follows:
(R F-11 1)! F!(R - 1)! mlm1)
se (p( 2 min(
[n/2]] R the number of the hidden units, n is the number words a sentence, m is the dimension of word vectors, I. is the number of layers and (.)! is the symbol of factorial.
where F= (22
(S1, S2; = [)]) is the case equal partition.
Next, we explore the relation between the lower bound and network parameters in detail. From Eq. [8] we can find that increasing both R and can all improve lower bound of separation rank in TSLM, i... the separation rank of recurrent network language model will be increased. In order to illustrate this quantitative relationship between separation rank in TSLM and the recurrent network architecture (i.e., R and L). In Eq. g we see that if L was increased, it is observed that the lower bound will be increased by an exponential level. Through increasing R, the lower bound is increased by linear level. Taable[in supplementary appendices shows the numerical relationship.
5 BIDIRECTIONAL ADAPTIVE RECURRENT NEURAL NETWORK
The effectiveness of neural network depth is an important research problem /Vinayakumar et al., 2017]. In Sec the analysis of lower bound has described that increasing the number layer can be more effective to improve the ability of modeling contextual dependency. In the processing for modeling sequence words, is important to search an adaptive network depth. Under most cireumstances, researches choose a fixed number of network layers L to model text. In this section, we describe how to construct an adaptive bidirectional recurrent neural network.
5.1 STRUCTURE CONSTRUCTION
backward recurrent neural network language model the right function p(w"- After that, forward recurrent neural network network language model R, can be seen as the left function p(wj (3)). In experiment, we first train the backward function and get the hidden output. When it predicts the next words in forward language model, the hidden vectors from the backward process are used to multiply the hidden vectors from the forward process. In Figure I we provide the diagram for the bidirectional adaptive RNN.
As in Eq. 6
In Figure I a),, 20-1 and we are the inputs of word vectors. S.-2. S,-1 and S, are some vectors from the outputs of hidden layer. Eq. shows how compute the S..
S, = Σ S
(9)
5

---

Under review as a conference paper at ICLR 2020
Narned Eniliy Recogniion (NER)
POS leoging
Lensy Parsing
024
010
0-
14-0
040
164
3rd layer 1STH 2nd layer LSTH cal ager
3rd laper L5T LSTM
315 lyer LSTV
0..
2nd byer
2nd byee ITM lacleyer
220 lay
the number of nadden un
the number of hidden unit
the number of oden unis
Word Sonse Disamb IWSD)
Semantic classifier
Coreference Reselution
048
610
040
0.0
2nd layer LSTM 1st ayer LSTN
2nd Layer LETM
Jst Uyper 15TH
644
2nd LSTH
Lst layer LSTM
20
10 19 1o 20
100 10. 16 de do 1se
130
10
number
de on za 25
Figure 2: Results on the short-range (first row) and long-range (second row) dependencies tasks. The displayed results show that the increasing of R is more helpful in short-range dependencies tasks (NER, Constituency Parsing and POS tagging), while the increasing of L is more effective for long-range dependencies tasks (WSD, Semantic classifier and Coreference Resolution), compared with the increasing of R.
where pf is a scalar value and presents the probability value of S The probability value can be computed as: pf of St. where vs is the trainable vector and be initialized the start of training. pł is equal to the inner product of v. and S
5.2 DYNAMIC HALTING
Since we wish to explore a recurrent neural network with the minimum computational cost. Here, first give out
basic assumptions. Since Eq. T6 to get the minimal K, and increasing the depth L can improve the ability of modeling contextual dependency by exponential level in Eq. 8 we can assume the computing of the depth L have straight relation. Inspired by Adaptive Computation Time to dynamically halt changes in certain representations (Graves, 2016). we design the condition of halting as follows. There are two kinds of halting conditional in our model. First, the times of computing is raised to the maximum number of calculations. Second, the steps of computing is satisfied ((I)=min({: Eirt
EXPERIMENTS
The about analysis shown that improving R and I. can improve the lower bound at different levels. The lower bound can imply the ability of recurrent neural network language models for modeling the contextual dependencies. For different NLP tasks, we need to choose the network structure with different modeling capabilities. We carry out six classification tasks and an intra- sentence similarity experiment to verify our theoretical, respectively.
6.1 NATURAL LANGUAGE PROCESSING TASKS
This set experiments involves six tasks including Named Entity Recognition (NER), POS tagging, Constituency parsing, Word Sense Disambiguation (WSD), Sentiment analysis and Coreference Resolution. According to previous study (Peters et al, 2018a), NER, constituency parsing and POS tagging required to model short-range dependency, while WSD, Semantic classifier and Coreference Resolution are required to model long-range dependency.
fe generally taken as 0.001 experiments.

---

Under review as a conference paper at ICLR 2020
Table 1: Results sentences classification dataset. Accuracy (ace) is used as the evaluation criteri- on, represents the results achieved by ourself.
Model
MR
SST2
CR
SUBJ
MPQA
FastText (Joulin et al.
2016)
0.765
0.788
0.789
0.916
0.874
Vent2Vec (Gupta ct al. 2019)
0.763
0.802
0.791
0,912
0.872
Transformer ( Vaaswan el al.
2017
0.731
0.761
0.762
0.863
0.723
BiLSTM (Conneau el al.
2017)
0.775
0.807
0.813
0.896
0.887
Adaptive-
MSTM LSTM
0.784
0.816*
0.821
0,92
0.898
Adaptive-BiLas BiLSTM
0.790
0.818*
0.820*
0,917
0.904*
Based on our analysis in the previous section, we have two hypotheses to be tested. (1) For tasks with short-range dependencies required, improving R some extent meet the modeling requirement, while improving the network depth L could be not necessary; (2) For tasks requiring long-range dependencies, is necessary improve L to improve the ability of modeling dependencies.
These tasks be considered as classification tasks, for which the experimental results are described in Sec. 6.1.2 In language modeling process, LSTM commonly used recurrent network structure. Therefore, we mainly carried out experiments based on the structure of LSTM and its variants.
6.1.1 TESTING TASKS
POS tagging We conduct experiments on the Wall Street Journal of the Penn Treebank dataset (PTB) (Marcus et al., 1993). The model[Lng et al. (2013) is used, which using bidirectional LSTMs. Named Entity Recognition The CoNLL 2003 NER task (Tjong Kim Sang & De Meulder, (003) consists of newswire from the Reuters RCVI corpus tagged with four different entity types. We follow the language model augmented sequence taggers (TagLM) (Peters al., 2017).
Constituency parsing The Penn Treebank (Marcus et al., 1993) which contains phrase structure annotation is used in this application. We use the model named Reconciled Span Parser, which used the biLM representation.
Word Sense Disambiguation We follow this model (Kágebäck & Salomonsson, 2016) which used the bidirectional LSTMs architecture to compute a probability distribution over the possible senses corresponding to that word. The dataset of Senserval-2 (Edmonds & Cotton 2001 is used.
Coreference Resolution An end-to-end coreference resolution model (Lee et al. (017)) has been proposed. We conduct experiments on the OntoNotes Release 5.0 benchmark (Pradhan et al. 2012).
Sentiment analysis We use the IMDB dataset (Maas et al. [2011) with two categories. We perform classification using a standard bidirectional LSTM with different hidden units and layers.
The experimental setup of the six testing tasks can be found in Supplementary Appendices
A.5.1
6.2.2 EXPERIMENTAL RESULTS AND ANALYSIS
The results six NLP tasks reported in Figure 2 The results of short-range dependency and long-range dependency tasks are shown as the first row and second row, respectively.
In short-range dependency tasks, using I-layer LSTM, its scores increase along the increasing of the number of hidden units Rt There are similar experimental phenomenons when using 2-layer and 3-layer LSTMs. For each R, after L be increased, we observe that increasing F; score is not very obvious. These phenomena mean that for the increasing of Fi, the increasing of R is more obvious than the increasing of L. Therefore, the results in the first row of Figure verify our first hypothesis about short-range dependent tasks.
In the second row of Figure 2 these trends of line charts show that the increasing for the number of hidden units R and the number of layers L can help to increase the Fi score, while the effects of increasing is more obvious, for long-range dependency tasks. The Fi scores are the highest through using 3-layer LSTMs in long-range dependency tasks. In order to investigate different effects of increasing R and L on Fi scores, we have detailed observations: When L is the largest (saying 3), the increasing of F1 score is getting relatively slow along with the increasing of R, as shown in the results of semantic classifier (the middle in the second row in (2). These phenomena and trends verify the second hypothesis. It is that increasing J. can improve the network modeling
7

---

Under review as a conference paper at ICLR 2020
ability more effectively than increasing R for long-range dependency tasks, In summary, R. is more necessary to increased for the short-range dependency tasks, while L is relatively more necessary to be increased in the long-range dependency tasks. Such experimental results and analysis are consistent with our theoretical analysis Sec and testing hypotheses in this section.
6.2 INTRA- SIMILARITY TASK
In this section, we present a detailed empirical study about the effect of different network architec- ture (i.e., network width and the number of layers) the representation ability of contextual word embeddings. Based on previous analysis in Sec. [ our hypothese are as follows. (1) Increasing R improve the similarity between short-range words. (2) Increasing L can improve the similarity between long-range words, (nccrrsggg L is more effective for modeling the similarity between words than the increasing of R.
1-Billion Word Benchmark dataset Chelba et al. (2013) is used, and it has approximately 800M tokens of news crawl data from WMT 2011. Different recurrent network architectures (i.e., using different layers and the number of hidden units) are trained in Lo,,o, which uses bidirectional LSTM structure. The training of F. To model will be stopped when the average perplexity is 40 in test dataset. After that, we select sentence from the test data and compute the intra-sentence similarity. We use cosine similarity, which is popular way to measure the similarity between word vectors encoded in different layers of EM Lo. Figure 3 in Supplementary Appendices shows the intra-sentence contextual similarity among words in a sentence. The details of the experimental results and analysis will be discussed in Supplementary Appendices A.5.2
6.3 BIDIRECTIONAL ADAPTIVE RECURRENT NEURAL NETWORK
In this section, we test the bidirectional adaptive recurrent neural network on five text classification tasks. Text classification task refer the process of automatically determining the text category based on the textual content under a given classification system. The Precision is used to evaluate the quality of the model. We test our model on MR, SST-2, CR, MPQA and TREC, respectively. The experiment results of this part is shown in Table [
Table [ shows the results about the adaptive network in sentence classification tasks. Results shows that the Adaptive-BB SilpttvveBiiSTMM BiLSTM model achieves better accuracy (acc) in these datasets than BiLSTM. Our method achieves larger improvement on the MPQA dataset. The adaptive bi-LSTM (bidirectional LSTM) and the model of adaptive LSTM achieve very close results.
7 CONCLUSION AND FURTHER WORK
In this paper, we provide quantitative analysis for modeling contextual dependencies in recurrent language model and its variants. First, we define the separation rank tensor space language model to measure the contextual dependencies of a sentence. Then, we analyze the lower bound of the separation rank in the recurrent neural network based LM. Based on the lower-bound analysis, we explain the quantitative relation between the value of separation rank and the network structure, For certain NLP tasks requiring different ranges of contextual dependencies, the lower-bound analysis will suggest different settings for the network depth and width. These suggestions based on the quantitative lower-bound analysis, are verified by various NLP tasks. order find the number of layers with both minimum cost and great performance results, we designed adaptive RNN and verify the effectiveness the method on text classification tasks.
In the future work, we design feasible methods to calculate the contextual dependencies, and use such signals to design automatic language modeling approaches, which can be adaptively controlled for specifie NLP tasks with certain requirement for the contextual dependencies. In addition, will explore the use of tensor space to explain or advance the more recent language modeling approach, e.g., Transformer and BERT.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv: 13Xiv:1312..0 1312.3002 2013.
Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 551-561, 2016. URL org/
http::/ http:///wa......rrrrrataoooooo// http///oo////////////i----////-/
Nadav Cohen and Amnon Shashua. Inductive bias of deep convolutional networks through pooling geometry. CoRR, abs/1605.06743, 201 URL http:::/arxx...orr//bb///////// 0677
Nadav Cohen and Amnon Shashua. Inductive bias of deep convolutional networks through pooling geometry. arXiv preprint arXiv:/605.067433 2016b.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Super- vised learning of universal sentence representations from natural language inference data, arXiv preprint arXiv: 170..0220..06444 2017.
Philip Edmonds and Scott Cotton. Senseval-2: overview. In The Proceedings of the Second Inter- national Workshop on Evaluating Word Sense Disambiguation Systems, pp. 1-5, Association for Computational Linguistics, 2001.
Alex Graves. Generating sequences with recurrent neural networks. CoRR, abs/1308.0850 2013.
URL wtt.httpp//axxx.ooottt aasrg///b//338... 0850
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arX-
0w:1603.08883 2016.
Prakhar Gupta, Matteo Pagliardini, and Martin Jaggi. Better word embeddings by disentangling contextual n-gram information. In NAACL-HLT (!), pp. 939.939, Association for Computational Linguistics, 2019.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov, Armand Joulin, Edouard Grave, Piotr Bojanowski. Tomas Mikolov, Armand Joulin, and Edouard Grave. Bag of tricks for efficient text classification. 2016.
Mikael Kågebäck and Hans Salomonsson. Word sense disambiguation using a bidirectional Istm. In Proceedings of the 5th Workshop on Cognitive Aspects of the Lexicon (CogALex- V), pp. 51-56, 2016.
Andrej Karpathy, Justin Johnson, and Fei-Fei Li. Visualizing and understanding recurrent networks. CoRR, abs/ 506.0778,, 2015.
Wei-Jen Ko, Bo-Hsiang Tseng, and Hung-yi Lee. Recurrent neural network based language mod- eling with controllable external memory. In 2017 IEEE International Conference on Acoustic- Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017, p- p. 5705-5709, 2017. doi: 10. 10.10//CASSP.2217795522, URL httpt:t/////m / http:::://doooooggge.
11C9/ICASSE 2017. 21017.9555 7953249
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. End-to-end neural coreference reso- lution. 2017.
Yoav Levine, Or Sharir, Alon Ziv, and Amnon Shashua. On the long-term memory of deep recurrent networks. 2017.
Wang Ling, Tiago Lufs, Luís Marujo, Ramn Fernandez Astudillo, and Isabel Trancoso. Finding function in form: Compositional character models for open vocabulary word representation. Com- puter Science, pp. 1899-1907, 2015.
Ning Liu, Benyu Zhang, Jun Yan, and Zheng Chen. Text representation: from vector to tensor. In IEEE International Conference Data Mining, pp. 725-728, 2005.

---

Under review as a conference paper at ICLR 2020
Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th anmual meeting of the association for computational finguistics: Human language technologies-volume p.p. 142-150. Association for Computational Linguistics, 2011.
Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building large annotated corpus of English: the penn treebank. 1993.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Conference on Empirical Methods in Natural Laguage Processing, pp. 1532- 1543, 2014.
Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. Semi-supervised se- quence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 1756- 1765, 2017.
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pp. 1499-1509, 2018a.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018b.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. Conll- 2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Confer- ence on Emmlp & Conll-shared Task, 2012.
Xipeng Qiu and Xuanjing Huang. Convolutional neural tensor network architecture for community- based question answering. In IJCAI, pp. 1305 1311, 2015.
Richard Socher, Danqi Chen, Christopher D. Manning, and Andrew Y. Ng. Reasoning with neural tensor networks for knowledge base completion. In International Conference on Neural Informa- tion Processing Systems, pp. 0364,,44, 2013.
Erik Tjong Kim Sang and Fien De Meulder. Introduction the conll-2003 shared task: Language- independent named entity recognition. In Proceedings of the seventh conference on Natural lan- guage learning at HLL-NAACL 3003-Volume 4, pp. 142-147. Association for Computational Lin- guistics, 2003.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural In- formation Processing Systems 30: Annual Conference on Neural Information Processing Sys- tems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 6000 2017. URL http: 7/papers nips. cctpaper//ppppccpepce 1tt_aaaatttntio--------------
R. Vinayakumar, K. P. Soman, and P. Poornachandran. Evaluating effectiveness of shallow and deep networks to intrusion detection system. In 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp. 1282-1289, Sep. 2017. doi: 10.
1109/CCACC1.2217..811 2.601888.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank RNN language model. CoRR, abs/1711 1es/171..0993, 2017. URL http:
//arxiv. org/ 0abbs//711.000
Lipeng Zhang, Peng Zhang, Xindian Ma, Shuqin Gu, Zhan Su, and Dawei Song. A generalized language model in tensor space. Proceedings of the 2019 Conference on AAAI Conference on Artificial Intelligence, 2019.
Tehseen Zia and Saad Razzaq. Residual recurrent highway networks for learning deep sequence prediction models. Journal of Grid Computing, pp. 1-8, 2018.
10

---

Under review as a conference paper at ICLR 2020
A SUPPLEMENTARY APPENDICES
A.1
CLAIM 1
The factorization of p(w]f) in Eq.2 (in paper) be obtained by the Singular Value Decomposi- tion (SVD) on the matrix (S.. After the decomposition, the matrix (8,(8) is written as follows:
K
(T.))
Σ Nu
(10)
/=1
where K is the number of non-zero singular values. Proof sketch of this claim
In the tensor space language model, the imput tensor rank-one tensor. To make easy to under- stand, we assume that the word vector is one hot vector. The high order tensor A can be given by the operator of tensor product. Therefore, the tensor A is one hot tensor, which means that the tensor only have one element is 1 and other elements are zero. The basic form of tensor space language can be re-written an element Tei Tdi....d.. from the tensor T. After that, the element can be consider a value in matrix which be given by the matricization of the tensor T. For all elements of the matrix, we can consider a method of SVD to get the value of K.
It would be helpful to understand if the definition of matricization given. Therefore, There is the definition of matricization as follows.
Definition Suppose T mR......m is tensor of order n, and let S2) be paratition of ser (I.....,.,, and S1 and S2 are disjoint subsets. The matricization of T denoted as [T] is the m by-mla) matrix holding the entries of Tdi....d.. is placed row index + Σ E
1 1) 1Sl-t and column index 1 E| (dj, 11mm where E S1 and S2.
Proof 4 In subsection 2.2, Eq.2 is the basic form of tensor space language model. In Eq.2, A is rank-one tensor A = wh ... & wa- We each word as an one-hot vector. The tensor A is tensor with only one entry being one and the other entries being zero. The Eq.2 can be rewritten as:
p(wf) =
Σ
Tai Adi..da
di....d.=1
where
( 1, d, = inder(we, V), Vk E [m]
Adi.. de
(12)
0, otherwise
which means tensor is one-hot tensor, and indez(w, V) means the index of in vocabulary V. le e (n]] also denoted as l E [I,...,n.. The probability of a sentence s is:
IVI
p(wi)=
Σ
Toi..dd Adi..dn
(13)
di d1,....,,;1 =1
Td......u. de = inder(we, (), V), E [n]
Therefore, the factorization of p(wf)) in Eq. 2(in paper) can be achieved by steps as follows.
Firstly, the matricization of weight tensor T can be achieved by the matricization definition 7 and represented as [T] (81.22) Secondly, we perform singular value decomposition (SVD) matrix [7]()),)))
[T](S,.)) USVT
(14)
Therefore, compute each element in the matrix [T] (S,,S,), in other words, get every entry from tensor T and represent as:
K Σ P=
=
N Un a. Voda
(15)

---

Under review as a conference paper at ICLR 2020
where Uv.da d; is the element from the matrix U and the Uv.di.....: is the element from the matrix V. Let do.d and Uv.di....dd can be given by p(wily,) lvv) and p( Wo), respectively. No is the singular value and seed as p(yo). We can represent Eq. as as Eq. (in paper).
A.2 CLAIM 2
Suppose sentence S (Wi:... (....... Wp) can be split into two disjoint subset S (W1,-.. ..... w.) and S2 = (WH+1.... ,,), the separation rank of function Eq. 2 by tensor space language model is formulated as following:
K
sep(s).s, = [ E N p(wf) plys))(
(16)
where K is the number of no-zero singular values, i is the position of separation.
Before proving the claim, we should give out a definition as reported (Cohen & Shashua, 2016b).
Definition Let (11 12) be a partition of ordered input indexes, i.e. 1 and In are disjoint subsets of [N] whose is the set of all indexes. I1 = "1,...: where it ... < dtg and similarly 12 --......-J where ji <... < jjn- For a function h (RRE) R. the separation rank w.r. the partition (11, is defined as follows:
sep(h; 12) min ( E NU (0} NU
Agi... ...... IR. 9 ... 9n s.t. h(11,..,, xN)
R
Σ ( Fin, (
is the minimal number of summands that together give h, where each summand is separable and equal to product of two function. 9: and 9 are the functions, (R*)))) and (I=(
Proof sketch of this claim
According to Eq. 15 and Definition 2 it will be easy prove the result that the separation rank between two different inputs S₁ and 52 is equal to the minimum value K for different separation.
Proof In Eq. 15 we the Vo.Uvd di is equal gu(wi..... wi) and Vo Vi.Vdid...dd is we can give the function of the separation
equal to g' (w w.). According to the definition 2
rank in tensor space language model.
.n ... (p)()) min (R N: 3gi.... ... .i..h....
s.t. p(w")=
(17)
Σ go (wi....... w.)g', (w.. (Wi+1.....)))
In a word, it is the minimal number of sunmands that together give pl w where each summand separable and equaltion to a product. After that, for different (2),S2) on a sentence, K is different. Therefore, we can define the minimum value of K as the separation rank in tensor space.
A.3 CLAIM 3
tensor space language model (TSLM), when number of layer one (i.e., L=1,) the separation rank in TSLM can be given as follows:
Se (p)(p(w;)) 2 R, for ==
(18)
where the number hidden units, (Si. S2) is the case equal partition, i.c., the position in Eq.4(in paper) is equal to 2
Proof sketch of this claim
We consider the RNN is function y. In addition, the function be re-written which is the sum of R groups factor functions product. R is the number of hidden units, and also the separation rank in RNNLM. It meets the definition of separation rank in definition
form
12

---

Under review as a conference paper at ICLR 2020
Proof 6 The recurrent network language model of the number of layers L=1 have been given:
h Whh-1 3 Uw.
(19)
Yt Vh.
where w is the current word inputs, he is the current hidden state, W; is the word vector of t-th word. l/ is probability distribution of the current state (i.e., the joint probability distribution). The s denotes the element-wise multiplication between vector (i.e., Hadamard product). W, are square matrix in TSLM and the dimension of W is the number of hidden units R. We set the W = I. is the full connection matrix. Further, we can compute current hidden state as follows:
h = Uw: O Wh:11
Uw, O (UU ① h(-2)
Uw, & -w 1 ② h1-2 (W =
(20)
+ Uwe ① ... ① Uw 3 h1/2
where set the b is the function g(wi,.... Wt/2) and the Uwe O... 2 Uw is the function g'(w (wa+i...... w.). After that, we can compute each probability the probability distribution y:. The process as follows:
Yt Vhg
R
(21)
(1) = Σ
Vegg, (w )g' (21
where i E 1,...,IV is the index in the vector yt and |V is the size vocabulary. matrix of R by [V]. At the same time, i means also the position of the last word of a sentence in the vocabulary. Therefore, the p(w?) can be written as:
p(w() (1)
Let be equal the inputs can be written (wi..... wt ( S₁ and (w/2 -1:-- -...... wn) Sccoriiggg to the definition of the separation separation rank in the definition 2 result as follow:
we can get the
seP(s,,s.)(p(uu)) 2 R
(22)
Therefore, the separation rank in TSLM is R when we using the 1-layer recurrent network.
A.4 SEPARATION RANK TENSOR NETWORK
Let yt be the function computing the output after & time-steps of recurrent tensor network with layers, hidden layer channel numbers, weights denoted by tensor T and initial hidden states ho. E [L]. The relations between start-end separation rank and the TN architecture (Levine al. 2017) are shown as follow:
seq (1")) = min mi T//3
L=1
seq(n (min(m, R} R3) )) T/2 T
L ==
where (minn(m.)))T/2 TT22) the multiset coefficient, given in the binomial form by 1)+11/2)-1) R is the hidden chamnel numbers, 7' is the length of sequence inputs, and [T/2]
m is the dimensions of each model high-order tensor. (Ti, Ta) is an average separation of time
steps T. For clearly understand the multi-set coefficient, We can understand that select ? types from H' min Rt, m) class elements. Now, a lower bound ((17/2) on the separation rank of depth I. have been provided. In the following, The conjecture that tighter lower bound
13

---

Under review as a conference paper at ICLR 2020
holds for networks of depth L the form of which implies that the dependency capacity of deep recurrent networks grows combinatorially with the network depth:
((CE
R
seq(y") 2 min!
m
m LN 2
(23)
where is the number of hidden units, sentence. In language model, for easy analysis the relation between separation rank and parameter from the recursive networks architecture. We set the R = m. We do not consider time series. Let's set the T in Eq. 23 to n. is the numbers of words in a sentence. Considering the constraints of tensor dimensions, the maximum of TSLM separation rank is mLT//2 . T is the temple in RNN, we suppose that it is the equal n, n is the length of sentence. the case where R and are controllable, the value to the left of the minimum function 23 is less than the value to the right. After that, we can get the relations as follow.
number of layers. n is the number of words
Se 12 (p(w))) 2 min! (R F 1)!
m
F!( - 1)!
(24)
([m/2] L 2)
where (L 1) [mmmm]]
where the R is the number of hidden size, n is the number of words in sentence, and L is the number of layers. Note that is the symbol of factorial.
A.5 EXPERIMENT
Table 2: The quantitative relation between the separation rank in TSLM and the parameters (i.e., R and L). Increasing L and I approximate the separation rank of recurrent network exponentially and linearly, respectively. Example: n (i.e., sentence length), R E {14, 15, 16, 17) and L E {1, 2, 43.
R
14
15
16
17
L
14
15
17
L
455
560
680
816
L
8008
12376
18564
27132
L 4 1.9+10
4,3+10*
9.2+10*
3,5+10"
A.5.1 EXPERIMENTAL SETUP
Sec. 4 analyzed the relation between contextual dependencies and recurrent network architec- ture through I, and R. Specifically, the lower bound of the recurrent language model model- ing the sentence contextual dependencies will be increased by increasing L (L E (1,2,3}) and R (R& {100, 125, 150, 175, 200, 225, 250, 300)) call that our hypotheses are: For short-range dependent tasks, increasing R is sufficient to reflect the network modeling ability, while for long-range dependent tasks, increasing can improve the network modeling ability more effectually than increasing R. We test our hypotheses through the trend of Fi score. F, a mainly and common indicator of evaluation in these task.
dependency tasks. Re-
A.5.2 ANALYSIS FOR INTRA-SENTIN
SIMILARITY TASK
Our experiments focus on the changing trends of the similarity between word embeddings along with the increasing of R and L. This experiment is different from the experiments reported by Peters et al. (0)8a)). They only consider the difference between upper layer and lower layer and do not consider the changing of network depth R.
Figure 3 shows the intra-sentence contextual similarity between all words a sentence. In Figure 3 obtain several observations. When the number of hidden units R is fixed, observe that the bright area is larger along with the increase of the number of layers L from 1 to 3. The bright area indicates the contextual dependency between words in work. Relatively speaking, the larger the
14

---

Under review as a conference paper at ICLR 2020
Layer
The Lagen 1-2
The layert
By
Stre
00/
Bey
Bay
the
Icdee
Hidden
Figure 3: Visualization of contextual similarity between words pairs single sentence. For the first row, we set the hidden layer units to be 256, with layers from 1 to 3. For the second row, we set the number of layers and the hidden layer units (256, 512 and 1024, respectively). The lighter yellow-colored areas indicate the higher contextual similarity.
Table 3: The statistical results about the average of the context similarity. Layers and hidden units are represented by L and respectively.
R(for L=
256
512
1024
average similarity
0.25
0.36
0.38
L(for R = 256)
average similarity
0.25
0.39
0.48
bright region is, the stronger the ability of a language model modeling the contextual dependency. When the number of layers is fixed, we observe that the bright area increased along with the increasing of the number of hidden units from 256 to 512. It may mean that short-range contextual dependency can be captured by adding the number of hidden units. These experimental phenomena verify the first hypothesis and the second hypothesis.
Correspondingly, we can also observe from Table that the larger the bright area is, the larger the average of contextual similarity is. In Table when the is equal 3 (when R=256), the average of contextual similarity is the largest. Combined Figure Bigure3aand and Table we can find that the increasing of the number of layers more effective to capture long-range similarity information. The results Table 3 and in Figure verify the third hypothesis.
15


Reviews:
[{"review_id": "Hylc0Nt8Yr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The goal of the work is to quantify the dependency between contents in NLP. The method relies on parametrization of the joint probability of words in (2), and discussed some connections between the rank of the unfolded tensor T and the dependency level between two sets of words in a sentence.\n\nThe goal of this work is quite interesting, but the reviewer feel a bit challenging to follow the writing. The work seems to build upon a previous work, namely, tensor space language model (TSLM). But the paper does not introduce TSLM in detail, making the part relevant to TSLM quite inaccessible.\n\nThe analysis of the work is based on the model in (2), which is merely an approximation for the joint probability. This is fine, but maybe this point should be more spelled out in the paper.\n\nThe work also has an assumption that a naive Bayes model in (4) always holds for a set of w_1 ... w_n. This may need some more discussion and maybe a reference. Since the authors are considering a sequence, this may be related to de finetti's theorem and its extensions. But in that theorem many more assumptions are needed, eg., the RVs are exchangeable. Also, it is unknown if a finite K exists.\n\nThe proof of Claim 1 is a bit trivial, if one only considers one-hot encoding. In fact, the statement and proof of Claim 1 might be a bit loose. If one wishes that the SVD reveals the rank K, K has to be smaller than the outer dimensions of the tensor T. This was not specified in the statement.\n\nThe above also brings up another question: are the proofs all based on one-hot encoding of words? We know in NLP pre-trained word embeddings may be more useful. Do the proofs also apply to those cases, e.g., GloVe or Word2Vec?\n\nDid all the experiments use one-hot embedding?\n\nOverall, the reviewer feels that the work has an interesting motivation, and the goal is meaningful. The writing is a bit hard to access and the proofs might be a bit loose (did not check all of them. But Claim 1 is already a bit loose).\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Bygw1CsiFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper first provides a theoretical interpretation of separation rank as a measure of a recurrent network's ability to capture contextual dependencies in natural language text. The analysis is primarily done in the context of tensor space language models (TSLM) that was demonstrated to be a generalisation of n-gram language models in earlier work. The theoretical derivations suggest that increasing a network's depth increases its separation rank exponentially, while increasing its width only increases its separation rank linearly. Based on this finding, the paper proposes a bidirectional adaptive recurrent neural network that adapts the network depth for each task using dynamic halting. Experiments on six NLP tasks demonstrate that increasing network depth helps more for tasks that generally require many long-range dependencies, while increasing network width is generally sufficient for tasks that require mostly short-term dependencies, although I disagree with the paper's dichotomy of tasks with long-term and short-term dependencies (see point 3 below).\n\nOverall, this paper suffers from several serious issues, as listed below. I am therefore recommending a rating of \"Reject\" ahead of the authors' response.\n\n1. This paper suffers from substantial clarity issues beyond simple grammatical errors. Some examples of the most serious clarity issues are as follows: (i) Section 6.1.1 lists \"sentiment analysis\" as one of the tasks, but Figure 2 has no entry for \"sentiment analysis\", yet instead features experimental results for \"semantic classifier\" which was never mentioned or introduced before; and (ii) Figure 2 shows \"1st layer LSTM\", \"2nd layer LSTM\", and \"3rd layer LSTM\", while (based on my reading) what the paper means are \"1-layer LSTM\", \"2-layer LSTM\", and \"3-layer LSTM\", hence highly confusing for the reader.\n\n2. The proposed explanation about the proposed bidirectional adaptive RNN is really sparse, despite being a central part of the paper. The explanation of the proposed model is only contained in two paragraphs (Sections 5.1 and 5.2), and are not sufficient for the readers to understand the model. A more extensive explanation and intuition about what the model is like, and how it is similar or different to TSLM and standard RNNs, is required to improve this.\n\n3. In the experiments, the paper assumes a false dichotomy of tasks that only require short-range dependencies (NER, POS tagging, and constituency parsing), and tasks that require long-range dependencies (WSD, sentiment analysis, and coreference resolution). This dichotomy is overly simplistic and ultimately false. For instance, constituency parsers often need to identify spans that are very long-distance in nature (for a recent investigation of this, see the work by Fried et al. (2019)), while sentiment analysis often only requires the model to identify a few salient words that are indicative of the sentiment, e.g. \"excellent\" or \"terrible\", hence not requiring much contextual dependencies. \n\n4. Related to point 3, a better evaluation is to examine the cases that require long-range dependencies within each task, rather than assuming which tasks require long-range dependencies and which ones do not. An example of this is reporting e.g. constituency parsing performance for long-range spans and coreference resolution performance for long-distance entity chains.\n\n5. The paper mostly fails to report performance comparison with existing numbers from prior work. For instance, the coreference performance (Fig. 2) are far below the result from Lee et al. (2017) that this paper is based on (Section 6.1.1), while the constituency parsing numbers (also Fig. 2) are also far below the reconciled span parser (Joshi et al., 2018) that this paper is also based on. This discrepancy calls into question the strength of the model implementation used in this paper.\n\n6. Some missing citations, e.g. the use of adaptive computation time and dynamic halting in Universal Transformers (Dehghani et al., 2019).\n\n7. This paper can benefit from careful copy-editing. Some examples of grammatical errors: (i) \"Eq. 16\" in Section 5.2 should be \"Eq. 6\", (ii) \"to verify our theoretical, respectively\" in Section 6 should be \"to verify our theoretical [findings/derivations]\", (iii) \"ourself\" on the caption of Table 1 should be \"ourselves\", (iv) \"The model Ling et al. (2015) is used, which using bidirectional LSTMs\" should be \"The model [of] Ling et al. (2015) is used, which [used] bidirectional LSTMs\", etc.\n\nReferences\nDaniel Fried, Nikita Kitaev, and Dan Klein. \"Cross-domain generalization of neural constituency parsers\". In Proc. of ACL 2019.\n\nKenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. \"End-to-end neural coreference resolution\". In Proc. of EMNLP 2017.\n\nVidur Joshi, Matthew E. Peters, and Mark Hopkins. \"Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples\". In Proc. of ACL 2018.\n\nMostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. \"Universal Transformers\". In Proc. of ICLR 2019.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJg5uXxkjB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper derives lower bounds on the separation rank of a wider class of recurrent NLP models in terms of its depth and number of hidden layers, demonstrating that both the number of hidden units as well as the number of layers improves the ability of NLP networks to model context dependency. It then introduces a novel bidirectional NLP variant that is supposed to capture a good trade-off between computational cost and performance.\n\nThe manuscript is very dense and does not follow a straight and easy-to-follow story line. In particular, the introduction of the bidirectional variant seems to substantially distract from the main story line of the paper (there is also no connection between the theoretical results to the bidirectional network). The improvements of the bidirectional models also seem to be minor, but no standard deviations for the performance results are reported.\n\nA clear description as to which language models are captured by the TSLM model is missing. Also, it is unclear how tight the bounds actually are given that no value for m (the word length) is given. Finally, the title does not reflect the content of the paper (there is nothing interpretable about the network structure).", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper a theoretical interpretation of separation rank as a measure of a recurrent network's ability to capture contextual dependencies in text, and introduces a novel bidirectional NLP variant and tests it on several NLP tasks to verify their analysis. 

Reviewer 3 found that the paper does not provide a clear description of the method and that a focus on single message would have worked better. Reviewer 2 made a claim of several shortcomings in the paper relating to lack of clarity, limited details on method, reliance on a 'false dichotomy', and failure to report performance. Reviewer 1 found the goals of the work to be interesting but that the paper was not clear, that the proofs were not rigorous enough, and clarity of experiments. The authors responded to the all the comments. The reviewers felt that their comments were still valid and did not adjust their ratings.

Overall, the paper is not yet ready in its current form. We hope that the authors will find valuable feedback for their ongoing research.
Decision: Reject