Trellis Networks for Sequence Modeling

1 INTRODUCTION . What is the best architecture for sequence modeling ? Recent research has produced significant progress on multiple fronts . Recurrent networks , such as LSTMs , continue to be optimized and extended ( Merity et al. , 2018b ; Melis et al. , 2018 ; Yang et al. , 2018 ; Trinh et al. , 2018 ) . Temporal convolutional networks have demonstrated impressive performance , particularly in modeling longrange context ( van den Oord et al. , 2016 ; Dauphin et al. , 2017 ; Bai et al. , 2018 ) . And architectures based on self-attention are gaining ground ( Vaswani et al. , 2017 ; Santoro et al. , 2018 ) . In this paper , we introduce a new architecture for sequence modeling , the Trellis Network . We aim to both improve empirical performance on sequence modeling benchmarks and shed light on the relationship between two existing model families : recurrent and convolutional networks . On the one hand , a trellis network is a special temporal convolutional network , distinguished by two unusual characteristics . First , the weights are tied across layers . That is , weights are shared not only by all time steps but also by all network layers , tying them into a regular trellis pattern . Second , the input is injected into all network layers . That is , the input at a given time-step is provided not only to the first layer , but directly to all layers in the network . So far , this may seem merely as a peculiar convolutional network for processing sequences , and not one that would be expected to perform particularly well . Yet on the other hand , we show that trellis networks generalize truncated recurrent networks ( recurrent networks with bounded memory horizon ) . The precise derivation of this connection is one of the key contributions of our work . It allows trellis networks to serve as bridge between recurrent and convolutional architectures , benefitting from algorithmic and architectural techniques developed in either context . We leverage these relationships to design high-performing trellis networks that absorb ideas from both architectural families . Beyond immediate empirical gains , these connections may serve as a step towards unification in sequence modeling . We evaluate trellis networks on challenging benchmarks , including word-level language modeling on the standard Penn Treebank ( PTB ) and the much larger WikiText-103 ( WT103 ) datasets ; character-level language modeling on Penn Treebank ; and standard stress tests ( e.g . sequential MNIST , permuted MNIST , etc . ) designed to evaluate long-term memory retention . On word-level 1https : //github.com/locuslab/trellisnet Penn Treebank , a trellis network outperforms by more than a unit of perplexity the recent architecture search work of Pham et al . ( 2018 ) , as well as the recent results of Melis et al . ( 2018 ) , which leveraged the Google Vizier service for exhaustive hyperparameter search . On character-level Penn Treebank , a trellis network outperforms the thorough optimization work of Merity et al . ( 2018a ) . On word-level WikiText-103 , a trellis network outperforms by 7.6 % in perplexity the contemporaneous self-attention-based Relational Memory Core ( Santoro et al. , 2018 ) , and by 11.5 % the work of Merity et al . ( 2018a ) . ( Concurrently with our work , Dai et al . ( 2019 ) employ a transformer and achieve even better results on WikiText-103 . ) On stress tests , trellis networks outperform recent results achieved by recurrent networks and self-attention ( Trinh et al. , 2018 ) . It is notable that the prior state of the art across these benchmarks was held by models with sometimes dramatic mutual differences . 2 BACKGROUND . Recurrent networks ( Elman , 1990 ; Werbos , 1990 ; Graves , 2012 ) , particularly with gated cells such as LSTMs ( Hochreiter & Schmidhuber , 1997 ) and GRUs ( Cho et al. , 2014 ) , are perhaps the most popular architecture for modeling temporal sequences . Recurrent architectures have been used to achieve breakthrough results in natural language processing and other domains ( Sutskever et al. , 2011 ; Graves , 2013 ; Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ; Vinyals et al. , 2015 ; Karpathy & Li , 2015 ) . Convolutional networks have also been widely used for sequence processing ( Waibel et al. , 1989 ; Collobert et al. , 2011 ) . Recent work indicates that convolutional networks are effective on a variety of sequence modeling tasks , particularly ones that demand long-range information propagation ( van den Oord et al. , 2016 ; Kalchbrenner et al. , 2016 ; Dauphin et al. , 2017 ; Gehring et al. , 2017 ; Bai et al. , 2018 ) . A third notable approach to sequence processing that has recently gained ground is based on self-attention ( Vaswani et al. , 2017 ; Santoro et al. , 2018 ; Chen et al. , 2018 ) . Our work is most closely related to the first two approaches . In particular , we establish a strong connection between recurrent and convolutional networks and introduce a model that serves as a bridge between the two . A related recent theoretical investigation showed that under a certain stability condition , recurrent networks can be well-approximated by feed-forward models ( Miller & Hardt , 2018 ) . There have been many combinations of convolutional and recurrent networks ( Sainath et al. , 2015 ) . For example , convolutional LSTMs combine convolutional and recurrent units ( Donahue et al. , 2015 ; Venugopalan et al. , 2015 ; Shi et al. , 2015 ) . Quasi-recurrent neural networks interleave convolutional and recurrent layers ( Bradbury et al. , 2017 ) . Techniques introduced for convolutional networks , such as dilation , have been applied to RNNs ( Chang et al. , 2017 ) . Our work establishes a deeper connection , deriving a direct mapping across the two architectural families and providing a structural bridge that can incorporate techniques from both sides . 3 SEQUENCE MODELING AND TRELLIS NETWORKS . Sequence modeling . Given an input x1 : T = x1 , . . . , xT with sequence length T , a sequence model is any function G : X T → YT such that y1 : T = y1 , . . . , yT = G ( x1 , . . . , xT ) , ( 1 ) where yt should only depend on x1 : t and not on xt+1 : T ( i.e . no leakage of information from the future ) . This causality constraint is essential for autoregressive modeling . In this section , we describe a new architecture for sequence modeling , referred to as a trellis network or TrellisNet . In particular , we provide an atomic view of TrellisNet , present its fundamental features , and highlight the relationship to convolutional networks . Section 4 will then elaborate on the relationship of trellis networks to convolutional and recurrent models . Notation . We use x1 : T = ( x1 , . . . , xT ) to denote a length-T input sequence , where vector xt ∈ Rp is the input at time step t. Thus x1 : T ∈ RT×p . We use z ( i ) t ∈ Rq to represent the hidden unit at time t in layer i of the network . We use Conv1D ( x ; W ) to denote a 1D convolution with a kernel W applied to input x = x1 : T . A basic trellis network . At the most basic level , a feature vector z ( i+1 ) t+1 at time step t+ 1 and level i+ 1 of TrellisNet is computed via three steps , illustrated in Figure 1a : 1 . The hidden input comprises the hidden outputs z ( i ) t , z ( i ) t+1 ∈ Rq from the previous layer i , as well as an injection of the input vectors xt , xt+1 . At level 0 , we initialize to z ( 0 ) t = 0 . 2 . A pre-activation output ẑ ( i+1 ) t+1 ∈ Rr is produced by a feed-forward linear transformation : ẑ ( i+1 ) t+1 =W1 [ xt z ( i ) t ] +W2 [ xt+1 z ( i ) t+1 ] , ( 2 ) where W1 , W2 ∈ Rr× ( p+q ) are weights , and r is the size of the pre-activation output ẑ ( i+1 ) t+1 . ( Here and throughout the paper , all linear transformations can include additive biases . We omit these for clarity . ) 3 . The output z ( i+1 ) t+1 is produced by a nonlinear activation function f : Rr × Rq → Rq applied to the pre-activation output ẑ ( i+1 ) t+1 and the output z ( i ) t from the previous layer . More formally , z ( i+1 ) t+1 = f ( ẑ ( i+1 ) t+1 , z ( i ) t ) . A full trellis network can be built by tiling this elementary procedure across time and depth . Given . an input sequence x1 : T , we apply the same production procedure across all time steps and all layers , using the same weights . The transformation is the same for all elements in the temporal dimension and in the depth dimension . This is illustrated in Figure 1b . Note that since we inject the same input sequence at every layer of the TrellisNet , we can precompute the linear transformation x̃t+1 = W x 1 xt +W x 2 xt+1 for all layers i . This identical linear combination of the input can then be added in each layer i to the appropriate linear combination of the hidden units , W z1 z ( i ) t + W z 2 z ( i ) t+1 , where W xj ∈ Rr×p , W zj ∈ Rr×q . Now observe that in each level of the network , we are in effect performing a 1D convolution over the hidden units z ( i ) 1 : T . The output of this convolution is then passed through the activation function f . Formally , withW ∈ Rr×q as the kernel weight matrix , the computation in layer i can be summarized as follows ( Figure 1b ) : ẑ ( i+1 ) 1 : T = Conv1D ( z ( i ) 1 : T ; W ) + x̃1 : T , z ( i+1 ) 1 : T = f ( ẑ ( i+1 ) 1 : T , z ( i ) 1 : T−1 ) . ( 3 ) The resulting network operates in feed-forward fashion , with deeper elements having progressively larger receptive fields . There are , however , important differences from typical ( temporal ) convolutional networks . Notably , the filter matrix is shared across all layers . That is , the weights are tied not only across time but also across depth . ( Vogel & Pock ( 2017 ) have previously tied weights across depth in image processing . ) Another difference is that the transformed input sequence x̃1 : T is directly injected into each hidden layer . These differences and their importance will be analyzed further in Section 4 . The activation function f in Equation ( 3 ) can be any nonlinearity that processes the pre-activation output ẑ ( i+1 ) 1 : T and the output from the previous layer z ( i ) 1 : T−1 . We will later describe an activation function based on the LSTM cell . The rationale for its use will become clearer in light of the analysis presented in the next section .

Summary: The authors propose a new type of neural network architecture for sequence modelling : Trellis Networks. A trellis network is a special case of temporal convolutional network with shared weights across time and layers and  with input at each layer. As stated by the authors, this architecture does not seem really interesting. The authors show that there exists  an equivalent Trellis Network to any truncated RNN and therefore that truncated RNN can be represented by temporal convolutional network. This result is not surprising since  truncated RNN can be  unrolled and that their time dependency is bounded.  The construction of the Trellis Network equivalent to a truncated RNN involves sparse weight matrices, therefore using full weight matrices provides a greater expressive power. One can regret that the authors do not explain what kind of modelling power one can gain with full weight matrices.