On the approximation properties of recurrent encoder-decoder architectures

1 Introduction . Encoder-decoder is an increasingly popular architecture for sequence to sequence modelling problems ( Sutskever et al. , 2014 ; Chiu et al. , 2018 ; Venugopalan et al. , 2015 ) . The core of this architecture is to first encode the input sequence into a vector using the encoder and then map the vector into the output sequence through the decoder . In particular , such architecture forms the main component in the transformer network ( Vaswani et al. , 2017 ) , which has become a powerful method for modelling sequence to sequence relationships ( Parmar et al. , 2018 ; Beltagy et al. , 2020 ; Li et al. , 2019 ) . The encoder-decoder family of structures differ significantly from direct application of recurrent neural networks ( RNNs , Elman ( 1990 ) ) and its generalisations ( Hochreiter & Schmidhuber , 1997 ; Cho et al. , 2014b ) for processing sequences . However , both architectures can be considered as modelling mappings between sequences , albeit with different underlying structures . Hence , a natural but unresolved question is : how are these approaches fundamentally different ? Answering this question is not only of theoretical importance but also of practical interest . Currently , architectural selection for different time series modelling tasks is predominantly empirical . Thus , it is desirable to develop a concrete mathematical framework to understand the key differences between separate architectures in order to guide practitioners in a principled way . ∗Equal contribution †Corresponding author In this paper , we investigate the approximation properties of encoder-decoder architectures . Approximation is one of the most basic and important problems for supervised learning . It considers to what extent the model can fit a target . In particular , we prove a general approximation result in the linear setting , which characterises the types of temporal input-output relationships that can be efficiently approximated by encoder-decoder architectures . These results reveal that such architectures essentially generalise RNNs by lifting the requirement of time-homogeneity ( see Remark 3.2 ) in the target relationships . Hence , it can be used to tackle a broader class of sequence to sequence problems . Furthermore , of particular interest is the identification of a “ temporal product structure ” — a precise property of the target temporal relationship that highlights another intrinsic difference between recurrent encoder-decoders and RNNs . Our main contributions can be summarised as follows . 1 . We prove a universal approximation result for recurrent encoder-decoder architectures in the linear setting , including the approximation rates . 2 . We show that in the considered setting , the recurrent encoder-decoder generalises the RNNs and can approximate time-inhomogeneous relationships , which further adapt to additional temporal product structures in the target relationship . This answers precisely how encoder-decoders are different from RNNs , at least in the considered setting . Organisation . In Section 2 , we review the related work on encoder-decoder architectures and general approximation theories of sequence modelling . The approximation problem is formulated in Section 3 . Our main results , their consequences and numerical illustrations are presented in Section 4 . All the proofs and numerical details are included in appendices . Notations . For consistency , we adhere to the following notations . Boldfaced letters are reserved for sequences or paths , which can be understood as functions of time . Lower case letters can mean vectors or scalars . Matrices are denoted by capital letters . For α ∈ N , Cα denotes the space of functions with continuous derivatives up to order-α . 2 Related work . We first review some previous works on sequence to sequence modelling . The encoderdecoder architecture first appeared in Kalchbrenner & Blunsom ( 2013 ) , where they map the input sequence into a vector using convolutional neural networks ( CNNs ) , and then using a recurrent structure to map the vector to the output sequence . With the flexibility of manipulating the underlying structure of encoder and decoder , numerous models based on this architecture have come out thereafter . For instance , Cho et al . ( 2014b ) used gated RNNs as both the encoder and decoder , while in the later work ( Cho et al. , 2014a ) , they proposed a CNN-based decoder . In Sutskever et al . ( 2014 ) , they proposed a deep LSTM for both the encoder and decoder . Bahdanau et al . ( 2015 ) first introduced the attention mechanism , which was further developed in the well-known transformer networks ( Vaswani et al. , 2017 ) . However , most of the research on encoder-decoder architectures focused on applications . A theoretical understanding is helpful for its further improvement and development . From the theoretical point of view , Ye & Sung ( 2019 ) studied several theoretical properties of CNN encoder-decoders , including expressiveness , generalisation capability and optimisation landscape . Of particular relevance to the current work is expressiveness , which considers the relationships that can be generated from the architecture . However , this is not approximation . Yun et al . ( 2020 ) proved the universal approximation property of transformers for certain classes of functions , for example , permutation equivariant functions , but they did not consider the actual dynamical properties of target relationships that affect approximation . Dynamical proprieties such as memory , smoothness and low rank structures are essential , because they can precisely characterise different temporal relationships and affect the approximation capabilities of models . Assuming the target generated from a hidden dynamical system is one approach , which is widely applied ( Maass et al. , 2007 ; Schäfer & Zimmermann , 2007 ; Doya , 1993 ; Funahashi & Nakamura , 1993 ) . In contrast , a functional-based approach is recently introduced , where the target temporal relationships are generated from functionals satisfying specific properties such as linearity , continuity , regularity and time-homogeneity ( Li et al. , 2021 ) . In Li et al . ( 2021 ) , the approximation properties of linear RNN models are studied , and the results therein show that the approximation efficiency is related to the memory structure . In Jiang et al . ( 2021 ) , similar formulations are applied to investigate convolutional architectures , where the results suggest that targets with certain spectrum regularity can be well approximated by dilated CNNs . Under this framework , the target temporal relationship that can be efficiently approximated is characterised by properties such as memory , smoothness and sparsity . This enables us to make precise mathematical comparisons between different architectures . Our results in this work reveal that the encoder-decoders have a special temporal product structure which is intrinsically different from other sequence modelling architectures . 3 Problem formulation . In this section , we precisely define the input space , output space , concept space and hypothesis space , respectively . Functional formulation of temporal modelling . First , we define the input and output space precisely . A temporal sequence can be viewed as a function of time t. The input space is defined by X = C0 ( ( −∞ , 0 ] , Rd ) . This is the space of continuous functions from ( −∞ , 0 ] to Rd vanishing at infinity , where d ∈ N+ is the dimension . Denote the element in X by x = { xt ∈ Rd : t ≤ 0 } , we equip X with the supremum norm ‖x‖X : = supt≤0 ‖xt‖∞ . We take the outputs space as Y = Cb ( [ 0 , ∞ ) , R ) , the space of bounded continuous functions from [ 0 , ∞ ) to R. We consider real-valued outputs , since each dimension can be handled individually for vector-valued outputs . The mapping between input and output sequences can be formulated as a sequence of functionals , i.e . yt = Ht ( x ) , t ≥ 0 . The output yt at the time step t depends on the input sequence x . The ground truth relation between inputs and outputs is formulated by the sequence of functionals H : = { Ht : t ≥ 0 } . We provide an example to illustrate the above formulation . Given an input x , the output y is a smoothed version of x , resulting from convolving x with the Gaussian kernel g ( s ) = 1√ 2π exp ( − s2 2 ) . This relation can be formulated as yt = Ht ( x ) = ∫∞ 0 g ( t+ s ) x−sds . The RNN encoder-decoder model . For the supervised learning problem , our goal is to use a model to learn the target relationship H. First , we define the model . Among all different variants of the encoder-decoder architectures , the RNN encoder-decoder introduced in Cho et al . ( 2014b ) can be considered as the most simple and representative model , where the encoder and decoder are both RNNs . We study this particular model as we try to eliminate other factors and only focus on the encoder-decoder architecture itself . Under our setting , the simplified model of Cho et al . ( 2014b ) with RNNs as both encoder and decoder can be formulated as hs = σE ( WEhs−1 + UExs + bE ) , v = hτ , gt = σD ( WDgt−1 + bD ) , g0 = v , ot = WOgt + bO , ( 1 ) where ht , gt are hidden states of the encoder and decoder respectively . Recurrent activation functions are denoted by σE and σD . Here , τ denotes the terminating time step of the encoder , and v is the summary of the input sequence , which is called as the coding vector . The model prediction is denoted as ot ∈ R. All the other notations are model parameters . Equation ( 1 ) describes the following model dynamics . First , the encoder reads the entire input x , and then summarises the input into a fixed size coding vector v , which is also the last hidden state of the encoder . Next , the coding vector is passed into the decoder as the initial state , and then the decoder produces an output at each time step . Note that the encoder has a terminating time , and the decoder has a starting time . This is the reason why we take the input and output as semi-infinite sequences . We study a linear , residual and continuous-time idealisation of the model dynamics ( 1 ) : ḣs = Whs + Uxs , v = Qh0 , s ≤ 0 ġt = V gt , g0 = Pv , ot = c > gt , t ≥ 0 , ( 2 ) where W ∈ RmE×mE , U ∈ RmE×d , Q ∈ RN×mE , V ∈ RmD×mD , P ∈ RmD×N and c ∈ RmD are parameters . mE and mD denote the width of encoder and decoder , respectively . The coding vector v has dimension N , where we apply linear transformations to control it . We assume h−∞ = 0 , which is the usual choice for the initial condition of RNN hidden states . Since our goal is to investigate approximation problems over large time horizons , we are supposed to consider the stable RNN encoder-decoders , where W ∈ WmE : = { W ∈ RmE×mE : eigenvalues of W have negative real parts } , ( 3 ) V ∈ VmD : = { V ∈ RmD×mD : eigenvalues of V have negative real parts } . ( 4 ) The hypothesis space of RNN encoder-decoder models with arbitrary widths and coding vector dimension is defined as Ĥ : = ⋃ mE , mD , N∈N+ ĤmE , mD , N , where ĤmE , mD , N : = { Ĥ : = { Ĥt : t ≥ 0 } : Ĥt ( x ) = c > eV tP ∫ ∞ 0 QeWsUx−sds , with ( W , U , Q , V , P , c ) ∈ WmE × RmE×d × RN×mE × VmD × RmD×N × RmD } . ( 5 ) The widths mE , mD and the coding vector dimension N together control the capacity/complexity of the hypothesis space . Note that the assumptions on eigenvalues of W and V ensure that the parameterized linear functionals are continuous . Due to the mathematical form ( 5 ) , not all functionals can be represented by RNN encoderdecoders . To achieve a good approximation , the target functionals must possess certain structures . We introduce the following definitions to clarify these structures . Definition 3.1 . Let H = { Ht : t ≥ 0 } be a sequence of functionals . 1 . For any t ≥ 0 , the functional Ht is linear and continuous if for any λ1 , λ2 ∈ R and x1 , x2 ∈ X , we have Ht ( λ1x1 + λ2x2 ) = λ1Ht ( x1 ) + λ2Ht ( x2 ) , and ‖Ht‖ : = supx∈X , ‖x‖X≤1 |Ht ( x ) | < ∞ , where ‖Ht‖ denotes the induced functional norm . 2 . For any t ≥ 0 , the functional Ht is regular if for any sequence { x ( n ) } ∞n=1 ⊂ X such that limn→∞ x ( n ) s = 0 for almost every s ≤ 0 ( Lebesgue measure ) , we have limn→∞Ht ( x ( n ) ) = 0 . For a sequence of functionals H , we define its norm by ‖H‖ : = ∫ ∞ 0 ‖Ht‖dt . Remark 3.1 . The definitions of linear and continuous functionals are standard . One can view regular functionals as those not determined by inputs on arbitrarily small time intervals , e.g . an infinitely thin spike ( i.e . δ-functions ) . Given the above definitions , we immediately have the following observation . Proposition 3.1 . Let Ĥ ∈ Ĥ be a sequence of functionals in the RNN encoder-decoder hypothesis space ( see ( 5 ) ) . Then for any t ≥ 0 , Ĥt ∈ Ĥ is a linear , continuous and regular functional . Furthermore , ‖Ĥt‖ decays exponentially as a function of t. The proof is found in Appendix A . This proposition characterises properties of the encoderdecoder hypothesis space . In particular , it is different from the RNN hypothesis space discussed in Li et al . ( 2021 ) , since the encoder-decoder is not necessarily time-homogeneous . Remark 3.2 . A sequence of functionals H is time-homogeneous if for any t , τ ≥ 0 , Ht ( x ) = Ht+τ ( x ( τ ) ) , with x ( τ ) s = xs−τ for all s ∈ R. That is , if the input is shifted to the right by τ , the output is also shifted by τ . Temporal convolution is an example of time-homogeneous operation ( recall the Gaussian convolution discussed in Section 3 . An example of timeinhomogeneous relationship is video captioning : shifts in the sequence of input video frames do not necessarily lead to corresponding shifts in the caption text sequence . Relation with RNNs . Here , we emphasise the differences between the encoder-decoder hypothesis space and the RNN hypothesis space discussed in Li et al . ( 2021 ) , where Ĥ ( RNN ) t ( x ) = ∫∞ 0 c > eW ( t+s ) Ux−sds . A key difference is that the encoder-decoder has a structure involving two temporal parameters t and s , while the RNN only has one depending on t + s , due to the time-homogeneity . Owing to this difference and the fact that Ĥ ( RNN ) ⊂ Ĥ , the encoder-decoder hypothesis space ( 5 ) is more general , with the extra capability to learn time-inhomogeneous relationships . Furthermore , eV t and eWs adapt to a temporal product structure , which is an intrinsic difference between encoder-decoders and other architectures . We will discuss this in detail in the next section .

Summary: This paper provides theoretical studies for why encoder-decoder can be seen as a generalization of RNNs in time-inhomogenous sequence modeling. The authors put an impressive amount of effort into mathematically defining approximation properties of RNN encoder-decoders beginning from a universal approximation result.   The authors first show the universal approximation property of RNN encoder-decoders, and subsequently, they show approximation rates of targets for RNN encoder-decoders. They introduce a notion of temporal products that can characterize the temporal relationships in the input/output pair.