On the approximation properties of recurrent encoder-decoder architectures

1 Introduction . Encoder-decoder is an increasingly popular architecture for sequence to sequence modelling problems ( Sutskever et al. , 2014 ; Chiu et al. , 2018 ; Venugopalan et al. , 2015 ) . The core of this architecture is to first encode the input sequence into a vector using the encoder and then map the vector into the output sequence through the decoder . In particular , such architecture forms the main component in the transformer network ( Vaswani et al. , 2017 ) , which has become a powerful method for modelling sequence to sequence relationships ( Parmar et al. , 2018 ; Beltagy et al. , 2020 ; Li et al. , 2019 ) . The encoder-decoder family of structures differ significantly from direct application of recurrent neural networks ( RNNs , Elman ( 1990 ) ) and its generalisations ( Hochreiter & Schmidhuber , 1997 ; Cho et al. , 2014b ) for processing sequences . However , both architectures can be considered as modelling mappings between sequences , albeit with different underlying structures . Hence , a natural but unresolved question is : how are these approaches fundamentally different ? Answering this question is not only of theoretical importance but also of practical interest . Currently , architectural selection for different time series modelling tasks is predominantly empirical . Thus , it is desirable to develop a concrete mathematical framework to understand the key differences between separate architectures in order to guide practitioners in a principled way . ∗Equal contribution †Corresponding author In this paper , we investigate the approximation properties of encoder-decoder architectures . Approximation is one of the most basic and important problems for supervised learning . It considers to what extent the model can fit a target . In particular , we prove a general approximation result in the linear setting , which characterises the types of temporal input-output relationships that can be efficiently approximated by encoder-decoder architectures . These results reveal that such architectures essentially generalise RNNs by lifting the requirement of time-homogeneity ( see Remark 3.2 ) in the target relationships . Hence , it can be used to tackle a broader class of sequence to sequence problems . Furthermore , of particular interest is the identification of a “ temporal product structure ” — a precise property of the target temporal relationship that highlights another intrinsic difference between recurrent encoder-decoders and RNNs . Our main contributions can be summarised as follows . 1 . We prove a universal approximation result for recurrent encoder-decoder architectures in the linear setting , including the approximation rates . 2 . We show that in the considered setting , the recurrent encoder-decoder generalises the RNNs and can approximate time-inhomogeneous relationships , which further adapt to additional temporal product structures in the target relationship . This answers precisely how encoder-decoders are different from RNNs , at least in the considered setting . Organisation . In Section 2 , we review the related work on encoder-decoder architectures and general approximation theories of sequence modelling . The approximation problem is formulated in Section 3 . Our main results , their consequences and numerical illustrations are presented in Section 4 . All the proofs and numerical details are included in appendices . Notations . For consistency , we adhere to the following notations . Boldfaced letters are reserved for sequences or paths , which can be understood as functions of time . Lower case letters can mean vectors or scalars . Matrices are denoted by capital letters . For α ∈ N , Cα denotes the space of functions with continuous derivatives up to order-α . 2 Related work . We first review some previous works on sequence to sequence modelling . The encoderdecoder architecture first appeared in Kalchbrenner & Blunsom ( 2013 ) , where they map the input sequence into a vector using convolutional neural networks ( CNNs ) , and then using a recurrent structure to map the vector to the output sequence . With the flexibility of manipulating the underlying structure of encoder and decoder , numerous models based on this architecture have come out thereafter . For instance , Cho et al . ( 2014b ) used gated RNNs as both the encoder and decoder , while in the later work ( Cho et al. , 2014a ) , they proposed a CNN-based decoder . In Sutskever et al . ( 2014 ) , they proposed a deep LSTM for both the encoder and decoder . Bahdanau et al . ( 2015 ) first introduced the attention mechanism , which was further developed in the well-known transformer networks ( Vaswani et al. , 2017 ) . However , most of the research on encoder-decoder architectures focused on applications . A theoretical understanding is helpful for its further improvement and development . From the theoretical point of view , Ye & Sung ( 2019 ) studied several theoretical properties of CNN encoder-decoders , including expressiveness , generalisation capability and optimisation landscape . Of particular relevance to the current work is expressiveness , which considers the relationships that can be generated from the architecture . However , this is not approximation . Yun et al . ( 2020 ) proved the universal approximation property of transformers for certain classes of functions , for example , permutation equivariant functions , but they did not consider the actual dynamical properties of target relationships that affect approximation . Dynamical proprieties such as memory , smoothness and low rank structures are essential , because they can precisely characterise different temporal relationships and affect the approximation capabilities of models . Assuming the target generated from a hidden dynamical system is one approach , which is widely applied ( Maass et al. , 2007 ; Schäfer & Zimmermann , 2007 ; Doya , 1993 ; Funahashi & Nakamura , 1993 ) . In contrast , a functional-based approach is recently introduced , where the target temporal relationships are generated from functionals satisfying specific properties such as linearity , continuity , regularity and time-homogeneity ( Li et al. , 2021 ) . In Li et al . ( 2021 ) , the approximation properties of linear RNN models are studied , and the results therein show that the approximation efficiency is related to the memory structure . In Jiang et al . ( 2021 ) , similar formulations are applied to investigate convolutional architectures , where the results suggest that targets with certain spectrum regularity can be well approximated by dilated CNNs . Under this framework , the target temporal relationship that can be efficiently approximated is characterised by properties such as memory , smoothness and sparsity . This enables us to make precise mathematical comparisons between different architectures . Our results in this work reveal that the encoder-decoders have a special temporal product structure which is intrinsically different from other sequence modelling architectures . 3 Problem formulation . In this section , we precisely define the input space , output space , concept space and hypothesis space , respectively . Functional formulation of temporal modelling . First , we define the input and output space precisely . A temporal sequence can be viewed as a function of time t. The input space is defined by X = C0 ( ( −∞ , 0 ] , Rd ) . This is the space of continuous functions from ( −∞ , 0 ] to Rd vanishing at infinity , where d ∈ N+ is the dimension . Denote the element in X by x = { xt ∈ Rd : t ≤ 0 } , we equip X with the supremum norm ‖x‖X : = supt≤0 ‖xt‖∞ . We take the outputs space as Y = Cb ( [ 0 , ∞ ) , R ) , the space of bounded continuous functions from [ 0 , ∞ ) to R. We consider real-valued outputs , since each dimension can be handled individually for vector-valued outputs . The mapping between input and output sequences can be formulated as a sequence of functionals , i.e . yt = Ht ( x ) , t ≥ 0 . The output yt at the time step t depends on the input sequence x . The ground truth relation between inputs and outputs is formulated by the sequence of functionals H : = { Ht : t ≥ 0 } . We provide an example to illustrate the above formulation . Given an input x , the output y is a smoothed version of x , resulting from convolving x with the Gaussian kernel g ( s ) = 1√ 2π exp ( − s2 2 ) . This relation can be formulated as yt = Ht ( x ) = ∫∞ 0 g ( t+ s ) x−sds . The RNN encoder-decoder model . For the supervised learning problem , our goal is to use a model to learn the target relationship H. First , we define the model . Among all different variants of the encoder-decoder architectures , the RNN encoder-decoder introduced in Cho et al . ( 2014b ) can be considered as the most simple and representative model , where the encoder and decoder are both RNNs . We study this particular model as we try to eliminate other factors and only focus on the encoder-decoder architecture itself . Under our setting , the simplified model of Cho et al . ( 2014b ) with RNNs as both encoder and decoder can be formulated as hs = σE ( WEhs−1 + UExs + bE ) , v = hτ , gt = σD ( WDgt−1 + bD ) , g0 = v , ot = WOgt + bO , ( 1 ) where ht , gt are hidden states of the encoder and decoder respectively . Recurrent activation functions are denoted by σE and σD . Here , τ denotes the terminating time step of the encoder , and v is the summary of the input sequence , which is called as the coding vector . The model prediction is denoted as ot ∈ R. All the other notations are model parameters . Equation ( 1 ) describes the following model dynamics . First , the encoder reads the entire input x , and then summarises the input into a fixed size coding vector v , which is also the last hidden state of the encoder . Next , the coding vector is passed into the decoder as the initial state , and then the decoder produces an output at each time step . Note that the encoder has a terminating time , and the decoder has a starting time . This is the reason why we take the input and output as semi-infinite sequences . We study a linear , residual and continuous-time idealisation of the model dynamics ( 1 ) : ḣs = Whs + Uxs , v = Qh0 , s ≤ 0 ġt = V gt , g0 = Pv , ot = c > gt , t ≥ 0 , ( 2 ) where W ∈ RmE×mE , U ∈ RmE×d , Q ∈ RN×mE , V ∈ RmD×mD , P ∈ RmD×N and c ∈ RmD are parameters . mE and mD denote the width of encoder and decoder , respectively . The coding vector v has dimension N , where we apply linear transformations to control it . We assume h−∞ = 0 , which is the usual choice for the initial condition of RNN hidden states . Since our goal is to investigate approximation problems over large time horizons , we are supposed to consider the stable RNN encoder-decoders , where W ∈ WmE : = { W ∈ RmE×mE : eigenvalues of W have negative real parts } , ( 3 ) V ∈ VmD : = { V ∈ RmD×mD : eigenvalues of V have negative real parts } . ( 4 ) The hypothesis space of RNN encoder-decoder models with arbitrary widths and coding vector dimension is defined as Ĥ : = ⋃ mE , mD , N∈N+ ĤmE , mD , N , where ĤmE , mD , N : = { Ĥ : = { Ĥt : t ≥ 0 } : Ĥt ( x ) = c > eV tP ∫ ∞ 0 QeWsUx−sds , with ( W , U , Q , V , P , c ) ∈ WmE × RmE×d × RN×mE × VmD × RmD×N × RmD } . ( 5 ) The widths mE , mD and the coding vector dimension N together control the capacity/complexity of the hypothesis space . Note that the assumptions on eigenvalues of W and V ensure that the parameterized linear functionals are continuous . Due to the mathematical form ( 5 ) , not all functionals can be represented by RNN encoderdecoders . To achieve a good approximation , the target functionals must possess certain structures . We introduce the following definitions to clarify these structures . Definition 3.1 . Let H = { Ht : t ≥ 0 } be a sequence of functionals . 1 . For any t ≥ 0 , the functional Ht is linear and continuous if for any λ1 , λ2 ∈ R and x1 , x2 ∈ X , we have Ht ( λ1x1 + λ2x2 ) = λ1Ht ( x1 ) + λ2Ht ( x2 ) , and ‖Ht‖ : = supx∈X , ‖x‖X≤1 |Ht ( x ) | < ∞ , where ‖Ht‖ denotes the induced functional norm . 2 . For any t ≥ 0 , the functional Ht is regular if for any sequence { x ( n ) } ∞n=1 ⊂ X such that limn→∞ x ( n ) s = 0 for almost every s ≤ 0 ( Lebesgue measure ) , we have limn→∞Ht ( x ( n ) ) = 0 . For a sequence of functionals H , we define its norm by ‖H‖ : = ∫ ∞ 0 ‖Ht‖dt . Remark 3.1 . The definitions of linear and continuous functionals are standard . One can view regular functionals as those not determined by inputs on arbitrarily small time intervals , e.g . an infinitely thin spike ( i.e . δ-functions ) . Given the above definitions , we immediately have the following observation . Proposition 3.1 . Let Ĥ ∈ Ĥ be a sequence of functionals in the RNN encoder-decoder hypothesis space ( see ( 5 ) ) . Then for any t ≥ 0 , Ĥt ∈ Ĥ is a linear , continuous and regular functional . Furthermore , ‖Ĥt‖ decays exponentially as a function of t. The proof is found in Appendix A . This proposition characterises properties of the encoderdecoder hypothesis space . In particular , it is different from the RNN hypothesis space discussed in Li et al . ( 2021 ) , since the encoder-decoder is not necessarily time-homogeneous . Remark 3.2 . A sequence of functionals H is time-homogeneous if for any t , τ ≥ 0 , Ht ( x ) = Ht+τ ( x ( τ ) ) , with x ( τ ) s = xs−τ for all s ∈ R. That is , if the input is shifted to the right by τ , the output is also shifted by τ . Temporal convolution is an example of time-homogeneous operation ( recall the Gaussian convolution discussed in Section 3 . An example of timeinhomogeneous relationship is video captioning : shifts in the sequence of input video frames do not necessarily lead to corresponding shifts in the caption text sequence . Relation with RNNs . Here , we emphasise the differences between the encoder-decoder hypothesis space and the RNN hypothesis space discussed in Li et al . ( 2021 ) , where Ĥ ( RNN ) t ( x ) = ∫∞ 0 c > eW ( t+s ) Ux−sds . A key difference is that the encoder-decoder has a structure involving two temporal parameters t and s , while the RNN only has one depending on t + s , due to the time-homogeneity . Owing to this difference and the fact that Ĥ ( RNN ) ⊂ Ĥ , the encoder-decoder hypothesis space ( 5 ) is more general , with the extra capability to learn time-inhomogeneous relationships . Furthermore , eV t and eWs adapt to a temporal product structure , which is an intrinsic difference between encoder-decoders and other architectures . We will discuss this in detail in the next section .

Summary: This paper provides theoretical insight for approximation properties of RNN encoder-decoder architecture in linear setting. More specifically, they study supervised learning problem of temporal modelling where a first RNN encodes a given sequence into a coding vector and a second RNN is responsible to decode said vector to a target output sequence. Linear setting here refers to a linear and continuous-time idealization in eq 6.  Their analysis summarized as following: 1.  Universal approximation: any linear, continuous, and regular temporal relation can be approximated by RNN encoder-decoder up to arbitrary accuracy. 2.  Approximation rate for large size coding vector: beside width of encoder and decoder, this error bound depends on $\alpha$ smoothness of $\mathbf{H}$ and $\beta$ temporal decay rates of the output of constant signal under $\mathbf{H}$ where $\mathbf{H}$ is a sequence of linear, continuous, and regular functionals on inputs. Each $\mathbf{H}$ has a unique two-parameter representation $\rho(t,s)$. Target functionals that are smooth and have fast decaying memory are identified as good target of this approximation. 3.  Approximation rate for small size coding vector: the architecture and the following assumptions in the paper give rise to an intrinsic structure to this type of RNN encoder-decoders called temporal product structure which can be deconstructed into encoder and decoder parts. By relating this structure to rank of temporal relationships they show approximation rate (in small size coding vector) is additionally a function of rank structure of target relationship. This analysis enables us to see the coding vector size as a knob to control number of parameters vs approximation error. 4.  Experiments: they show that the theoretical analysis holds in their experiments.