Seq2Tens: An Efficient Representation of Sequences by Low-Rank Tensor Projections

1 INTRODUCTION . A central task of learning is to find representations of the underlying data that efficiently and faithfully capture their structure . In the case of sequential data , one data point consists of a sequence of objects . This is a rich and non-homogeneous class of data and includes classical uni- or multi-variate time series ( sequences of scalars or vectors ) , video ( sequences of images ) , and text ( sequences of letters ) . Particular challenges of sequential data are that each sequence entry can itself be a highly structured object and that data sets typically include sequences of different length which makes naive vectorization troublesome . Contribution . Our main result is a generic method that takes a static feature map for a class of objects ( e.g . a feature map for vectors , images , or letters ) as input and turns this into a feature map for sequences of arbitrary length of such objects ( e.g . a feature map for time series , video , or text ) . We call this feature map for sequences Seq2Tens for reasons that will become clear ; among its attractive properties are that it ( i ) provides a structured , parsimonious description of sequences ; generalizing classical methods for strings , ( ii ) comes with theoretical guarantees such as universality , ( iii ) can be turned into modular and flexible neural network ( NN ) layers for sequence data . The key ingredient to our approach is to embed the feature space of the static feature map into a larger linear space that forms an algebra ( a vector space equipped with a multiplication ) . The product in this algebra is then used to “ stitch together ” the static features of the individual sequence entries in a structured way . The construction that allows to do all this is classical in mathematics , and known as the free algebra ( over the static feature space ) . Outline . Section 2 formalizes the main ideas of Seq2Tens and introduces the free algebra T ( V ) over a space V as well as the associated product , the so-called convolution tensor product . Section 3 shows how low rank ( LR ) constructions combined with sequence-to-sequence transforms allows one to efficiently use this rich algebraic structure . Section 4 applies the results of Sections 2 and 3 to build modular and scalable NN layers for sequential data . Section 5 demonstrates the flexibility and modularity of this approach on both discriminative and generative benchmarks . Section 6 makes connections with previous work and summarizes this article . In the appendices we provide mathematical background , extensions , and detailed proofs for our theoretical results . 2 CAPTURING ORDER BY NON-COMMUTATIVE MULTIPLICATION . We denote the set of sequences of elements in a set X by Seq ( X ) = { x = ( xi ) i=1 , ... , L : xi ∈ X , L ≥ 1 } ( 1 ) where L ≥ 1 is some arbitrary length . Even if X itself is a linear space , e.g . X = R , Seq ( X ) is never a linear space since there is no natural addition of two sequences of different length . Seq2Tens in a nutshell . Given any vector space V we may construct the so-called free algebra T ( V ) over V . We describe the space T ( V ) in detail below , but as for now the only thing that is important is that T ( V ) is also a vector space that includes V , and that it carries a non-commutative product , which is , in a precise sense , “ the most general product ” on V . The main idea of Seq2Tens is that any “ static feature map ” for elements in X φ : X → V can be used to construct a new feature map Φ : Seq ( X ) → T ( V ) for sequences in X by using the algebraic structure of T ( V ) : the non-commutative product on T ( V ) makes it possible to “ stitch together ” the individual features φ ( x1 ) , . . . , φ ( xL ) ∈ V ⊂ T ( V ) of the sequence x in the larger space T ( V ) by multiplication in T ( V ) . With this we may define the feature map Φ ( x ) for a sequences x = ( x1 , . . . , xL ) ∈ Seq ( X ) as follows ( i ) lift the map φ : X → V to a map ϕ : X → T ( V ) , ( ii ) map Seq ( X ) → Seq ( T ( V ) ) by ( x1 , . . . , xL ) 7→ ( ϕ ( x1 ) , . . . , ϕ ( xL ) ) , ( iii ) map Seq ( T ( V ) ) → T ( V ) by multiplication ( ϕ ( x1 ) , . . . , ϕ ( xL ) ) 7→ ϕ ( x1 ) · · ·ϕ ( xL ) . In a more concise form , we define Φ as Φ : Seq ( X ) → T ( V ) , Φ ( x ) = L∏ i=1 ϕ ( xi ) ( 2 ) where ∏ denotes multiplication in T ( V ) . We refer to the resulting map Φ as the Seq2Tens map , which stands short for Sequences-2-Tensors . Why is this construction a good idea ? First note , that step ( i ) is always possible since V ⊂ T ( V ) and we discuss the simplest such lift before Theorem 2.1 as well as other choices in Appendix B . Further , if φ , respectively ϕ , provides a faithful representation of objects in X , then there is no loss of information in step ( ii ) . Finally , since step ( iii ) uses “ the most general product ” to multiply ϕ ( x1 ) · · ·ϕ ( xL ) one expects that Φ ( x ) ∈ T ( V ) faithfully represents the sequence x as an element of T ( V ) . Indeed in Theorem 2.1 below we show an even stronger statement , namely that if the static feature map φ : X → V contains enough non-linearities so that non-linear functions from X to R can be approximated as linear functions of the static feature map φ , then the above construction extends this property to functions of sequences . Put differently , if φ is a universal feature map for X , then Φ is a universal feature map for Seq ( X ) ; that is , any non-linear function f ( x ) of a sequence x can be approximated as a linear functional of Φ ( x ) , f ( x ) ≈ 〈 ` , Φ ( x ) 〉 . We also emphasize that the domain of Φ is the space Seq ( X ) of sequences of arbitrary ( finite ) length . The remainder of this Section gives more details about steps ( i ) , ( ii ) , ( iii ) for the construction of Φ . The free algebra T ( V ) over a vector space V . Let V be a vector space . We denote by T ( V ) the set of sequences of tensors indexed by their degree m , T ( V ) : = { t = ( tm ) m≥0 | tm ∈ V ⊗m } ( 3 ) where by convention V ⊗0 = R. For example , if V = Rd and t = ( tm ) m≥0 is some element of T ( Rd ) , then its degreem = 1 component is a d-dimensional vector t1 , its degreem = 2 component is a d× d matrix t2 , and its degree m = 3 component is a degree 3 tensor t3 . By defining addition and scalar multiplication as s + t : = ( sm + tm ) m≥0 , c · t = ( ctm ) m≥0 ( 4 ) the set T ( V ) becomes a linear space . By identifying v ∈ V as the element ( 0 , v , 0 , . . . , 0 ) ∈ T ( V ) we see that V is a linear subspace of T ( V ) . Moreover , while V is only a linear space , T ( V ) carries a product that turns T ( V ) into an algebra . This product is the so-called tensor convolution product , and is defined for s , t ∈ T ( V ) as s · t : = ( m∑ i=0 si ⊗ tm−i ) m≥0 = ( 1 , s1 + t1 , s2 + s1 ⊗ t1 + t2 , . . . ) ∈ T ( V ) ( 5 ) where ⊗ denotes the usual outer tensor product ; e.g . for vectors u = ( ui ) , v = ( vi ) ∈ Rd the outer tensor product u ⊗ v is the d × d matrix ( uivj ) i , j=1 , ... , d. We emphasize that like the outer tensor product⊗ , the tensor convolution product · is non-commutative , i.e . s ·t 6= t ·s . In a mathematically precise sense , T ( V ) is the most general algebra that contains V ; it is a “ free construction ” . Since T ( V ) is realized as series of tensors of increasing degree , the free algebra T ( V ) is also known as the tensor algebra in the literature . Appendix A contains background on tensors and further examples . Lifting static feature maps . Step ( i ) in the construction of Φ requires turning a given feature map φ : X → V into a map ϕ : X → T ( V ) . Throughout the rest of this article we use the lift ϕ ( x ) = ( 1 , φ ( x ) , 0 , 0 . . . ) ∈ T ( V ) . ( 6 ) We discuss other choices in Appendix B , but attractive properties of the lift 6 are that ( a ) the evaluation of Φ against low rank tensors becomes a simple recursive formula ( Proposition 3.3 , ( b ) it is a generalization of sequence sub-pattern matching as used in string kernels ( Appendix B.3 , ( c ) despite its simplicity it performs exceedingly well in practice ( Section 4 ) . Extending to sequences of arbitrary length . Steps ( i ) and ( ii ) in the construction specify how the map Φ : X → T ( V ) behaves on sequences of length-1 , that is , single observations . Step ( iii ) amounts to the requirement that for any two sequences x = ( x1 , . . . , xK ) , y = ( y1 , . . . , yL ) ∈ Seq ( V ) , their concatenation defined as z = ( x1 , . . . , xK , y1 , . . . , yL ) ∈ Seq ( V ) can be understood in the feature space as ( non-commutative ) multiplication of their corresponding features Φ ( z ) = Φ ( x ) · Φ ( y ) . ( 7 ) In other words , we inductively extend the lift ϕ to sequences of arbitrary length by starting from sequences consisting of a single observation , which is given in equation 2 . Repeatedly applying the definition of the tensor convolution product in equation 5 leads to the following explicit formula Φm ( x ) = ∑ 1≤i1 < ··· < im≤L xi1 ⊗ · · · ⊗ xim ∈ V ⊗m , Φ ( x ) = ( Φm ( x ) ) m≥0 , ( 8 ) where x = ( x1 , . . . , xL ) ∈ Seq ( V ) and the summation is over non-contiguous subsequences of x . Some intuition : generalized pattern matching . Our derivation of the feature map Φ ( x ) = ( 1 , Φ1 ( x ) , Φ2 ( x ) , . . . ) ∈ T ( V ) was guided by general algebraic principles , but equation 8 provides an intuitive interpretation . It shows that for each m ≥ 1 , the entry Φm ( x ) ∈ V ⊗m constructs a summary of a long sequence x = ( x1 , . . . , xL ) ∈ Seq ( V ) based on subsequences ( xi1 , . . . , xim ) of x of length-m . It does this by taking the usual outer tensor product xi1 ⊗ · · · ⊗ xim ∈ V ⊗m and summing over all possible subsequences . This is completely analogous to how string kernels provide a structured description of text by looking at non-contiguous substrings of length-m ( indeed , Appendix B.3 makes this rigorous ) . However , the main difference is that the above construction works for arbitrary sequences and not just sequences of discrete letters . Readers with less mathematical background might simply take this as motivation and regard equation 8 as definition . However , the algebraic background allows to prove that Φ is universal , see Theorem 2.1 below . Universality . A function φ : X → V is said to be universal for X if all continuous functions on X can be approximated as linear functions on the image of φ . One of the most powerful features of neural nets is their universality ( Hornik , 1991 ) . A very attractive property of Φ is that it preserves universality : if φ : X → V is universal forX , then Φ : Seq ( X ) → T ( V ) is universal for Seq ( X ) . To make this precise , note that V ⊗m is a linear space and therefore any ` = ( ` 0 , ` 1 , . . . , ` M , 0 , 0 , . . . ) ∈ T ( V ) consisting of M tensors ` m ∈ V ⊗m , yields a linear functional on T ( V ) ; e.g . if V = Rd and we identify ` m in coordinates as ` m = ( ` i1 , ... , imm ) i1 , ... , im∈ { 1 , ... , d } then 〈 ` , t〉 : = M∑ m=0 〈 ` m , tm〉 = M∑ m=0 ∑ i1 , ... , im∈ { 1 , ... , d } ` i1 , ... , imm t i1 , ... , im m . ( 9 ) Thus linear functionals of the feature map Φ , are real-valued functions of sequences . Theorem 2.1 below shows that any continuous function f : Seq ( X ) → R can by arbitrary well approximated by a ` ∈ T ( V ) , f ( x ) ≈ 〈 ` , Φ ( x ) 〉 . Theorem 2.1 . Let φ : X → V be a universal map with a lift that satisfies some mild constraints , then the following map is universal : Φ : Seq ( X ) → T ( V ) , x 7→ Φ ( x ) . ( 10 ) A detailed proof and the precise statement of Theorem 2.1 is given in Appendix B .

Summary: This paper proposes to embed static feature maps into a larger linear space and shows that the proposed method achieves good performance on standard benchmarks. Detailed proofs and theoretical results are given in the appendices. The use of free algebras in ML seems novel and under-explored, although it is classical in mathematics. This paper shows that algebraic structure can significantly elevate the performance of existing models empirically.