Representing Unordered Data Using Multiset Automata and Complex Numbers

1 INTRODUCTION . Neural networks which operate on set-structured input have been gaining interest for their ability to handle unordered and variable-sized inputs ( Vinyals et al. , 2015 ; Wagstaff et al. , 2019 ) . They have been applied to various tasks , such as processing graph nodes ( Murphy et al. , 2018 ) , hypergraphs ( Maron et al. , 2019 ) , 3D image reconstruction ( Yang et al. , 2019 ) , and point cloud classification and image tagging ( Zaheer et al. , 2017 ) . Similar network structures have been applied to multiple instance learning ( Pevný and Somol , 2016 ) . In particular , the DeepSets model ( Zaheer et al. , 2017 ) computes a representation of each element of the set , then combines the representations using a commutative function ( e.g. , addition ) to form a representation of the set that discards ordering information . Zaheer et al . ( 2017 ) provide a proof that any function on sets can be modeled this way , by encoding sets as base-4 fractions and using the universal function approximation theorem , but their actual proposed model is far simpler than the model constructed by the theorem . In this paper , we propose to compute representations of multisets using weighted multiset automata , a variant of weighted finite-state ( string ) automata in which the order of the input symbols does not affect the output . In some sense , this is the most general representation of a multiset that can be computed incrementally using only a finite amount of memory , and it can be directly implemented inside a neural network . We show how to train these automata efficiently by approximating them with string automata whose weights form complex , diagonal matrices . Our representation generalizes DeepSets slightly , and it also turns out to be a generalization of the Transformer ’ s position encodings ( Vaswani et al. , 2017 ) . In Sections 4 and 5 , we discuss the application of our representation in both cases . • The Transformer ( Vaswani et al. , 2017 ) models the absolute position of a word within a sentence . This position can be thought of as a multiset over a single element , and indeed the Transformer uses a position encoding involving sinusoidal functions that turns out to be a special case of our representation . So weighted multiset automata provide a new theoretical and intuitive justification for sinusoidal position encodings . We also experiment with several variations on position encodings inspired by this justification , and although they do not yield any improvement , we do find that learned position encodings in our representation do better than learning a different vector for each absolute position . • We extend the DeepSets model to use our representation , which amounts to upgrading it from real to complex numbers . On an extension of one of their tasks ( adding a sequence of one-digit numbers and predicting the units digit ) , our model is able to reach perfect performance , whereas the original DeepSets model does no better than chance . 2 WEIGHTED MULTISET AUTOMATA . We define weighted finite automata below using a matrix formulation . Throughout , let K be either R or C. Definition 1 . A K-weighted finite automaton ( WFA ) over Σ is a tuple M = ( Q , Σ , λ , µ , ρ ) , where Q = { 1 , . . . , d } is a finite set of states , Σ is a finite alphabet , λ ∈ K1×d is a row vector of initial weights , µ : Σ → Kd×d assigns a transition matrix to every symbol , and ρ ∈ Kd×1 is a column vector of final weights . ( We do not use the final weights ρ in this paper , but include them for completeness . ) We extend the mapping µ to strings : If w = w1 · · ·wn ∈ Σ∗ , then µ ( w ) = µ ( w1 ) · · ·µ ( wn ) . Then , the vector of forward weights of a string w is fwM ( w ) = λ ( ∏n p=1 µ ( wp ) ) . Note that , different from many definitions of weighted automata , this definition does not allow -transitions , and there may be more than one initial state . ( Throughout this paper , we use to stand for a small real number . ) The analogue of finite automata for multisets is the special case of the above definition where multiplication of the transition matrices µ ( a ) does not depend on their order . Definition 2 . A K-weighted multiset finite automaton is one whose transition matrices commute pairwise . That is , for all a , b ∈ Σ , we have µ ( a ) µ ( b ) = µ ( b ) µ ( a ) . Our proposal , then , is to represent a multiset w by the vector of forward weights , fwM ( w ) , with respect to some weighted multiset automatonM . In the context of a neural network , the transition weights µ ( a ) can be computed by any function as long as it does not depend on the ordering of symbols , and the forward weights can be used by the network in any way whatsoever . 3 TRAINING . Definition 2 does not lend itself well to training , because parameter optimization needs to be done subject to the commutativity constraint . Previous work ( DeBenedetto and Chiang , 2018 ) suggested approximating training of a multiset automaton by training a string automaton while using a regularizer to encourage the weight matrices to be close to commuting . However , this strategy can not make them commute exactly , and the regularizer , which has O ( |Σ|2 ) terms , is expensive to compute . Here , we pursue a different strategy , which is to restrict the transition matrices µ ( a ) to be diagonal . This guarantees that they commute . As a bonus , diagonal matrices are computionally less expensive than full matrices . Furthermore , we show that if we allow complex weights , we can learn multisets with diagonal matrices almost as well as with full matrices . We show this first for the special case of unary automata ( §3.1 ) and then general multiset automata ( §3.2 ) . 3.1 UNARY AUTOMATA . Call an automaton unary if |Σ| = 1 . Then , for brevity , we simply write µ instead of µ ( a ) where a is the only symbol in Σ . Let ‖ · ‖ be the Frobenius norm ; by equivalence of norms ( Horn and Johnson , 2012 , 352 ) , the results below should carry over to any other matrix norm , as long as it is monotone , that is : if A ≤ B elementwise , then ‖A‖ ≤ ‖B‖ . As stated above , our strategy for training a unary automaton is to allow µ to be complex but restrict it to be diagonal . The restriction does not lose much generality , because any matrix can approximated by a complex diagonal matrix in the following sense ( Horn and Johnson , 2012 , 116 ) : Proposition 1 . For any complex square matrix A and > 0 , there is a complex matrix E such that ‖E‖ ≤ and A+ E is diagonalizable in C. Proof . Form the Jordan decomposition A = PJP−1 . We can choose a diagonal matrix D such that ‖D‖ ≤ κ ( P ) ( where κ ( P ) = ‖P‖‖P −1‖ ) and the diagonal entries of J + D are all different . Then J + D is diagonalizable . Let E = PDP−1 ; then ‖E‖ ≤ ‖P‖‖D‖‖P−1‖ = κ ( P ) ‖D‖ ≤ , and A + E = P ( J + D ) P−1 is also diagonalizable . Thus , for a unary automaton M with transition matrix µ , we can choose Qµ′Q−1 close to µ such that µ′ is diagonal . So M is close to the automaton with initial weights λ′ = λQ and transition weights µ′ ≈ Q−1µQ . This means that in training , we can directly learn complex initial weights λ′ and a complex diagonal transition matrix µ′ , and the resulting automaton ( M ′ ) should be able to represent multisets almost as well as a general unary automaton ( M ) can . It might be thought that even if µ′ approximates µ well , perhaps the forward weights , which involve possibly large powers of µ , will not be approximated well . As some additional assurance , we have the following error bound on the powers of µ : Proposition 2 . For any complex square matrix A , > 0 , and 0 < r < 1 , there is a complex matrix E such that A+ E is diagonalizable in C and , for all n ≥ 0 , ‖ ( A+ E ) n −An‖ ≤ rn if A nilpotent , ‖ ( A+ E ) n −An‖ ‖An‖ ≤ n otherwise . For the proof , please see Appendix A . 3.2 GENERAL CASE . In this section , we allow Σ to be of any size . Proposition 1 unfortunately does not hold in general for multiple matrices ( O ’ Meara and Vinsonhaler , 2006 ) . That is , it may not be possible to perturb a set of commuting matrices so that they are simultaneously diagonalizable . Definition 3 . Matrices A1 , . . . , Am are simultaneously diagonalizable if there exists an invertible matrix P such that PAiP−1 is diagonal for all i ∈ { 1 , · · · , n } . We say thatA1 , · · · , Am are approximately simultaneously diagonalizable ( ASD ) if , for any > 0 , there are matrices E1 , . . . , Em such that ‖Ei‖ ≤ and A1 + E1 , . . . , Am + Em are simultaneously diagonalizable . O ’ Meara and Vinsonhaler ( 2006 ) give examples of sets of matrices that are commuting but not ASD . However , if we are willing to add new states to the automaton ( that is , to increase the dimensionality of the weight matrices ) , we can make them ASD . Proposition 3 . Any weighted multiset automaton is close to an automaton that can be converted to a complex-weighted diagonal automaton , possibly with more states . Proof . First we start with a fact from O ’ Meara and Vinsonhaler ( 2006 ) . Lemma 4 . Suppose A1 , . . . , Ak are commuting n × n matrices over an algebraically closed field F . Then there exists an invertible matrix C such that C−1A1C , . . . , C−1AkC are block diagonal matrices with matching block structures and each diagonal block has only a single eigenvalue ( ignoring multiplicities ) . That is , there is a partition n = n1 + · · ·+ nr of n such that C−1AiC = Bi = Bi1 Bi2 . . . Bir , ( 1 ) where each Bij is an nj × nj matrix having only a single eigenvalue for i = 1 , . . . , k and j = 1 , . . . , r. Moreover , if B1j , . . . , Bkj are ASD for j = 1 , . . . , r , then A1 , . . . , Ak are ASD . Furthermore , O ’ Meara and Vinsonhaler ( 2006 ) observe that each block can be written as Bij = λijI +Nij where Nij is nilpotent , so A1 , . . . , Ak are ASD iff N1j , . . . , Nkj are for all j . So the transition matrices of the automaton can be rewritten in the above form , and the problem of converting an automaton to one that is ASD is reduced to the problem of converting an automaton with nilpotent transition matrices ( equivalently , an automaton recognizing a finite language ) to one that is ASD ( possibly with more states ) . See Appendix B for one such construction . This means that if we want to learn representations of multisets over a finite alphabet Σ , it suffices to constrain the transition matrices to be complex diagonal , possibly with more states . Unfortunately , the best construction we know of ( Appendix B ) increases the number of states by a lot . But this does not in any way prevent the use our representation ; we can choose however many states we want , and it ’ s an empirical question whether the number of states is enough to learn good representations . The following two sections look at two practical applications of our representation .

Summary: This work presents an encoding approach for unordered set input to neural networks. The authors base their approach on weighted finite automata, where in order to absorb unordered sets, they enforce multiplicative commutativity on transition matrices by approximating them as complex diagonal matrices. The authors furthermore provide mathematical references and results to derive bounds for their approximation. They show that positional encoding in Transformer network can be seen as a special case of their multiset encoding scheme, which also generalizes DeepSets encoding from real to complex numbers.