Abelian Neural Networks

1 INTRODUCTION . The vector representations of words called word2vec ( Mikolov et al. , 2013a ; b ) trained only on large unlabeled text data are known to capture linear regularities between words . For example , vec ( “ king ” ) −vec ( “ man ” ) +vec ( “ woman ” ) results in the most similar vector to vec ( “ queen ” ) . Similar results have been observed not only in other word embeddings ( Mnih & Kavukcuoglu , 2013 ; Pennington et al. , 2014 ) but also in various embedding spaces , such as combined embeddings of text and image ( Kiros et al. , 2014 ) , emoji embeddings ( Eisner et al. , 2016 ) , latent representation of deep convolutional generative adversarial networks ( Radford et al. , 2016 ) , and feature space of pretrained image models ( Upchurch et al. , 2017 ) . Although the reports are interesting and attractive , such approaches have shortcomings . Since those methods of learning from unlabelled data usually do not explicitly incorporate learning analogical structure , there is no guarantee that the acquired embedding space has linear relations between instance pairs even if training itself works well . Even when an embedding space captures some kinds of analogies , it might not work for other kinds of analogies . Indeed , word2vec representation works well for inflectional analogies ( 68.22 % accuracy ) but poorly for encyclopedic analogies ( 7.11 % accuracy ) in our preliminary experiments ( see Table 3 in Section 4.2 ) . In such a case , it is quite difficult to tune the training algorithm for certain kinds of analogies you want to use . To alleviate these issues , we propose to directly learn analogical relations on the embedding space from labeled data . One challenge in learning analogy in a supervised manner is how to model analogical functions . A naive way to do this is to train two separate models corresponding to addition and subtraction , respectively ; however , it does not reflect the analogical structure and might be inefficient . In this work , we propose an Abelian group network to incorporate an analogical inductive bias into a neural network . The proposed network is designed to satisfy the Abelian group condition by using an invertible neural network . We also show that the Abelian group network is a universal approximator of smooth Abelian group operations . Since the inverse element in the Abelian group network and its gradient are analytically computable , we can train it for analogy tasks by common techniques for deep learning , such as stochastic gradient descent . As a side effect of the algebraic structure , we can construct a permutation invariant function , i.e. , a function for multisets , by repeatedly composing an Abelian group operation for multiple inputs . Multiset models can handle inputs of different sizes , and it is important for the models to automatically generalize between different size inputs , especially from small to large ones . However , existing multiset models ( Zaheer et al. , 2017 ; Qi et al. , 2017 ) have no such theoretical guarantee . On the other hand , our multiset models naturally induce the size-generalization capability because the output for larger inputs can be written as the composition of small elements . Further , we show that a necessary and sufficient condition for the composed function being permutation invariant is that the binary operation forms an Abelian semigroup , and we propose an Abelian semigroup network , by using the characterization of associative symmetric polynomials . 2 PRELIMINARIES . 2.1 DEFINITIONS . In this section , let us introduce some basic notations and important definitions that will play a key role in this work . 2.1.1 BASIC NOTATIONS . By N , we represent the set of the natural numbers including 0 . We denote a vector by a bold symbol , e.g. , x . Let x ∈ Rd be a d-dimensional vector . We represent the i-th element ( 1 ≤ i ≤ d ) of x by xi . For 1 ≤ k ≤ d , x≤k ∈ Rk is the k-dimensional vector ( x1 , . . . xk ) and x < k ∈ Rk−1 is the ( k − 1 ) -dimensional vector ( x1 , . . . xk−1 ) . We denote the elementwise product of two vectors x , y ∈ Rd by x ⊗ y , such that ( x ⊗ y ) i = xiyi and the elementwise division of two vectors x ∈ Rd , y ∈ ( R \ { 0 } ) d by x y , such that ( x y ) i = xi/yi . By ‖ · ‖ , we represent the L2 ( Euclidean ) norm . 2.1.2 MULTISET AND PERMUTATION INVARIANCE . Here , we use X and Y to describe some domains , which are typically Euclidean spaces , i.e. , X = Rd1 and Y = Rd2 . We denote the set of multisets over X by NX . We use { x1 , . . .xn } ∈ NX to describe a multiset composed of x1 , . . . , xn ∈ X ( any confusion with sets is not problematic in this paper ) . The cardinality of a multiset is the number of elements with multiplicity and is expressed by | · | , e.g. , | { 1 , 2 , 2 , 3 } | = 4 . For n ∈ N , a symmetric group Sn is the set of all n ! bijective functions σ : { 1 , 2 , . . . , n } → { 1 , 2 , . . . , n } . For a permutation σ ∈ Sn andX ∈ X , σ ·X is defined such that ( σ ·X ) i = Xσ ( i ) for all i ∈ { 1 , . . . , n } . A function f : Xn → Y is said to be permutation invariant if for any X ∈ Xn and for any permutation σ ∈ Sn , f ( σ ·X ) = f ( X ) holds . This concept can be extended to functions that take vectors of different dimensions . Namely , a function f : ⋃ k∈N X k → Y is called permutation invariant if for any k ∈ N , for anyX ∈ X k and for any permutation σ ∈ Sk , f ( σ ·X ) = f ( X ) holds . When f : ⋃ k∈N X k → Y is permutation invariant , it can be also viewed as a function that takes multisets as input . For notation simplicity , we sometimes use the same symbol to express the multiset function : f : NX → Y . 2.1.3 UNIVERSALITY . Universality is an important theoretical property of neural networks ’ expressive power . Let X and Y be an input domain and an output domain , respectively . We consider a modelM and a class of target functions F , both of which are sets of functions X → Y . The modelM is a sup-universal approximator of F if for any target function f∗ ∈ F , for any > 0 , and for any compact subset K ⊂ X , there exists a function f ∈M such that sup x∈K ‖f ( x ) − f∗ ( x ) ‖ < . ( 1 ) If not noted otherwise , universality refers to the sup-universal property . 2.1.4 BASIC ALGEBRA . Here , we introduce the basic definition of important algebraic structures in this study . Let G be a set and ◦ : G × G → G be a binary operation . Below , we review four properties to define Abelian semigroups and groups . Associativity For any x , y , z ∈ G , ( x ◦ y ) ◦ z = x ◦ ( y ◦ z ) . Identity Element There exists an element e ∈ G , called the identity element , such that for any x ∈ G , x ◦ e = e ◦ x = x. Inverse Element For any x ∈ G , there exists an element x−1 ∈ G , called the inverse element of x , such that x ◦ x−1 = x−1 ◦ x = e. Commutativity For any x , y ∈ G , x ◦ y = y ◦ x . Table 1 shows which properties are required in each algebraic structure . A semigroup only requires associativity to the binary operation . A group is a semigroup with an identity element and inverse elements . An Abelian ( semi ) group is a ( semi ) group with commutativity . 2.2 INVERTIBLE NEURAL NETWORKS . Invertible neural networks are neural networks that approximate invertible functions Rd → Rd . Here , we review some existing studies for multi-dimensional case , i.e. , d ≥ 2 , and singledimensional case , i.e. , d = 1 . 2.2.1 NORMALIZING FLOWS . Multi-dimensional invertible neural networks have been studied mainly in the context of normalizing flows ( Tabak & Vanden-Eijnden , 2010 ) , which iteratively apply invertible functions to a simple original probability distribution to express complex probability distributions ( Kobyzev et al. , 2020 ; Papamakarios et al. , 2019 ) . There have been many variants proposed including residual flows ( Behrmann et al. , 2019 ) , neural ODEs ( Chen et al. , 2018 ) , and autoregressive flows ( Kingma et al. , 2017 ) . Here we review affine coupling flows ( Dinh et al. , 2015 ) , one of the most popular models with parallelizable efficient inverse computation . Each layer of the affine coupling flows maps x = ( x1 , . . . , xd ) ∈ Rd to y = ( y1 , . . . , yd ) ∈ Rd such that { y≤k = x≤k , y > k = x > k ⊗ exp ( α ( x≤k ) ) + β ( x≤k ) , ( 2 ) where exp is applied elementwise and α , β : Rk → Rd−k are trainable functions . The inverse is computed as follows : { x≤k = y≤k , x > k = ( y > k − β ( y≤k ) ) ⊗ exp ( −α ( y≤k ) ) . ( 3 ) They are used in many successful applications such as NICE ( Dinh et al. , 2015 ) , Real NVP ( Dinh et al. , 2017 ) , and Glow ( Kingma & Dhariwal , 2018 ) . Although the normalizing flows have a limited form of transform , they still admit universalities on certain classes of functions ( Teshima et al. , 2020 ) . The affine coupling flows are Lp-universal ( weaker condition of universality ) for C2-diffeomorphisms . Some more complex models , including deep sigmoidal flows ( Huang et al. , 2018 ) and sum-of-squares polynomial flows ( Jaini et al. , 2019 ) , are universal for C2-diffeomorphisms . 2.2.2 SINGLE-DIMENSIONAL INVERTIBLE NEURAL NETWORKS . For single-dimensional functions , invertibility is equivalent to strict monotonicity . Monotonic networks ( Sill , 1997 ) model strictly monotonic functions . The monotonically increasing version with K groups and Jk units for k-th group is as follows : f ( x ) = min 1≤k≤K max 1≤j≤Jk exp ( w̃ ( k , j ) ) · x+ b ( k , j ) , ( 4 ) where w̃ ( k , j ) , b ( k , j ) ∈ R are trainable parameters . The monotonic networks are a universal approximator for strictly monotonic differentiable functions . Monotonic rational-quadratic transforms ( Durkan et al. , 2019 ) are another universal model for the single-dimensional case .

Summary: This paper introduces a new kind of neural network for multisets of vector inputs. Whereas DeepSets uses a function h(x, y) = g(f(x) + f(y)), the proposed model uses h(x, y) = f^{-1}(f(x) + f(y)), where f is an invertible neural network. It applies this network to the problem of learning word analogies.