Abelian Neural Networks

1 INTRODUCTION . The vector representations of words called word2vec ( Mikolov et al. , 2013a ; b ) trained only on large unlabeled text data are known to capture linear regularities between words . For example , vec ( “ king ” ) −vec ( “ man ” ) +vec ( “ woman ” ) results in the most similar vector to vec ( “ queen ” ) . Similar results have been observed not only in other word embeddings ( Mnih & Kavukcuoglu , 2013 ; Pennington et al. , 2014 ) but also in various embedding spaces , such as combined embeddings of text and image ( Kiros et al. , 2014 ) , emoji embeddings ( Eisner et al. , 2016 ) , latent representation of deep convolutional generative adversarial networks ( Radford et al. , 2016 ) , and feature space of pretrained image models ( Upchurch et al. , 2017 ) . Although the reports are interesting and attractive , such approaches have shortcomings . Since those methods of learning from unlabelled data usually do not explicitly incorporate learning analogical structure , there is no guarantee that the acquired embedding space has linear relations between instance pairs even if training itself works well . Even when an embedding space captures some kinds of analogies , it might not work for other kinds of analogies . Indeed , word2vec representation works well for inflectional analogies ( 68.22 % accuracy ) but poorly for encyclopedic analogies ( 7.11 % accuracy ) in our preliminary experiments ( see Table 3 in Section 4.2 ) . In such a case , it is quite difficult to tune the training algorithm for certain kinds of analogies you want to use . To alleviate these issues , we propose to directly learn analogical relations on the embedding space from labeled data . One challenge in learning analogy in a supervised manner is how to model analogical functions . A naive way to do this is to train two separate models corresponding to addition and subtraction , respectively ; however , it does not reflect the analogical structure and might be inefficient . In this work , we propose an Abelian group network to incorporate an analogical inductive bias into a neural network . The proposed network is designed to satisfy the Abelian group condition by using an invertible neural network . We also show that the Abelian group network is a universal approximator of smooth Abelian group operations . Since the inverse element in the Abelian group network and its gradient are analytically computable , we can train it for analogy tasks by common techniques for deep learning , such as stochastic gradient descent . As a side effect of the algebraic structure , we can construct a permutation invariant function , i.e. , a function for multisets , by repeatedly composing an Abelian group operation for multiple inputs . Multiset models can handle inputs of different sizes , and it is important for the models to automatically generalize between different size inputs , especially from small to large ones . However , existing multiset models ( Zaheer et al. , 2017 ; Qi et al. , 2017 ) have no such theoretical guarantee . On the other hand , our multiset models naturally induce the size-generalization capability because the output for larger inputs can be written as the composition of small elements . Further , we show that a necessary and sufficient condition for the composed function being permutation invariant is that the binary operation forms an Abelian semigroup , and we propose an Abelian semigroup network , by using the characterization of associative symmetric polynomials . 2 PRELIMINARIES . 2.1 DEFINITIONS . In this section , let us introduce some basic notations and important definitions that will play a key role in this work . 2.1.1 BASIC NOTATIONS . By N , we represent the set of the natural numbers including 0 . We denote a vector by a bold symbol , e.g. , x . Let x ∈ Rd be a d-dimensional vector . We represent the i-th element ( 1 ≤ i ≤ d ) of x by xi . For 1 ≤ k ≤ d , x≤k ∈ Rk is the k-dimensional vector ( x1 , . . . xk ) and x < k ∈ Rk−1 is the ( k − 1 ) -dimensional vector ( x1 , . . . xk−1 ) . We denote the elementwise product of two vectors x , y ∈ Rd by x ⊗ y , such that ( x ⊗ y ) i = xiyi and the elementwise division of two vectors x ∈ Rd , y ∈ ( R \ { 0 } ) d by x y , such that ( x y ) i = xi/yi . By ‖ · ‖ , we represent the L2 ( Euclidean ) norm . 2.1.2 MULTISET AND PERMUTATION INVARIANCE . Here , we use X and Y to describe some domains , which are typically Euclidean spaces , i.e. , X = Rd1 and Y = Rd2 . We denote the set of multisets over X by NX . We use { x1 , . . .xn } ∈ NX to describe a multiset composed of x1 , . . . , xn ∈ X ( any confusion with sets is not problematic in this paper ) . The cardinality of a multiset is the number of elements with multiplicity and is expressed by | · | , e.g. , | { 1 , 2 , 2 , 3 } | = 4 . For n ∈ N , a symmetric group Sn is the set of all n ! bijective functions σ : { 1 , 2 , . . . , n } → { 1 , 2 , . . . , n } . For a permutation σ ∈ Sn andX ∈ X , σ ·X is defined such that ( σ ·X ) i = Xσ ( i ) for all i ∈ { 1 , . . . , n } . A function f : Xn → Y is said to be permutation invariant if for any X ∈ Xn and for any permutation σ ∈ Sn , f ( σ ·X ) = f ( X ) holds . This concept can be extended to functions that take vectors of different dimensions . Namely , a function f : ⋃ k∈N X k → Y is called permutation invariant if for any k ∈ N , for anyX ∈ X k and for any permutation σ ∈ Sk , f ( σ ·X ) = f ( X ) holds . When f : ⋃ k∈N X k → Y is permutation invariant , it can be also viewed as a function that takes multisets as input . For notation simplicity , we sometimes use the same symbol to express the multiset function : f : NX → Y . 2.1.3 UNIVERSALITY . Universality is an important theoretical property of neural networks ’ expressive power . Let X and Y be an input domain and an output domain , respectively . We consider a modelM and a class of target functions F , both of which are sets of functions X → Y . The modelM is a sup-universal approximator of F if for any target function f∗ ∈ F , for any > 0 , and for any compact subset K ⊂ X , there exists a function f ∈M such that sup x∈K ‖f ( x ) − f∗ ( x ) ‖ < . ( 1 ) If not noted otherwise , universality refers to the sup-universal property . 2.1.4 BASIC ALGEBRA . Here , we introduce the basic definition of important algebraic structures in this study . Let G be a set and ◦ : G × G → G be a binary operation . Below , we review four properties to define Abelian semigroups and groups . Associativity For any x , y , z ∈ G , ( x ◦ y ) ◦ z = x ◦ ( y ◦ z ) . Identity Element There exists an element e ∈ G , called the identity element , such that for any x ∈ G , x ◦ e = e ◦ x = x. Inverse Element For any x ∈ G , there exists an element x−1 ∈ G , called the inverse element of x , such that x ◦ x−1 = x−1 ◦ x = e. Commutativity For any x , y ∈ G , x ◦ y = y ◦ x . Table 1 shows which properties are required in each algebraic structure . A semigroup only requires associativity to the binary operation . A group is a semigroup with an identity element and inverse elements . An Abelian ( semi ) group is a ( semi ) group with commutativity . 2.2 INVERTIBLE NEURAL NETWORKS . Invertible neural networks are neural networks that approximate invertible functions Rd → Rd . Here , we review some existing studies for multi-dimensional case , i.e. , d ≥ 2 , and singledimensional case , i.e. , d = 1 . 2.2.1 NORMALIZING FLOWS . Multi-dimensional invertible neural networks have been studied mainly in the context of normalizing flows ( Tabak & Vanden-Eijnden , 2010 ) , which iteratively apply invertible functions to a simple original probability distribution to express complex probability distributions ( Kobyzev et al. , 2020 ; Papamakarios et al. , 2019 ) . There have been many variants proposed including residual flows ( Behrmann et al. , 2019 ) , neural ODEs ( Chen et al. , 2018 ) , and autoregressive flows ( Kingma et al. , 2017 ) . Here we review affine coupling flows ( Dinh et al. , 2015 ) , one of the most popular models with parallelizable efficient inverse computation . Each layer of the affine coupling flows maps x = ( x1 , . . . , xd ) ∈ Rd to y = ( y1 , . . . , yd ) ∈ Rd such that { y≤k = x≤k , y > k = x > k ⊗ exp ( α ( x≤k ) ) + β ( x≤k ) , ( 2 ) where exp is applied elementwise and α , β : Rk → Rd−k are trainable functions . The inverse is computed as follows : { x≤k = y≤k , x > k = ( y > k − β ( y≤k ) ) ⊗ exp ( −α ( y≤k ) ) . ( 3 ) They are used in many successful applications such as NICE ( Dinh et al. , 2015 ) , Real NVP ( Dinh et al. , 2017 ) , and Glow ( Kingma & Dhariwal , 2018 ) . Although the normalizing flows have a limited form of transform , they still admit universalities on certain classes of functions ( Teshima et al. , 2020 ) . The affine coupling flows are Lp-universal ( weaker condition of universality ) for C2-diffeomorphisms . Some more complex models , including deep sigmoidal flows ( Huang et al. , 2018 ) and sum-of-squares polynomial flows ( Jaini et al. , 2019 ) , are universal for C2-diffeomorphisms . 2.2.2 SINGLE-DIMENSIONAL INVERTIBLE NEURAL NETWORKS . For single-dimensional functions , invertibility is equivalent to strict monotonicity . Monotonic networks ( Sill , 1997 ) model strictly monotonic functions . The monotonically increasing version with K groups and Jk units for k-th group is as follows : f ( x ) = min 1≤k≤K max 1≤j≤Jk exp ( w̃ ( k , j ) ) · x+ b ( k , j ) , ( 4 ) where w̃ ( k , j ) , b ( k , j ) ∈ R are trainable parameters . The monotonic networks are a universal approximator for strictly monotonic differentiable functions . Monotonic rational-quadratic transforms ( Durkan et al. , 2019 ) are another universal model for the single-dimensional case .

Summary: The authors introduce Abelian group networks (AGN) that explicitly model the operational relation between elements in an Abelian group. The authors prove that the design has the ability to model such relations and present feasible neural network realizations. The authors also prove the ability of AGN to learn representations of multisets. The authors design experiments to test AGN's ability to model word analogy by viewing the "a:b = c:d" relation as $c - a + b$. The results seem to suggest that AGN outperforms MLP baselines always and word vectors in some cases. However, closer examinations reveal concerns about their strength and validity to support the efficacy of AGN.