Deep processing of structured data

1 INTRODUCTION . Neural networks are one of the most powerful machine learning tools . However , in its basic form , neural networks can only process data represented as vectors with fixed size . Since very often the real data can not be directly represented in this form ( e.g . graphs , sequences , sets ) , one needs to construct a data specific architecture to process them . In natural language processing ( NLP ) , documents are represented as sequences with different lengths . Typical approach relies on embedding words into a vector space ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ) and processing them with use of recurrent ( Bahdanau et al. , 2014 ) or convolutional neural network ( Kim , 2014 ) . Since the number of words in a document varies , it is impossible to directly apply fully connected layers afterwards , e.g . for classification of machine translation ( Sundermeyer et al. , 2012 ; Cho et al. , 2014 ) . To do so , an intermediate pooling operator ( sum or max ) or zero padding are used to produce a fixed-length vectors ( Kalchbrenner & Blunsom , 2013 ; Sutskever et al. , 2014 ) . Alternatively , one may sum all word embeddings before applying recurrent network to make inputs equally sized , which is a basis of typical bag-of-words representation . Both approaches have difficulties in handling context of long documents , which was the reason of introducing attention models ( Bahdanau et al. , 2014 ) . Analogically , in computer vision , fully convolutional networks can process inputs of arbitrary shapes , but produced outputs have different sizes as well ( Ciresan et al. , 2011 ; Karpathy & Fei-Fei , 2015 ) . This is acceptable for image segmentation ( Ronneberger et al. , 2015 ) or inpainting ( Iizuka et al. , 2017 ) , but to classify such images , we need to produce fixed-length output vectors . This can be obtained by scaling images at preprocessing stage ( He et al. , 2015 ) or by applying pooling layer at the top of the network ( Long et al. , 2015 ; Maggiori et al. , 2016 ) . Clearly , every new data type causes nontrivial problems when processed with neural network . This is observed in the case of graphs ( Brin & Page , 1998 ; Frasconi et al. , 1998 ) . One option is to extract predefined features , e.g . fingerprints in cheminformatics ( Klekota & Roth , 2008 ; Ewing et al. , 2006 ) , which however requires extensive domain knowledge . Alternatively , graph recurrent or convolutional networks can be used , but then we need to deal again with outputs of different sizes ( Scarselli et al. , 2009 ; Li et al. , 2015 ; Hagenbuchner et al. , 2003 ; Tsoi et al. , 2003 ) . The aim of this paper is to construct a general approach for learning representation of structured objects . We replace pooling operation by our set aggregation network ( SAN ) , which maps an input set ( our representation of structured object ) to a fixed-length vector in high dimensional latent space , see Figure 1 . Our main theoretical result ( Theorem 3.1 ) shows that for latent space of sufficiently large dimension , SAN can learn a unique representation of every input set , which justifies this ap- proach from a theoretical perspective . We show that SAN can be easily implemented in typical deep learning frameworks . In consequences it is a promising alternative to typical pooling operation used in a wide range of architectures , such as recurrent or convolutional networks . We conduct extensive experiments on different types of structured data ( Section 4 ) . Its direct application to text and graph data allows to obtain results close to SOTA , by simpler networks with smaller number of parameters than competitive methods . Moreover , the combination of SAN with convolutional network outperforms typical pooling operation in the case of classifying images with different sizes ( section 5 ) . For the convenience of the reader , we describe the content of the paper . In the following section , we recall a typical way for processing structured data and next describe our approach . In Section 3 we give a theoretical justification of our model . Basic capabilities of SAN on various types of structured data is presented in Section 4 . Its practical usefulness is verified in the case of classifying images of various sizes in section 5 . 2 SET AGGREGATION NETWORK : CONSTRUCTION . Suppose that we want to process a structured data X = ( xi ) i ⊂ RD by a neural network . It can be for example a sequence of word embeddings , image with missing attributes or some graph structure . Typical approach relies on creating two networks combined with an intermediate pooling layer : X = ( xi ) i Ψ→ ( Ψxi ) i Pool→ Pool { Ψ ( xi ) : i } Φ→ RN . The first network Ψ : RD → RK is usually a convolutional or recurrent network , which transforms elements of X sequentially one-by-one and produces a set ( or sequence ) of vectors . Next , a pooling layer summarizes the response of Ψ and returns a single K-dimensional vector . A pooling layer , commonly implemented as sum or maximum operation , gives a vector representation of structured object . Finally , a network Φ : RK → RN maps resulting representation to the final output1 . A basic problem with the above pipeline lies in the pooling layer . This layer is often a bottleneck and can lead to the lose of meaningful information contained in the whole set structure . Our goal is to learn a representation of a set by replacing classical pooling operation by a specific neural network – set aggregation network ( SAN ) , see Figure 1 . SAN is a neural network T : RK → RM , which transforms a set2 of K-dimensional vectors into a single vector in M -dimensional latent space . The output neurons of T give a representation of a given set by aggregating information contained in a whole set and thus will be referred to as aggregative neurons . 1One can also create simpler architecture , where the sum or average pooling is applied directly to the set of word embeddings , which is analogical to the use of bag-of-words representation . 2One can always add an index coordinate to every vector to map a sequence to a set . Our basic idea comes from computer tomography , where Radon transform ( Radon , 1986 ; van Ginkel et al. , 2004 ) is used to reconstruct a function ( in practice the 2D or 3D image ) from the knowledge of its integration over all one-dimensional lines . A similar statement is given by the Cramer-Wold Theorem ( Cramér & Wold , 1936 ) , which says that for every two distinct measures one can find a one-dimensional projection which discriminates them . This implies that without loss of information we can process the set X ⊂ RK through its all one-dimensional projections vTX ⊂ R , where v ∈ RK is arbitrary . In consequence , we reduced the question of representing a multidimensional set to the characterization of one-dimensional sets . One can easily see ( for a more general statement see Section 3 ) that the one-dimensional set S ⊂ R can be retrieved from the knowledge of aggregated ReLU on its translations : b → ∑ i ReLU ( si + b ) , see Figure 2 . Summarizing the above reasoning , we obtain that the full knowledge of a set X ⊂ RK is given by the scalar function RK × R 3 ( v , b ) → ∑ i ReLU ( vTxi + b ) . Now , given M vectors vi ∈ RK and biases bi ∈ R , we obtain the fixed-size representation of the set X ⊂ RK as a point in RM given by [ ∑ i ReLU ( vT1 xi + b1 ) , . . . , ∑ i ReLU ( vTMxi + bM ) ] ∈ RM . We arrived at the formal definition of aggregative neuron : Definition 2.1 . Let τ : RK : → R be a given neuron ’ s activation function ( where as the default we take the ReLU ) . We extend the use of τ for finite sets X = ( xi ) i ⊂ RK by the formula : τ ( X ) : = ∑ i τ ( xi ) . ( 1 ) Every aggregative neuron computes a summarized activation of all set elements . A composition of M aggregative neurons gives M -dimensional vector representation of the input set . Parameters of aggregation neurons are trainable and thus we allow for learning a representation of structured objects . The final equation ( 1 ) in SAN is quite simple and can be easily added to neural network architecture since it reflects simultaneously the structure of neural networks and model natural representation of the set of points . Remark 2.1 . Observe that max-pooling is a special case of SAN . Clearly , for non-negative scalar data X = ( Xi ) ⊂ R and function τp ( x ) = xp , we have : τ−1 ( ∑ i τ ( xi ) ) → max i ( xi ) , as p→∞ . To obtain a maximum , we use τ as a activity function in aggregative neuron , which is followed by a layer with its inverse . By extending this scheme , we can get a maximum value for every coordinate . Additionally , to deal with negative numbers , we first take the exponent followed by logarithm after the aggregation . We explain how to implement SAN in typical deep learning frameworks . Let X = { X1 , . . . , Xn } be a batch , which consists of finite sets Xi ⊂ RK . While typical networks process vector by vector , SAN needs to compute a summary activation over all elements of Xi before passing the output to the subsequent network . To deal with sets of different sizes in one batch , we first transform every set Xi to the set of tuples Ti = { ( x , 1 ) : x ∈ Xi } . Next , we complement every Ti to have the same sizes by adding zero tuples ( x , 0 ) , where x ∈ RK can be an arbitrary vector . The second element of each tuple indicates whether a vector appears in a set or is a dummy vector . The set of tuples Ti can be understood as a uniform discrete measure with a support defined on a given set Xi . After the above operation every element from a batch is represented as a matrix of the same size . To compute the value of aggregative neuron of Xi , we apply 1D convolution to the first element of each tuple from Ti and multiply the outputs by their weights . Taking a sum , we get the activation of aggregative neuron τ for Xi : τ ( Xi ) = ∑ x∈Xi 1 · τ ( xi ) = ∑ ( x , f ) ∈Ti f · τ ( x ) . These operations are applied to every element Xi of a batch . Summarizing , adding SAN to existing network can be implemented in a few lines of codes .

Summary: The paper at hand describes an approach to enable neural networks to take arbitrary structured data (basically any data that is not easily represented as a fixed-dimensional vector) as input. The paper describes how such data can be represented as a set (e.g. a sequence is represented as a set of index + data) and then an auxiliary network called the set aggregating network (SAN)  is used to represent the data in a high dimensional latent space. In addition to the idea, the paper provides a theoretical analysis of the approach which shows that with a sufficiently high dimensional representation the network is able to learn a unique representation for each input example.