Distribution-Based Invariant Deep Networks for Learning Meta-Features

Recent advances in deep learning from probability distributions successfully achieve classification or regression from distribution samples , thus invariant under permutation of the samples . The first contribution of the paper is to extend these neural architectures to achieve invariance under permutation of the features , too . The proposed architecture , called DIDA , inherits the NN properties of universal approximation , and its robustness with respect to Lipschitz-bounded transformations of the input distribution is established . The second contribution is to empirically and comparatively demonstrate the merits of the approach on two tasks defined at the dataset level . On both tasks , DIDA learns meta-features supporting the characterization of a ( labelled ) dataset . The first task consists of predicting whether two dataset patches are extracted from the same initial dataset . The second task consists of predicting whether the learning performance achieved by a hyper-parameter configuration under a fixed algorithm ( ranging in k-NN , SVM , logistic regression and linear SGD ) dominates that of another configuration , for a dataset extracted from the OpenML benchmarking suite . On both tasks , DIDA outperforms the state of the art : DSS and DATASET2VEC architectures , as well as the models based on the hand-crafted meta-features of the literature . 1 INTRODUCTION . Deep networks architectures , initially devised for structured data such as images ( Krizhevsky et al. , 2012 ) and speech ( Hinton et al. , 2012 ) , have been extended to enforce some invariance or equivariance properties ( Shawe-Taylor , 1993 ) for more complex data representations . Typically , the network output is required to be invariant with respect to permutations of the input points when dealing with point clouds ( Qi et al. , 2017 ) , graphs ( Henaff et al. , 2015 ) or probability distributions ( De Bie et al. , 2019 ) . The merit of invariant or equivariant neural architectures is twofold . On the one hand , they inherit the universal approximation properties of neural nets ( Cybenko , 1989 ; Leshno et al. , 1993 ) . On the other hand , the fact that these architectures comply with the requirements attached to the data representation yields more robust and more general models , through constraining the neural weights and/or reducing their number . Related works . Invariance or equivariance properties are relevant to a wide range of applications . In the sequence-to-sequence framework , one might want to relax the sequence order ( Vinyals et al. , 2016 ) . When modelling dynamic cell processes , one might want to follow the cell evolution at a macroscopic level , in terms of distributions as opposed to , a set of individual cell trajectories ( Hashimoto et al. , 2016 ) . In computer vision , one might want to handle a set of pixels , as opposed to a voxellized representation , for the sake of a better scalability in terms of data dimensionality and computational resources ( De Bie et al. , 2019 ) . Neural architectures enforcing invariance or equivariance properties have been pioneered by ( Qi et al. , 2017 ; Zaheer et al. , 2017 ) for learning from point clouds subject to permutation invariance or equivariance . These have been extended to permutation equivariance across sets ( Hartford et al. , 2018 ) . Characterizations of invariance or equivariance under group actions have been proposed in the finite ( Gens & Domingos , 2014 ; Cohen & Welling , 2016 ; Ravanbakhsh et al. , 2017 ) or infinite case ( Wood & Shawe-Taylor , 1996 ; Kondor & Trivedi , 2018 ) . On the theoretical side , ( Maron et al. , 2019a ; Keriven & Peyré , 2019 ) have proposed a general characterization of linear layers enforcing invariance or equivariance properties with respect to the whole permutation group on the feature set . The universal approximation properties of such architectures have been established in the case of sets ( Zaheer et al. , 2017 ) , point clouds ( Qi et al. , 2017 ) , equivariant point clouds ( Segol & Lipman , 2019 ) , discrete measures ( De Bie et al. , 2019 ) , invariant ( Maron et al. , 2019b ) and equivariant ( Keriven & Peyré , 2019 ) graph neural networks . The approach most related to our work is that of ( Maron et al. , 2020 ) , handling point clouds and presenting a neural architecture invariant w.r.t . the ordering of points and their features . In this paper , the proposed distribution-based invariant deep architecture ( DIDA ) extends ( Maron et al. , 2020 ) as it handles ( discrete or continuous ) probability distributions instead of point clouds . This enables to leverage the topology of the Wasserstein distance to provide more general approximation results , covering ( Maron et al. , 2020 ) as a special case . Motivations . A main motivation for DIDA is the ability to characterize datasets through learned meta-features . Meta-features , aimed to represent a dataset as a vector of characteristics , have been mentioned in the ML literature for over 40 years , in relation with several key ML challenges : ( i ) learning a performance model , predicting a priori the performance of an algorithm ( and the hyperparameters thereof ) on a dataset ( Rice , 1976 ; Wolpert , 1996 ; Hutter et al. , 2018 ) ; ( ii ) learning a generic model able of quick adaptation to new tasks , e.g . one-shot or few-shot , through the so-called meta-learning approach ( Finn et al. , 2018 ; Yoon et al. , 2018 ) ; ( iii ) hyper-parameter transfer learning ( Perrone et al. , 2018 ) , aimed to transfer the performance model learned for a task , to another task . A large number of meta-features have been manually designed along the years ( Muñoz et al. , 2018 ) , ranging from sufficient statistics to the so-called landmarks ( Pfahringer et al. , 2000 ) , computing the performance of fast ML algorithms on the considered dataset . Meta-features , expected to describe the joint distribution underlying the dataset , should also be inexpensive to compute . The learning of meta-features has been first tackled by ( Jomaa et al. , 2019 ) to our best knowledge , defining the DATASET2VEC representation . Specifically , DATASET2VEC is provided two patches of datasets , ( two subsets of examples , described by two ( different ) sets of features ) , and is trained to predict whether those patches are extracted from the same initial dataset . Contributions . The proposed DIDA approach extends the state of the art ( Maron et al. , 2020 ; Jomaa et al. , 2019 ) in two ways . Firstly , it is designed to handle discrete or continuous probability distributions , as opposed to point sets ( Section 2 ) . As said , this extension enables to leverage the more general topology of the Wasserstein distance as opposed to that of the Haussdorf distance ( Section 3 ) . This framework is used to derive theoretical guarantees of stability under bounded distribution transformations , as well as universal approximation results , extending ( Maron et al. , 2020 ) to the continuous setting . Secondly , the empirical validation of the approach on two tasks defined at the dataset level demonstrates the merit of the approach compared to the state of the art ( Maron et al. , 2020 ; Jomaa et al. , 2019 ; Muñoz et al. , 2018 ) ( Section 4 ) . Notations . �1 ; m� denotes the set of integers { 1 , . . .m } . Distributions , including discrete distributions ( datasets ) are noted in bold font . Vectors are noted in italic , with x [ k ] denoting the k-th coordinate of vector x . 2 DISTRIBUTION-BASED INVARIANT NETWORKS FOR META-FEATURE LEARNING . This section describes the core of the proposed distribution-based invariant neural architectures , specifically the mechanism of mapping a point distribution onto another one subject to sample and feature permutation invariance , referred to as invariant layer . For the sake of readability , this section focuses on the case of discrete distributions , referring the reader to Appendix A for the general case of continuous distributions . 2.1 INVARIANT FUNCTIONS OF DISCRETE DISTRIBUTIONS . Let z= { ( xi , yi ) ∈ Rd , i ∈ �1 ; n� } denote a dataset including n labelled samples , with xi ∈ RdX an instance and yi ∈ RdY the associated multi-label . With dX and dY respectively the dimensions of the instance and label spaces , let d def.= dX +dY . By construction , z is invariant under permutation on the sample ordering ; it is viewed as an n-size discrete distribution 1n �n i=1 δzi in Rd with δzi the Dirac function at zi . In the following , Zn ( Rd ) denotes the space of such n-size point distributions , with Z ( Rd ) def.= ∪nZn ( Rd ) the space of distributions of arbitrary size . Let G def.= SdX × SdY denote the group of permutations independently operating on the feature and label spaces . For σ = ( σX , σY ) ∈ G , the image σ ( z ) of a labelled sample is defined as ( σX ( x ) , σY ( y ) ) , with x = ( x [ k ] , k ∈ �1 ; dX� ) and σX ( x ) def.= ( x [ σX ( k ) ] , k ∈ �1 ; dX� ) . For simplicity and by abuse of notations , the operator mapping a distribution z = ( zi , i ∈ �1 ; n� ) to { σ ( zi ) } def.= σ�z is still denoted σ . Let Z ( Ω ) denote the space of distributions supported on some domain Ω ⊂ Rd , with Ω invariant under permutations in G. The goal of the paper is to define and train deep architectures , implementing functions ϕ on Z ( Ω ⊂ Rd ) that are invariant under G , i.e . such that ∀σ ∈ G , ϕ ( σ�z ) = ϕ ( z ) 1 . By construction , a multi-label dataset is invariant under permutations of the samples , of the features , and of the multi-labels . Therefore , any meta-feature , that is , a feature describing a multi-label dataset , is required to satisfy the above sample and feature permutation invariance properties . 2.2 DISTRIBUTION-BASED INVARIANT LAYERS . The building block of the proposed architecture , the invariant layer meant to satisfy the feature and label invariance requirements , is defined as follows , taking inspiration from ( De Bie et al. , 2019 ) . Definition 1 . ( Distribution-based invariant layers ) Let an interaction functional ϕ : Rd×Rd → Rr be G-invariant : ∀σ ∈ G , ∀ ( z1 , z2 ) ∈ Rd × Rd , ϕ ( z1 , z2 ) = ϕ ( σ ( z1 ) , σ ( z2 ) ) . The distribution-based invariant layer fϕ is defined as fϕ : z = ( zi ) i∈�1 ; n� ∈ Z ( Rd ) �→ fϕ ( z ) def.= 1 n n� j=1 ϕ ( z1 , zj ) , . . . , 1 n n� j=1 ϕ ( zn , zj ) ∈ Z ( Rr ) . ( 1 ) By construction , fϕ is G-invariant if ϕ is G-invariant . The construction of fϕ is extended to the general case of possibly continuous probability distributions by essentially replacing sums by integrals ( Appendix A ) Remark 1 . ( Varying dimensions dX and dY ) . Both in practice and in theory , it is important that fϕ layers ( in particular the first layer of the neural architecture ) handle datasets of arbitrary number of features dX and number of multi-labels dY . The proposed approach , used in the experiments ( Section 4 ) , is to define ϕ as follows . Letting z = ( x , y ) and z� = ( x� , y� ) be two samples in RdX × RdY , let u be defined from R4 onto Rt , consider the sum of u ( x [ k ] , x� [ k ] , y [ � ] , y� [ � ] ) for k ranging in �1 ; dX� and � in �1 ; dY � , and apply mapping v from Rt to Rr on this sum : ϕ ( z , z� ) = v � dX� k=1 dY� �=1 u ( x [ k ] , x� [ k ] , y [ � ] , y� [ � ] ) � ( 2 ) Likewise , by construction ϕ is G-invariant , i.e . it is invariant to both feature and label permutations . As shown in Section 4 , this invariance property is instrumental to a good empirical performance . Remark 2 . ( Varying sample size n ) . By construction , fϕ is defined on Z ( Rd ) = ∪nZn ( Rd ) ( independent of n ) , such that it supports inputs of arbitrary cardinality n. Remark 3 . ( Discussion w.r.t . ( Maron et al. , 2020 ) ) The above definition of fϕ is based on the aggregation of pairwise terms ϕ ( zi , zj ) . The motivation for using a pairwise ϕ is twofold . On the one hand , capturing local sample interactions allows to create more expressive architectures , which is important to improve the performance on some complex data sets , as illustrated in the experiments ( Section 4 ) . On the other hand , interaction functionals are crucial to design universal architectures 1As opposed to G-equivariant functions that are characterized by ∀σ ∈ G , ϕ ( σ�z ) = σ�ϕ ( z ) ( Appendix C , theorem 2 ) . The proposed theoretical framework relies on the Wasserstein distance ( corresponding to the convergence in law of probability distributions ) , which enables to compare distributions with varying number of points or even with continuous densities . In contrast , Maron et al . ( 2020 ) do not use interaction functionals , and establish the universality of their DSS architecture for fixed dimension d and number of points n. Moreover , DSS happens to resort to max pooling operators , discontinuous w.r.t . the Wasserstein topology ( see Remark 6 ) . Two particular cases are when ϕ only depends on its first or second input : ( i ) if ϕ ( z , z� ) = ψ ( z� ) , then fϕ computes a global “ moment ” descriptor of the input , as fϕ ( z ) = 1n �n j=1 ψ ( zj ) ∈ Rr . ( ii ) if ϕ ( z , z� ) = ξ ( z ) , then fϕ transports the input distribution via ξ , as fϕ ( z ) = { ξ ( zi ) , i ∈ �1 ; n� } ∈ Z ( Rr ) . This operation is referred to as a push-forward . Remark 4 . ( Localized computation ) In practice , the quadratic complexity of fϕ w.r.t . the number n of samples can be reduced by only computing ϕ ( zi , zj ) for pairs zi , zj sufficiently close to each other . Layer fϕ thus extracts and aggregates information related to the neighborhood of the samples . Remark 5 . ( Link to kernels ) The use of an interaction functional ϕ is inspired from kernel ideas , albeit with significant differences : ( i ) in fϕ ( zi ) , the detail of the pairwise interactions ϕ ( zi , zj ) is lost through averaging ; ( ii ) ϕ takes into account labels ; ( iii ) ϕ is learnt . Further work will be devoted to investigating the properties of the fϕ ( zi ) matrix .

Summary: The paper presents a neural network layer designed to process distribution samples that is invariant to permutations of the samples and the features. The proposed method is compared empirically to DSS, which achieves the same types of invariance but is restricted to point sets rather than discrete or continuous probability distributions. The two tasks used for the empirical evaluation in the paper are: a) patch identification (are two blocks of data extracted from the same original dataset?) and b) model configuration assessment (is one configuration of a learning algorithm going to produce a more accurate model for a particular dataset than another one?). On the first task, the paper compares to models built using Dataset2Vec embeddings as well as DSS. On the second task, the paper compares to handcrafted features as well as DSS. In both tasks, the proposed method produces more accurate predictors than DSS, etc. The paper also has some theoretical results regarding the universality of the proposed architecture and its robustness w.r.t. Lipschitz-bounded transformations.