Distribution-Based Invariant Deep Networks for Learning Meta-Features
Authors: Gwendoline de Bie
Herilalaina Rakotoarison
Gabriel Peyré
Michèle Sebag
Venue:  (2021)
Keywords: invariant neural networks
universal approximation
meta-feature learning

Abstract: Recent advances in deep learning from probability distributions successfully achieve classification or regression from distribution samples, thus invariant under permutation of the samples. The first contribution of the paper is to extend these neural architectures to achieve invariance under permutation of the features, too.  The proposed architecture, called Dida, inherits the NN properties of universal approximation, and its robustness with respect to Lipschitz-bounded transformations of the input distribution is established. The second contribution is to empirically and comparatively demonstrate the merits of the approach on two tasks defined at the dataset level. On both tasks, Dida learns meta-features supporting the characterization of a (labelled) dataset. The first task consists of predicting whether two dataset patches are extracted from the same initial dataset. The second task consists of predicting whether the learning performance achieved by a hyper-parameter configuration under a fixed algorithm (ranging in k-NN, SVM, logistic regression and linear SGD) dominates that of another configuration, for a dataset extracted from the OpenML benchmarking suite. On both tasks, Dida outperforms the state of the art: DSS and Dataset2Vec architectures, as well as the models based on the hand-crafted meta-features of the literature. 

Under review as a conference paper at ICLR 2021
DISTRIBUTION-BASER INVARIANT DEEP NETWORKS FOR LEARNING META-FEATURES
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in deep learning from probability distributions successfully achieve classification or regression from distribution samples, thus invariant under permutation of the samples. The first contribution of the paper is to extend these neural architectures achieve invariance under permutation of the features, too. The proposed architecture, called DIDA, inherits the NN properties of universal approximation, and its robustness with respect to ppsschitz-boundee transforma- tions of the input distribution is cstablished. The second contribution is to em- pirically and comparatively demonstrate the merits of the approach two tasks defined the dataset level. On both tasks, DIDA learns meta-feaue meta-features supporting (labelled) dataset. The first task consists predicting
the characterization whether two dataset patches are extracted from the same initial dataset. The sec- ond task consists of predicting whether the learning performance achieved by typer-parameter configuration under fixed algorithm (ranging in k-NN, SVM, logistic regression and linear SGD) dominates that of another configuration, for a dataset extracted from the OpenML. benchmarking suite. On both tasks, DIDA outperforms the state of the art: DSS and DATASET2VIC architectures, well as the models based on the hand-crafted meta- vetaureetures of the literature.
INTRODUCTION
Deep networks architectures, initially devised for structured data such images (Krizhevsky et al., 2012) and speech (Hinton et al., 2012), have been extended enforce some invariance or equivari- ance properties (Shawe-Taylor, 1993) for more complex data representations, Typically, network output is required to be invariant with respect to permutations of the input points when dealing with point clouds (Qi et al., 2017), graphs (Henaff et al., 2015) prohability distributions (De Bie et al., 2019). The merit of invariant equivariant neural architectures twofold. On the one hand, they inherit the universal approximation properties of neural nets (Cybenko. 1989; Leshno et al., 1993). On the other hand, the fact that these architectures comply with the requirements attached to the data representation yields more robust and more general models, through constraining the neural weights and/or reducing their number.
Related works. Invariance or equivariance properties are relevant to wide range of applications. In the sequence-to-sequeeee framework, one might want to relax the sequence order (Vinyals et al., 2016). When modelling dynamic cell processes, one might want to follow the cell evolution at macroscopic level, in terms of distributions as opposed to, set of individual cell trajectories (Hashimoto al.. 2016). computer vision, one might want to handle a set of pixels, as opposed to a voxellized representation, for the sake of better scalability in terms data dimensionality and computational resources (De Bie et al., 2019).
Neural architectures enforcing invariance equivariance properties have been pioneered by (Qi al., 2017; Zaheer et al., 2017) for lcarning from point clouds subject to permutation invariance or equivariance. These have been extended to permutation equivariance across sets (Hartford et al., 2018). Characterizations of invariance or equivariance under group actions have been proposed in the finite (Gens & Domingos, 2014; Cohen & Welling, 2016; Ravanbakhsh et al., 2017) or infinite case (Wood& & Shawe-Taylo 1996; Kondor & Trivedi, 2018).

---

Under review as a conference paper at ICIR 2021
On the theoretical side, (Maron et al., 2019a; Keriven & Peyré, 2019) have proposed a general char- acterization of linear layers enforcing invariance or equivariance properties with respect to the whole permutation group on the feature set. The universal approximation properties of such architectures have been established in the case of sets (Zaheer et al., 2017). point clouds (Qi et al., 2017), equivari- ant point clouds (Segol Lipman, 2019), discrete measures (De Bie et al., 2019), invariant (Maron et al., 2019b) and equivariant (Keriven & Peyré. 2019) graph neural networks. The approach most related to our work is that of (Maron et al., 2020), handling point clouds and presenting neural architecture invariant w.r.t. the ordering of points and their features. In this paper, the proposed distribution-buu -bassd invariant deep architecture (DIDA) extends (Maron al.. 2020) as handles (discrete or continuous) probability distributions instead of point clouds. This enables to leverage the topology of the Wasserstein distance to provide more general approximation results, covering (Maron et al., 2020) special case.
Motivations. A main motivation for DIDA the ability to characterize datasets through learned meta-features. Meta-features, aimed to represent a dataset as a vector of characteristics, have been mentioned in the ML literature for over 40 years, in relation with several key ML challenges: (i) learning performance model, predicting priori the performance of an algorithm (and the hyper- parameters thereof) on dataset (Rice. 1976; Wolpert, 1996; Hutter et al., 2018); (ii) learning a generic model able of quick adaptation to new tasks, c.g. one-shot or fow-shot, through the so-called meta-learning approach (Finn et al., 2018; Yoon et al., 2018); (iii) hyper-paramett transfer learning (Perrone al., 2018) 8), aimed to transfer the performance model learned for task, to another task. large number of meta-features have been manually designed along the years (Muñoz et al., 2018), ranging from sufficient statistics to the so-called landmarks (Pfahringer et al., 2000), computing the performance of fast ML algorithms on the considered dataset. Meta-features, expected to descrihe the joint distribution underlying the dataset, should also be inexpensive to compute. The learning of meta-features has been first tackled by (Jomaa et al., 2019) DATASET2VEC representation. Specifically, DATASET2VEC (two subsets of examples, described by two (different) sets of features), and is trained to predict whether those patches are extracted from the same initial dataset.
best knowledge, defining the provided two patches of datasets,
Contributions. The proposed DIDA approach extends the state of the (Maron et al., 2020; Jomaa et al., 2019) in two ways. Firstly, it is designed to handle discrete or continuous probability distributions, as opposed to point sets (Section 2). As said, this extension enables to leverage the more general topology of the Wasserstein distance opposed to that of the Haussdorf distance (Section 3). This framework is used to derive theoretical guarantees of stability under bounded distribution transformations, as well as universal approximation results, extending (Maron et al., 2020) to the continuous setting. Secondly, the empirical validation of the approach on two tasks defined at the dataset level demonstrates the merit of the approach compared to the state of the art (Maron et al., 2020; Jomaa et al., 2019; Muñoz et al., 2018) (Section 4).
Notations. [L: m| denotes the set of integers (l....m). m). Distributions, including discrete distri- butions (datasets) are noted in bold font. Vectors are noted in italic, with æ [bk] denoting the k-th coordinate of vector x.
DISTRIBUTION-BASS INVARIANT NETWORKS FOR META-FEATURE LEARNING
This section describes the core of the proposed distribution-hasee invariant neural architectures. specifically the mechanism of mapping point distribution onto another one subject to sample and feature permutation invariance, referred to as invariant layer. For the sake of readability, this section focuses on the case of discrete distributions, referring the reader to Appendix A for the general case of continuous distributions.
2.1 INVARIANT FUNCTIONS OF DISCRETE DISTRIBUTIONS
Z= z= (Z,, y.) E IRd i [1;=]} denote a dataset including n labelled samples, with x, E Rdx an instance and y: E dy the associated multi-label. With dx and dy respectively the dimensions
2

---

Under review as a conference paper at ICIR 2021
of the instance and label spaces, let d _ dx + dy. By construction, z is invariant under permutation on the sample ordering; it is viewed an re-ize discrete distribution 2 -i== 8s, in R with &s, the Dirac function at zi. In the following, Z,, (R.) denotes the space of such n--ize point distributions, with ((R del Zn the space of distributions of arbirary size.
Let G d Sax x Say denote the group of permutations independently operating on the feature and label spaces. For
my) G, the image (z) of labelled sample is defined as (a (y with I (=(k), E [1; dx]) and (x) = (x o (k)), e [1; dx For simplicity and by abuse of notations, the operator mapping a distribution z = (2x, i E [1; n:]) to
{o(z)) del. ozz is still denoted a.
Let Z(52) denote the space of distributions supported some domain 52 C Ré, with 52 invariant der permutations in G. The goal of the paper is to define and train deep architectures, implementing functions & on Z(S2 C I2")) that are invariant under G, i.e. such that Vo E G, p(azz) = p(z)'. By construction, a multi-label dataset is invariant under permutations of the samples, of the features, and of the muuli-llbels, Therefore, any meta feature, that is, a feature describing a multi-labe dataset, is required to satisfy the above sample and feature permutation invariance properties.
2.2 DISTRIBUTION- BASED INVARIANT LAYERS
The building block of the proposed architecture, the invariant layer meant to satisfy the feature and label invariance requirements, is defined as follows, taking inspiration from (De Bie et al., 2019).
Definition 1. (Distribution-basel) invariant layers) Let an interaction functional sp RR Rd + I2 be G-invariant:
Yo G, (211 Rd x R., p(= 22) = ((o(zs).
The distribution based invariant layer f is defined as
fip z =Zz))ee) E EZ(R") + fx(2)
-
((1 Σ p((n:)))
E EZ(R'..
(1)
By construction, fo is G-invariant if is G-invariant. The construction of fo is extended to the general case of possibly continuous probability distributions by essentially replacing sums by integrals (Appendix A)
Remark 1. (Varying dimensions dx and dy). Both in practice and in theory, it is important that fi layers (in particular the first layer of the neural architecture) handle datasets of arbitrary number of features dx and number of multi-labels dy. The proposed approach, used in the experiments (Section 4), is to define & follows, Letting = (x, 3) and 2' (x', sf) be two samples in Rdx x Rdy let be defined from R onto R consider the sum of uaar[k), of [K], vll, s I(] for k ranging in [1; dx] and f [1; dy and apply mapping v from TR' to R² on this sum:
dx dy
x(2,z)= =
Σ
u(<[k), of A) v(())
Likewise, by construction y is G-invariant, i.e. it invariant both feature and label permutations. As shown in Section 4, this invariance property is instrumental to a good empirical performance. Remark (Varying sample size n)). By construction, fo is defined on Z(le)) = (R (inde- pendent of n), such that it supports inputs of arbitrary cardinality n.
Remark 3. (Discussion w.r.t. (Maron et al., 2020) The above definition of fo is based on the ag- gregation of pairwise terms ((2,, The motivation for using a pairwise şp is twofold. On the one hand, capturing local sample interactions allows to create more expressive architectures, which is important to improve the performance on some complex data sets, as illustrated in the experiments (Section 4). On the other hand, interaction functionals are crucial to design universal architectures
Ass opposed to G-equivariant functions that characterized by Vo E G. G(azz) ozp(z)
3

---

Under review as a conference paper at ICLR 2021
(Appendix C, theorem 2). The proposed theoretical framework relies on the Wasserstein distance (corresponding to the convergence in law of probability distributions), which enables to compare distributions with varying number of points even with continuous densities. In contrast, Maron al. (2020) do not use interaction functionals, and establish the universality of their DSS architec- ture for fixed dimension d and number of points n. Moreover, DSS happens to resort to max pooling operators, discontinuous w.r.t. the Wasserstein topology (see Remark 6).
Two particular cases are when yp only depends on its first or second input:
① if 2(2,z") (=²), then fo computes global "moment" descriptor of the input, as fz(z) = - x0(2,) R..
(i) if p(2, 2') E(2). then fo transports the input distribution via É. fz) (E(2)) [1: E 2(2"). This operation is referred to as a push-forward.
Remark 4. (Localized computation) In practice, the quadratic complexity of foe w.r.t the number of samples can be reduced by only computing ((z,; 2j) for pairs 2, sufficiently close each other. Layer fy thus extracts and aggregates information related to the neighborhood of the samples. Remark (Link to kernels) The use of an interaction functional yp is inspired from kernel ideas, albeit with significant differences: (i) in (2.). the detail of the pairwise interactions ( Æÿ) is lost through averaging; (ii) takes into account labels; (iii) şo is learnt. Further work will be devoted investigating the properties of the f=():) matrix.
2.3 LEARNING META- PEATURES
The proposed distributional neural architectures defined on point distributions (DIDA) are sought as
Z(R") Fe(z) def. fym fpm. o... fep, Rd
architecture (below). Only case dy = is considered in the remainder. The k-thh layer is built on the top of Sx. mapping pairs of vectors in 12d onto I with dj = d (the dimension of the input samples). Last layer is built on imm only depending on its second argument; it maps the distribution in layer referred to as meta- features.
where Ç are trainable parameters
1 onto vector, whose coordinates
The G-invariance and camension-agnostin of the whole architecture only depend on the first layer fer satisfying these properties. In the first layer, 1p is sought as si y), ()),)))
2 u(=[kk) 22]k], 3)) (Remark 1). with [(]]k] 2 [k. v.y') ) = (x (k) yey) in R. (0,1), 1). where p is non-linear activation function, A. (t,2) matrix, (=[k); 2 lk) the 2-dimensional vector concatenating z[k] and x' r'(k). and d a t-dimensional vector With
e = En u(>(A)) x' [k), y, I), function likewise applies a non-linear activation function P on an
affine transformation of e: v(e) = p(A, e + b,), with A, a (t, r) matrix and b a a r-dimensional vector.
Note that the subsequent layers need neither be invariant w.r.t. the number of samples, nor handle varying number of dimensions. However, maintaining the distributional nature among several layers shown to improve performance practice (Section 4). Every spx, k 2 is defined as PP = =({AA +bx). with p an activation function, A a (da, d241) matrix and by a -d41-dimensional vector. The DIDA neural net thus is parameterized by Ç del (Au, by. A,. by {g. by hx). that is classically learned by stochastie gradient descent from the loss function defined after the task hand (Section 4).
THEORETICAL ANALYSIS
This section analyzes the propertics of invariant-layer based neural architectures, specifically their robustness w.r.t. bounded transformations of the involved distributions, and their approximation abilities w.r.t. the convergence in law, which is the natural topology for distributions. As already said, the discrete distribution case is considered in this section for the sake of readability, referring the reader to Appendix A for the general case of continuous distributions.
4

---

Under review as a conference paper at ICLR 2021
3.1 OPTIMAL TRANSPORT COMPARISON OF DATASETS
Point clouds vs. distributions. Our claim is that datasets should be seen as probability distribu- tions, rather than point clouds. Typically, including many copies of a point in a dataset amounts to increasing its importance, which usually makes difference in standard machine learning setting. Accordingly, the topological framework used define and learn mela-features in the following that of the convergence in law, with the distance among two datasets being quantified using the Wasserstein distance (below). In contrast, the point clouds setting (see for instance (Qi et al., 2017)) relies on the Haussdorff distance among sets to theoretically assess the robustness of these archi- tectures. While is standard for 2D and 3D data involved in graphics and vision domains, it faces some limitations in higher dimensional domains, e.g. due to max-pooling being a non-continuous operator w.r.t. the convergence in law topology.
Wasserstein distance. Referring the reader to (Santambrogio, 2015; Peyré & Cuturi, 2019) for a more comprehensive presentation, the standard L-Wasserstein distance between two discrete proba- bility distributions z E Zm (R²) x Zm (RR) is defined as:
1 Σ f(2))
Wi 2')
f(z))
max SELip, (RR TE
with Lip, (I the space of 1-Lipschitz functions f Rd R. To account for the invariance require- ment (making indistinguishable x = (21,-.., Zn. and its permuted image (a((1),..., (o()....,((()))) del 0.* under a E G), we introduce the Gr-ivvariant 1- I-Wasserstein distance: for z 7m ( E Zm (R=)):
(1, z') min W 21(zzz,z')
oEG
such that W1 i.e. are equal in the sense of probability distributions up to sample and feature permutations.
(2(z.z) x') if and only if z and x' helong to the same equivalence class (Appendix A);
Lipschitz property. In this context, a map f from 7(RR)) onto Z(R²) is continuous for the conver- gence in law ka. weak convergence on distributions, denoted -) iff for any sequence z(k) z, then f(z (+) - fz)). The Wasserstein distance metrizes the convergence in law, in the sense that (8) z is equivalent to W, (x(5), z) + Furthermore, map f is said to be C'Lipschitz for the permutation invariant 1- I-Wasserstein distance iff
Vz, z' Z(RR), f(()() & CW (z,z'))
(4)
The C-Lipschitz property entails the continuity w.r.t. its input: if two input distributions are close in the permutation invariant 1- I-Wasserstein sense, the corresponding outputs are close too.
3.2 REGULARITY OF DISTRIBUTION- BASED INVARIANT LAYERS
Assuming the interaction functional to satisfy the Lipschitz property:
Yz Rd, ((2,-) and x(-22) are Cp-Liipshite
(5)
the robustness of invariant layers with respect to different variations their input is established (proofs in Appendix B). We first show that invariant layers also satisfy Lipschitz property, ensuring that deep architectures of the form (3) map close inputs onto close outputs.
Proposition 1. Invariant layer fo type (I) is Liipsshiiz in the sense of (4).
A second result regards the case where two datasets z and z' are such that z' the image of z through some diffeomorphism т (z (21...., 2n.) and z' Tgz (T(2)).... (2x)). If is close identity, then the following proposition shows that fT((zz) and (z) are close too. More generally, if continuous transformations T and & respectively apply on the input and output space fig. and are close to identity, then & (TzZ) and f (z) are also close.
Proposition 2. Letr: Rd + Rd and 5: B - IR" be two Lipschitz mups with respectiivlly Lipschitz constants C, and Ce. Then,
Vz Z(1)), WI(EFFoTTH)) Fz(z)) & sup
[(()) - -0r1 2r Lip(pp sup (T(x) - 1/2 xtfo(((0)
zen
Yz, x' Z(S),, if is equivariant, W (12)) C,
5

---

Under review as a conference paper at ICIR 2021
3.3 UNIVERSALITY OF INVARIANT LAYERS
Lastly, the universality the proposed architecture is established, showing that the composition of an invariant layer (1) and a fully-connected layer is enough to enjoy the universal approximation property, over all functions defined on Z(R") with dimension d less than some D (Remark 1).
Z(S2) + R be a G-invariant map on compact Q2 continuous for the conver- gence in law. Then Ve > 0, there exists two continuous maps 1b, up such that
Theorem 1. Let F:
Vz E Z(2),
(F(z) oo fz(z)) <E
where ip is G-invariant and independent F.
Proof. The sketch of the proof is as follows (complete proof Appendix C). Let us define yp = goh where: (i) h is the collection of dx elementary symmetric polynomials in the features and dy elementary symmetric polynomials in the labels, which is invariant under G; (ii) a discretization of h([)) on a grid is then considered, achieved thanks 9 that aims at collecting integrals over each cell of the discretization; (iii) v applies function F on this discretized measure; this requires à be bijective, and is achieved by ht through projection on the quotient space Sa/G and restriction to its image compact !!' To sum up. J defined such computes expectation which collects integrals over each cell of the grid to approximate measure h,z by a discrete counterpart hyz. Hence i applies F to hr (hiz). Continuity is obtained as follows: (i) proximity hyz and hyz follows from Lemma 1 in (De Bie et al., 2019)) and gets tighter as the grid discretization step tends to 0; (ii) Map h is 1/ L/d-Hölder, after Theorem 1.3.1 from (Rahman & Schmeisser, 2002)); therefore Lemma 2 entails that W (z, hy lizz) can be upper-bounded; (iii) since f2 is compact, by Banach- Alaoglu theorem, Z(2) also is. Since F is continuous, it is thus uniformly weakly continuous: choosing a discretization step small enough ensures the result.
Remark 6. (Comparison with (Maron et al., 2020) The above proof holds for functionals of arbitrary input sample size n, as well as continuous distributions, generalizing results in (Maron et al., 2020). Note that the two types of architectures radically differ (more in Section 4).
Remark 7. (Approximation by an invariant NN) After theorem any invariant continuous function defined on distributions with compact support can be approximated with arbitrary precision by an invariant neural network (Appendix C). The proof involves mainly three steps: (i) an invariant layer fi can be approximated by an invariant network; (ii) the universal approximation theorem (Cybenko, 1989; Leshno al., 1993): (iii) uniform continuity is used to obtain uniform bounds.
Remark 8, (Exteension to different spaces) Theorem also extends to distributions supported on dif- ferent spaces, via embedding them into nigh-dimensional space. Therefore, any invariant function on distributions with compact support R with I can be uniformly approximated by an invariant network (Appendix C).
4 EXPERIMENTAL VALIDATION
The experimental validation presented in this section considers two goals of experiments: (i) assess- ing the ability of DIDA to learn accurate meta-features; (ii) assessing the merit of the DIDA invariant layer design, building invariant fe on the top of an interactional function yP (Fq. 1). As said, this architecture is expected to grasp contrasts among samples, c.g. belonging to different classes; the proposed experimental setting aims to empirically investigate this conjecture.
Baselines. These goals of experiments are tackled by comparing DIDA to three baselines: DSS layers (Maron et al., 2020); hand-crafted meta-featured (HC) (Muñoz et al., 2018) (Table 5 in Ap- pendix D): DATASET2VEC (Jomaa et al., 2019). We implemented DSS, the code being not avail- able.? In order to cope with varying dataset dimensions (as required by the UCI and OpenML benchmarks), the original DSS was augmented with an aggregator summing over the features. Three DSS baselines are considered; linear non- linear invariant layers, possibly preceded by equivariant layers. Similarly, the original DATASET2V VEC implementation has been augmented to address our experimental setting. The baselines are detailed in Appendix D.3.
"The code source of DIDA and (our implementation of) baselines available in supplementary materials.
6

---

Under review as a conference paper at ICTR 2021
DIDA
FC (258) BN
FC (128)) BN
BN
Tz(2)
Eq.
Section 4.1
Section 4.2
DIDA
Z
DIDA
Log
0
Softmax
Log Loss
DIDA
(
0₂
MLP
Figure 1:
Learning meta-features with DIDA. Top: the DIDA architecture (BN stands for batch norm; FC for fully connected layer). Bottom left: Learning meta-features for patch identification using Siamese architecture (section 4.1). Bottom right: learning meta-features for performance modelling, specifically to rank two hyper-parameter configurations 0 and 0g (section 4.2).
Experimental setting. Two tasks defined at the dataset level are considered: patch identification (section 4.1) and performance modelling (section 4.2). The dataset preprocessing protocols are detailed in Appendix D. On both tasks, the same DIDA architecture is considered (Fig 1). involving 2 invariant layers followed by fully connected (FC) layers. Meta-features (z( consist of the output of the third FC layer, with denoting the trained DIDA () parameters. All experiments run on AVIDIA-Tesla-Vie Maa---100--MMMM- GPU with 32GB memory, using Adam optimizer with base learning
rate 10
4.1 TASK I: PATCH IDENTIFICATION
The patch identification task consists of detecting whether two blocks of data are extracted from the same original dataset (Jomaa et al., 2019). Letting u denote a n-sample, d-dimensional dataset, n2, de patch constructed from by selecting n, examples in (sampled uniformly with replacement) and retaining their description along d, features (sampled uniformly with replace- ment). The size n. and number d, of features of the patch are uniformly selected in fixed in- tervals (Table 4. Appendix D). To each pair of patches z with same number of instances n. R2,, is associated binary meta-label f(z,x' set to iff z and z' are extracted from the same initial dataset u. DIDA parameters Ç are trained to minimize the cross-entropy loss of model 8 (z.zz exp (z) (z)))) with (z) and (z.z.) the meta-features computed for z and
Minimize ((C) = Σ ((z, 2') & (z) + E(z, z')) log(1 & (z, 2'))
(6)
DIDA and all baselines are trained using a Siamese approach (Figure 1, bottom left): the same DIDA (or baseline) architecture used to compute meta-fcatures (z) and Fz(z') from patches z and x', and trained to minimize the cross-entropy loss w.r.t. f(z, z'). The classification results on toy datasets and UCI datasets (Table detailed Appendix D) show the pertinence of the DIDA features, particularly so on UCT datasets where number of features widely varies from one dataset to another. The relevance of the interactional invariant layer design is established on this problem as DIDA outperforms both DATASET2VEC: DSS as well as the function learned on the top of the hand-crafted meta-features.
ablation study is conducted assess the impact of (i) the feature permutation invariance; (ii) considering one ys two invariant layers of type (1). The so-called No-FINv. FINV-DSS baseline, detailed in Appendix D, is built upon (Zaheer et al., 2017); only differs from the DSS baseline as it is not feature permutation invariant. With ca the same number of parameters as DSS, its performances
7

---

Under review as a conference paper at ICLR 2021
Method
# parameters
TOY
UCI
53312
77.05 % 1.63
58,36 % 2.64
No-FINv-DSS (no invariance in fcatures)
1297692
90.49 % 1.73
64.69 % 4.89
DATASET2VEC (our implementation)
257088
97.90 %+ 1.87
77.0.5 %+ 3.49
DSS layers (Linear aggregation)
1338684
89.32 %4 1.85
76.23 %+ 1.84
DSS layers (Non-linear uggregation)
1338684
96.24 %+ 2.04
83.97
%+ 2.89
DSS layers (Fquivariant--ivvariat))
1338692
96.26 %+ 1,40
82.94 %+ 3.36
DIDA (I invariant layer)
323028
91.37 % 1.39
81.03 % 3.23
DIDA (2 invariant layers)
1389089
97.20
0.10
89.70 % + 1.89
Table 1: Patch identification (binary classification accuracy) on 10 runs of DIDA and considered baselines.
Method
SGD
SVM
LR
k-NN
Hand-crafted
71.18 %1 0.41
75.39 % 0.29
86.41 %c 0.419
65.44 %1:: 0.73
DATASET2
74.43 %: 0.90
81.75 % 1.85
89.18 % 0.45
72.90 %+ 1.13
DSS (Lincar aggregation)
73.46 %+ 1.44
82.91 % 0.22
87.93
%6 0.58
70.07
%1 2.82
DSS (Equivariant--Inval Invarian))
73.54
%1 0.26
81.29 %+ 1.65
87.65 %4 0.03
68.55 %1 2.84
DSS (Nou-linear aggregation)
74.13
%: 1.01
83.38
%1 0.37
87.92
%... 0.27
73.07
%1 0.77
DIDA (I invariant layer)
77.31
0.10.16
84.05
%+ 0.71
90.16 % 0.17
74.41
%++ 0.93
DIDA (2 invariant layers)
78.41
%: 0.41
84.14 % 0.02
89.77 % 0.50
78.91
% 0.54
Table 2: Pairwise ranking of configurations, for ML algorithms SGD, SVM, LR and k-NN: perfor- mance on test set of DIDA, hand-crafied, DATASET2VEC and DSS (average and std deviation on 3 runs).
significantly lower (Table 1), showcasing the benefits of enforcing the feature invariance property. Secondly, we compare the 2-invariant layers DIDA, with the 1-invariant layer DIDA (IL-DIDA and 2L-DIDA for short): IL-DIDA yiclds significantly lower performances, which confirms the advan- tages of maintaining the distributional nature among several layers, already noted by (De Bie al., 2019). Note that the IL-DIDA still outperforms the non feature-invarial baseline, while requiring much fewer parameters.
4.2 TASK PERFORMANCE MODEL LEARNING
The performance modelling task aims to assess a priori the accuracy of the classifier learned from a given machine learning algorithm with given configuration (vector of hyper-parameter ranging in syper-paramedt space θ, Table 6 in Appendix D), on a dataset z (for brevity, the performance of z) (Rice, 1976).
For each ML algorithm, ranging in Logistic regression (LR), SVM, k-Nearest Neighbours (k-NN), linear classifier learned with stochastic gradient descent (SGD), a set meta meta-features is learned to predict whether some configuration 0; outperforms some configuration 02 on dataset z: to each triplet (z, 0,, 02) is associated binary value E(z, 01, 02), set to 1 iff 0g yields better performance than 81 on z. DIDA parameters ( are trained to build model &c. minimizing the (weighted version of) cross-entropy loss (6), where (z, 0,, 02) is a 2-layer FC network with input vector [z) 0:; 02). depending on the considered ML algorithm and its configuration space.
In each epoch, batch made of triplets (z, f1,02) 02) is built, with 0r, uniformly drawn in the al- gorithm configuration space (Table 6) and n-sample d-dimensional patch of dataset in the OpenML CC-2018 (Bischl et al., 2019) with n uniformly drawn in [700; 900) and in [3; 10). Al- gorithm summarizes the training procedure.
The quality of the DIDA meta-fcatures is assessed from the ranking accuracy (Table 2), showing their relevance. The performance gap compared to the baselines is higher for the k-NN modelling
8

---

Under review as a conference paper at ICTR 2021
Algorithm 1 Performance Modeling
1: Fc <- meta-feature extractor (DIDA, DSS, DATASET2VEC, or Hand-crafted)
2: MLP NN]Linear(644) Line--LLar(332 ReLU-Linear(]]
3: CLF < machine learning classifier (SGD, SVM, LR K-NN)
4: error f 3-CV classification error function
5: for iteration=1,2, 2,... do
6: Sample (2). two syyper-prametent of CLF
Search space: Table 6 P. Patch dimension: Table
7: Sample patch from dataset
8: pred +- softma((MLP(F((4 (z), MLP(Fe( (2),
9: Backpropagate logloss(pred, O error(z, CLF(O,)) < error(z, CLF((2)) else 1)
10: end for
task; this is explained the sought performance model only depends the local geometry of the examples. Still, good performances are observed over considered algorithms. Note that 21.- DIDA yields significantly better (respectively, similar) performances than IL-DIDA on the k-NN model (resp. all other models).
Meta-feature assessment. A regression setting is thereafter considered, aimed to predict the ac- tual performance of configuration 0 based on the (frozen) meta-features Fz(z). The regression accuracy is illustrated the configurations of the k-NN algorithm on Figure 2, left (results for other algorithms are presented in Appendix D). The comparison with the regression models based on DSS meta-features or sand-crafie features (Figure 2. middle and right) shows the merits of the DIDA architecture; tentative interpretation for the DIDA better performance is based the interactional nature DIDA architecture, better capturing local interactions.
Dida
D55
Hand-crafted
08-
02-
02
25
05
pm1
part
Figure k-NN: True performance vs performance predicted by regression on top of the meta- features (i) learned by DIDA, (ii) DSS or (iii) Hand-crafted statistics.
5 CONCLUSION
The theoretical contribution of the paper is the DIDA architecture, able to learn from discrete and continuous distributions on Rd, invariant w.r.t. feature ordering, agnostic w.r.t. the size and di- mension the considered distribution sample (with less than some upper bound D). This architecture enjoys universal approximation and robustness properties, generalizing former results obtained for point clouds (Maron et al., 2020). The merits of DIDA demonstrated on two tasks defined at the dataset level: patch identification and performance model learning, comparatively the state art (Maron et al., 2020; Jomaa et al., 2019; Muñoz al., 2018). The ability to ac- curately describe a dataset in the landscape defined by MI. algorithms opens new perspectives to compare datasets and algorithms, e.g. for domain adaptation (Ben-Davi et al., 2007; 2010) and meta-learning (Finn al., 2018; Yoon al., 2018), in light of kernel methods.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in Neural Information Processing Systems 19, pp. 137-144,
2007.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine Learning, 79():151-175, 2010. ISSN 0885-6125, 1573-0565.
Bernd Bischl, Giuseppe Casalicchio. Matthias Feurer, Frank Hutter, Michel Lang, Rafael G Man- tovani, Jan N van Rijn, and Joaquin Vanschoren. Openml benchmarking suites. arXiv preprint arXiv:1708.0377: 2019.
Taco Cohen and Max Welling. Group equivariant convolutional networks. Proceedings of The 33rd International Conference on Machine Learning. 18:2990-2999) 20-22 Jun 2016.
David A. Cox, John Little and Donal O'She Ideals, Varieties, and Algoritims: An Introduction Computational Algebraic Geometry and Commutative Algebra, 3/e (Undergraduate Texts in Mathematics). Springer- ppringer-Verlag, Berlin, Heidelberg. 2007. ISBN 0387356509.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics control, signals and systems, 2(4):303-314, 1989.
Gwendoline De Bie, Gabriel Peyré, and Marco Cuturi. Stochastic deep networks. Proceedings of the 36th International Conference Machine Learning, pp. 1556-1565, 2019.
Dheeru Dua and Casey Graff. UCI machine learning repository. 2017. URL URR.httt::/arrchiee ics. uci. edu /ml.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning Advances in Neural Information Processing Systems 31, pp. 9516-9527, 2018.
Robert Gens and Pedro M Domingos. Deep symmetry networks. Advances in Neural Information Processing Systems 27, pp. 2537-2545, 2014.
Jason Hartford, Devon Graham, Kevin Leyton-Brown, and Siamak Ravanbakhsh. Deep models interactions across sets. Proceedings of the 35th International Conference on Machine Learning, 2018.
Tatsunori Hashimoto, David Gifford, and Tommi Jaakkola. Learning population-lewe diffusions with generative rnns. Proceedings of The 33rd International Conference on Machine Learning. 48:2417-2426, 20-22 Jun 2016.
Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. ArXiv, hts//500.051ee 161000116611611 2015.
Geoffrey Hinton, Li Deng. Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6):2-977 2012.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automated Machine Learning: Meth- ods, Systems, Challenges. Springer, 2018. In press, available at http://auttomlorr/boook
Hadi S. Jomaa, Josif Grabocka, and Lars Shhmidt-Thiema. Dataser2vec:
Learning dataset meta-
features. arXiv, abs/1905.1 1063, 2019.
Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks. Advances in Neural Information Processing Systems 32, pp. 7090-7099, 2019.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. Proceedings of The 35th International Conference on Machine Learning, 2018
10

---

Under review as a conference paper at ICLR 2021
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, pp. 1097-1105, 2012.
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward net- works with a nonpolynomial activation function approximate any function. Neural networks, 6(6):861-87,, 1993.
Haggai Maron, Hcli Ben- Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, IA, USA, May 6-9, 2019, 2019a.
Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Califorria, USA, pp. 4363-4371, 2019b.
Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning sets of symmetric elements. International Conference on Machine Learning (ICML), 2020.
Mario A. Muñoz. Laura Villanova, Davaatseren Baatar, and Kate Smith-Miles. Instance spaces for machine learning classification. Machine Learning, 107(1):109-144 147, January 2018. ISSN 0885-6125 6125. doi: 10.
Mario A Muñoz, Laura Villanova, Davaatseren Baatar, and Kate Smith-Miles, Instance spaces for machine learning classification. Machine Learning, (7(1)):09-147, 2018.
Valerio Perrone, Rodolphe Jenatton, Matthias W Seeger, and Cedric Archambeau. Scalable hyperpa- rameter transfer learning. Advances in Neural Information Processing Systems 31, pp. 6845-6855, 2018.
Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundationn and Trends Machine Learning. ((6--))355-607 2019. ISSN 1935-8237. doi: 10.1561/2200000033
Bernhard Pfahringer, Hilan Bensusan, and Christophe Giraud-Carrier. Meta-learning by land- marking various learning algorithms. Proceedings of the Seventeenth International Conference on Machine Learning, pp. 743-750, 2000.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
Qazi Inamur Rahman and Gerhard Schmeisser. Analytic theory of polynomials. Oxford University Press, 2002.
Siamak Ravanbakhsh, Jeff Schneider, and Barnabás Póczos. Equivariance through parameter- sharing. Proceedings of the 34th International Conference on Machine Learning. 70:2892-2901, 2017.
John R. Rice. The algorithm selection problem. Advances in Computers, 15:65-118, 1976.
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 2015.
Nimrod Segol and Yaron Lipman. On universal equivariant set networks. 8th International Confer- ence on Learning Reprexentations, ICLR 2020, 2019.
J. ShylorTaylor. Symmetries and discriminabilitt in feedforward network architectures. JEEE Transactions on Neural Networks, 4(5):81--22,, Sep. 1993. ISSN 1941-0093. doi: 10.1 248459.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. Advances in Neural Information Processing Systems 29, pp. 4134-4144 2016.
11

---

Under review as a conference paper at ICTR 2021
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
David H. Wolpert. The lack priori distinctions between learning algorithms. Neural Computa- tion, 8(7):1441-390, 1996.
Jeffrey Wood and John Shawe- Shylo-Taylor. Representation theory and invariant neural networks. Discrete applied mathematics, 69(1-2):33-60, 1996.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. Advances Neural Information Processing Systems 31, pp. 7332-7342, 2018.
Manzil 7aheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Advances in Neural Information Processing Systems 30, pp. 3391-3401, 2017.
12


Reviews:
[{"review_id": "u2MVu6O0v8U", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The method introduces the DIDA architecture to learn from distributions and be invariant to feature ordering and size.  The authors extend the ideas proposed by Maron et al. (2020) to the continuous domain and generalize their results.  The experiments are done on two tasks.  The patch identification (out-of-distribution test) clearly show the invariance to feature and dataset size. Nevertheless, it is not clear whether the method is invariant to feature permutation.  The performance model task shows properties of the architecture to predict global structures of the dataset within their meta-features.\n\nPros:\n+ The paper is well written and easy to follow (there are some minor errors or descriptions that need to be improved, but they are not major issues).  \n+ The extension of Maron et al. (2020) is interesting and provides theoretical arguments.  \n\nCons:\n- On the other hand, the experimental section is weak.  Since the main claim is the invariant meta-features (and architecture) to permutations, I would expect to see some experiments showing it.  I see the theoretical guarantees given, but an experiment showing the improvements and benefits will make the results more robust.\n- The experimental protocols need to be clearly defined.  If space is an issue, I recommend using the appendix for the details.  Despite that, authors manage to show improvement over Dataset2Vec and DSS methods.  In their current form, I found it will be difficult to reproduce the experiments.\n\nDue to the problems on the experimental details, and simple tests I'm giving the paper a score 5.\n\nComments:\n\n- I found the definition of $\\sigma_X (x)$ rather strange since it depends on its inverse (i.e., it is circular).  Are you just saying that $\\sigma_X$ is a permutation feature-wise of x?\n- I found the bracket notation rather misleading in (1).  It seems that you are concatenating the output of $f_\\varphi$ instead of creating a set for the discrete distribution $\\mathbf{Z}_n$ ($\\mathbb{R}^r$).\n- It is not clear the shape of the matrix $A_u$.  Assuming the concatenation of the slices of $x$ to be a $2 \\times 1$ column vector, and $b_u$ a column vector $t \\times 1$, then $A_u$ should be the transposed of what is stated, i.e., $t \\times 2$.  Review your notation convention and be clear on what type of vector column/row you are using, and how the dimensions of your matrices are given.\n\n- What is the exact training protocol for the samples in the given datasets when training to extract the meta-features?  \n- Are you applying a particular set of permutations $S_{d_X}$ to the inputs (or features) while training?  \n- Are the DIDA networks shared for the patch identification task?  Are you training them simultaneously with the \"meta cross entropy loss\" (5)?  In the case of the patches, how are they selected?  It is not clear what you mean by \"retaining samples with index $I \\in [n]$\".  Are you doing something else than extracting a subset of the features in the original sample?  I can visualize a patch in an image or 2D data, but if the data is in $\\mathbb{R}^{d_X}$ is not straightforward.  Hence, a proper protocol for the extraction is needed.\n\n- Your evaluations don't mention if you test for the performance given permutations at testing time.  Are you evaluating your method with different permutations at testing time?  What is the performance in this case?  It will be interesting to see the actual invariance working experimentally.\n\n- In the performance modeling task, it seems from the description that the classifiers are trained directly on the dataset $\\mathbf{z}$.  However, the bottom right of Fig. 1 seems to show that the classifiers are trained on the DIDA output, i.e., the meta features.  Which one is it?  Make them consistent.\n\n\nMinor comments:\n- Your abstract shouldn't contain citations, since most of the time it will be read outside of the paper, where the citations are missing.\n- Use appropriate textual and parenthetical citations.  E.g., \"by (Qi et al., 2017; Zaheer et al., 2017)\" should be textual instead of parenthetical.\n- Typo P2 p2: \"characteristcs\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "QQyk-PXjYp3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a neural network layer designed to process distribution samples that is invariant to permutations of the samples and the features. The proposed method is compared empirically to DSS, which achieves the same types of invariance but is restricted to point sets rather than discrete or continuous probability distributions. The two tasks used for the empirical evaluation in the paper are: a) patch identification (are two blocks of data extracted from the same original dataset?) and b) model configuration assessment (is one configuration of a learning algorithm going to produce a more accurate model for a particular dataset than another one?). On the first task, the paper compares to models built using Dataset2Vec embeddings as well as DSS. On the second task, the paper compares to handcrafted features as well as DSS. In both tasks, the proposed method produces more accurate predictors than DSS, etc. The paper also has some theoretical results regarding the universality of the proposed architecture and its robustness w.r.t. Lipschitz-bounded transformations. \n\nA weakness of the paper is that its primary theoretical advantage over DSS (according to my limited understanding) is that it is applicable to distributions rather than just point clouds, but the experiments only consider point clouds (presumably to be able to compare to DSS). The main selling point is then that the proposed method outperforms DSS in this setting in the experiments. However, why this is the case is not clearly explained in the paper. The paper itself states that DSS is a special case of the proposed approach. If possible, an ablation study showing which particular aspects of the new method contribute the most to the observed difference in predictive performance seems appropriate.\n\nPage 3 refers to \"Remark 5\", but this remark does not appear to exist.\n\nRather than using the handcrafted features directly, it would be useful to train an embedding network (e.g., a Siamese network), to yield better embeddings for the two tasks concerned. This would be an obvious approach from a practical point of view that is not considered in the paper. \n\nWhy are handcrafted features not included in the first task (Table 1)?\n\nWhy is Dataset2Vec not included in the second task (Table 2)?\n\nThe Dataset2Vec results in Table 1 are from (Jomaa et al., 2019) but Appendix D.2 states that the publicly available implementation of Dataset2Vec was used. For what? Also, are the results shown in the table for DSS, etc., obtained under exactly the same experimental conditions as those used in (Jomaa et al., 2019)? This is important to enable a fair comparison.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LUAVOIfsWg5", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper proposes a new deep learning architecture that is invariant under permutations of both the data points and the features. The paper shows that this new architecture also has the universal approximation property. Empirical experiments were performed to demonstrate the effectiveness of this new architecture.\n\nOverall, the paper seems to be well-motivated. It is also technically sound and the presentation of the idea is clear. I have some comments that are detailed below.\n\n##########################################################################\n\nComments: \n\n- One of the main features of the proposed architecture is that it is invariant under the permutation of the data features. While it is intuitive that the results should be invariant under the permutation of samples, I am not fully convinced why it is desirable for the architecture to be invariant under the permutation of features. Are there some solid motivating use cases? \n\n- It seems that the proposed architecture is related to kernel methods. Particularly, the interaction functional \\phi is similar to a kernel function. The authors should discuss the connection between their method and kernel methods. \n\n\n\n- In Section 2.3, the proposed architecture uses one invariant layer. However, in the experiments, two invariant layers were used. Why do we need one more invariant layer in the experiments? \n\n \n##########################################################################\n\nMinor comments: \n\n- Line 6 in abstract: avoid using abbreviations in the abstract ('w.r.t.' --> 'with respect to')\n\n- Section 4 1st paragraph last line: \"allocated ca the\" --> \"allocated the\"\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2cVCd3R59M", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel set/distribution representation architecture DIDA, which leverages pairwise embedding of the set’s elements. The method can be used to represent discrete and continuous distribution representation. The authors also provide the theoretical proofs of the universality of the invariant layers, the local consistency. The experiments show that the architecture improves some dataset representation tasks\n\nOn quality: A single idea was developed and well-executed in this work. The theoretical considerations are on point and improve the understanding of the applicability of the architecture.\n\nOn clarity: This idea is rather clear but the writing and the structure of the paper are sometimes difficult to follow. For example, without a previous background in the field, it is quite hard to understand why some theoretical considerations were made. It is also difficult to understand/verify some parts of the proofs that refer to the appendix, which is missing. \n\nOn originality: Previous works on distributions representations, only consider quantities that are related to their first moments. Using pairwise interactions mainly leads to considering second moments when representing a distribution. In that sense, the paper brings some originality to the field. By also making small adjustments in their theoretical analysis, the authors make the method general enough to be applied to distributions instead of points of clouds.\n\nQuestions/Concerns:\nBuilding the representation of pairwise interactions alone can lead to representations that ignore the first moments of the distribution. To fully characterize a distribution, all the moments should be considered (or at least the first and the second moments in your case). \nClearly, the complexity here is O(N^2), so how do you scale your method to large sets/ distributions?\nWhy didn’t try classical point cloud benchmarks? \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper invariantizes distribution based deep networks by using pairwise embedding of the set’s elements.  The idea is inspired from De Bie et al. (2019), which allows invariance to be incorporated through the interaction functional.  Although the paper is well executed with solid theoretical analysis and solid response to the reviewers' comments, the novelty is limited, and reviewers have concerns with experiments and presentation.


Decision: Reject