Pooling by Sliced-Wasserstein Embedding
Authors: Navid Naderializadeh
Joseph F. Comer
Reed W Andrews
Heiko Hoffmann
Soheil Kolouri
Venue: NeurIPS 2021 Poster (2021)
Keywords: Pooling
feature aggregation
permutation invariance
optimal transport
set learning
representation learning

Abstract: Learning representations from sets has become increasingly important with many applications in point cloud processing, graph learning, image/video recognition, and object detection. We introduce a geometrically-interpretable and generic pooling mechanism for aggregating a set of features into a fixed-dimensional representation. In particular, we treat elements of a set as samples from a probability distribution and propose an end-to-end trainable Euclidean embedding for sliced-Wasserstein distance to learn from set-structured data effectively. We evaluate our proposed pooling method on a wide variety of set-structured data, including point-cloud, graph, and image classification tasks, and demonstrate that our proposed method provides superior performance over existing set representation learning approaches. Our code is available at https://github.com/navid-naderi/PSWE.

Pooling by Sliced- Embedding
Navid Naderializadeh"
Department of Electrical and Systems Engineering
University of Pennsylvania Philadelphia, PA 19104
nnaeeri@seas. wue.nneees...eeee.e..
Joseph F. Comer, Reed W. Andrews, Heiko Hoffmann
HRL Laboratories, LLC.
Malibu, CA 90265 ( (jfcomer, rwandrews, hhof fmann J 1@rrh com
Soheil Kolouri
Computer Science Department
Vanderbilt University Nashville, TN 37235
Abstract
Learning representations from sets has become increasingly important with many applications in point cloud processing, graph learning, image/video recognition, and object detection. We introduce mechanism for aggregating set of features into fixeddimenssooal representation. particular, we treat elements of a set as samples from probability distribution and propose an end-to-end trainable Euclidean embedding for sliced- Sosdd--sssrstein distance to learn from set-structured data effectively. We evaluate our proposed pooling method graph, and image classification tasks, and demonstrate that our proposed method provides superior performance over existing set representation learning approaches. Our code is available at hattps://github com/ navid swwviidndderiii naderi/PSMi
cameetrically-inerrreeete
and generic pooling
a wide variety of set-structured data, including point-cloud,
Introduction
Many modern machine learning (ML) tasks deal with learning from set-structured data. In some cases, the input object itself a set, as in point cloud clasiffcation/reggrssion, and in other cases, the complex input object is described as set of features after being processed through backbone, i.e., feature extractor. For instance, in graph mining, graph is represented as a set of node embeddings, and in computer vision, image is represented as set of local features extracted from its different regions (i.e., fields of view). There are unique challenges namely: i) the set cardinalities could differ from one instance another, and ii) the elements of the set do not necessarily have an inherent ordering. These challenges call for MI. models that can both handle varied input sizes and invariant permutations, i.e., the model output does not change under any permutation of the input set elements.
dealing with such set-structured data,
Prior work on learning from set-structured data can be broadly categorized methods based on either implicit or explicil embedding of sets into Hilbert space. Implicit embedding approaches
Work done while at HRL Laboratories, LLC.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(i.e.. kernel methods) rely on defining a distance/similarity measure (i.e., kernel) between two sets 2 7 9 These methods involve one of the two strategies of 1) solving a correspondence problem between elements of the input sets and measuring the similarity between corresponding elements, or 2) comparing all pairs of elements between the two sets based on a similarity measure (e.g., approaches based on Maximum Mean Discrepancy). On the other hand, explicit embedding methods learn a -nniatatiinnnnaiian mapping into a Hilbert space and provide bxedddiieeniioal representation for given input set that classic ML approaches could further process [T More recently, algorithms based on a composition of perruutation-equivarae pooling mechanisms have been proposed to
neural network backbones and cermutation-invariia define a parametric permutation invariant mapping [II 13
14 15 [6]. Notably, Zaheer et al. [ proved that such composition provides a universal approximator for any set function. Lee et al. [4] further showed that utilizing permutation-equiaa equivvarint backbones that do not process set elements independently but model the interactions between the set elements (e.g., using self-attention) is theoretically and numerically advantageous. Similar observations have been made in the field of
graph learning using various graph neural network (GNN) architectures [17 18| 19). In parallel,
several works have studied the importance of permutation-invaaian pooling mechanisms to go beyond
the commonly used mean, sum, max, or similar operators [12 13 15 [6].
convenient interpretation in dealing with sets is considering their elements as samples from an these probability distributions
unknown underlying probability distribution and comparing/embeddee to perform set learning. Due to this interpretation, optimal transport has played a prominent role in learning from sets. For instance, Kusner et al. 20) and later Huang al. (2I] represented document as a set of words. They leveraged the /- 1-Wasserstein distance (i.e., the earth mover's distance) to compare these sets with one another and define a measure of document similarity. Various researchers have devised similarly flavored approaches in computer vision by comparing images via calculating the Wasserstein distance between their sets of local features. For instance, Zhou et al. [22] use this distance to learn prototypes for image classes and perform few-shot inference, while Lin et al. [23 leverage it for designing diverse adversarial examples. More recently, similar ideas were used for image enhancement [24]. Finally, comparing sets via Wasserstein distances has also been proven to
be useful in other applications including graph learning 9 12 domain adaptation 25 26 and
transfer learning (27].
In this work, we propose novel cheoretically-gronnde. and simple compute permutation-invaaie pooling mechanism for embedding sets of various sizes into a fixed-size representation. Our proposed method, which refer to as Pooling by Sliced- Wasserstein Embedding (PSWE), provides an exact Euclidean embedding for the (generalized) sliced-Wassersee Wasserstein (SW) distance. We start by defining a similarity measure between sets of samples based on the SW distance. We then propose an explicit set embedding for which the Euclidean distance between embedded sets equals the SW distance between them. In our experiments, we follow the recent work on set learning [ [4 -et.tttttiioeuuvviiiiiiina backbone followed by our permutation- invariint pooling method
and use to perform end-to-end learning on different data modalities. We demonstrate the scalability and effectiveness of our approach on various learning tasks, including point cloud classification, graph classification, and image recognition. Aside from introducing a novel pooling mechanism, one of the key numerical insights of our work is that basic pooling mechanisms, such as mean meen-poolingg provide competitive performance when the permutation-cen -qquvvvrran backbone is complex. However, for plain backbones (e.g.. shared multi-layer perceptron (MLP) among the set elements), more sophisticated pooling mechanisms, including our proposed PSWE method as well other recently-proposed pooling mechanisms in the literature (e.g.. Pooling by Multi-Head Attention (PMA) [4] and Featurewise Sort Pool (FSPool) [5) significantly boost the performance compared to basic pooling mechanisms.
2 Related Work
Permutation-invarte functions are crucial components in learning from sets and are often used as pooling layers to aggregate features from a set and provide constant-size representation regardless of the set cardinality. Max, sum, and mean pooling are simple, yet very widely used, examples of such functions. Recently, various works have shown the effectiveness of more sophisticated and often learnable pooling operators in improving the performance of learning from set-structured data [3 14 28
Murphy et al. [13] introduced a pooling mechanism based on the average of permutation snnivarriantfuncttion applied to all re-orderings of the set elements. Summing over all
2

---

re-orderings of an input set is, of course, computationally prohibitive. Hence, one can use a canonical ordering of set elements (e.g.. via sorting [5].. or learn to predict the optimal permutation for an input set [29 30 28
Another prevalent idea is to perform pooling based comparing input set with trainable and fixed-size reference sets. For instance, Skianis et al. [10] proposed a pooling that consists of the distances between an input set and trainable reference sets, where the distance was calculated by solving the correspondence problems between the input and each reference set. More interestingly. this idea is analogous to pooling by multi-head attention (PMA), an important building block in the Set Transformer and Perceiver architectures reference sets and an input set is used as a permutation-invarr
[14 311, where the cross- apsss-tttntionn between trainable function. Attention-baser pooling has been shown to perform really well in practice on wide range of applications.
14 32
We introduce novel pooling mechanism by treating sets as empirical probability measures and calculating an embedding for these probability measures in which the Euclidean distance between two embedded sets is equal to the Wassersten distance between their empirical distributions. Our work is closely related to the work by Mialon et al. [12] and Kolouri et al. [16]. In short, [12] proposes an approximate Euclidean embedding for the Wasserstein distance, similar [6], reproducing kernel Hilbert space (RKHS), while our proposed framework is based on devising an exact Euclidean embedding for the (generalized) sliced- Wasserstein distance. Interestingly, our proposed pooling by sliced- Wasserstein embedding (PSWE) can also be viewed as theoretically-groune of the sorting-basse based FSPool mechanism proposed in [15), where we show that the introduction of trainable slicers, as well as trainable reference sets, further boost the end-to-end performance in wide spectrum of classification tasks.
generalization
3 Background
Let X, (arh Rd) denote an input with N; elements living in Rd. We assume that the set elements are samples from unknown underlying probability measure, f4, defined in X' R with probability density d1)(() = (r))dr, and what we have observed is the empirical distribution (x())) N. No 8(x= xr)), where d(-) is the Dirac delta function.
3.1 2- Wasserstein Distance
Let μ4 and μU denote two Borel probability measures with finite 2nd moment defined on N;, X2 C Rd. with corresponding probability density functions and Py, respectively. The 2- Wassseesttiin distance between μ4 and μ4 the solution to the optimal mass transportation problem with &z transport cost
33
inf / (EI)())))))) XXXX, ( ((Tr)
1/4
||=; 1,11dd7
where 74,4, the set of all transportation plans y E [( (4y) such that 7(4 x X,) = (A) and (&& x BB) μ, (B) for any Borel subsets A C X; and B C Ay. Due to Brenier's theorem B4] for absolutely continuous probability measures μi; and μy (with respect to the Lebesgue measure), the 2- Wasserstein distance can be equivalently obtained from the Monge formulation [33]
We(mi, ( (n inf (EMPP(EN))) J / ||= ((0 (()))(())()))))
where MP(44, = ( X 1 = (1} and f414 represents the pushforward of measure Ms. characterized as f= (B) Hf(f (BB) for any Borel subset B C Xy. The mapping f is referred to as a transport map [35. and the optimal transport map is called the Monge map. For discrete probability measures, when the transport plan y is deterministic optimal coupling, such a transport plan is referred to as a Monge coupling 33 In case of non-deterministic transport plan 7, one can obtain an approximation of the Monge coupling via barycenteric projection, e.g., see [16] 12] In this paper, we mainly use the 2- Wasserstein distance and hereafter, for brevity, we refer as the Wasserstein distance.
3

---

For one-dimensional probability measures, the Wasserstein distance has closed-fore form solution and can be calculated
We(ue, = 14 - (1
(3)
where F is the quantile function of μ4. The simplicity of calculating Wasserstein distances between one-dimensional probability measures has led to the idea of sliced- Wasserstein 33 and generalized sliced- Wasserstein [40] distances, which we will review next.
3.2 (Generalized) Sliced- Wasserseessr Distances
Let go R2 + R be parametric function with parameters 0 E Ωg C satisffing the regularity conditions in both inputs and parameters as presented in [40] For -aaeed-tessrtteinn distance, go(x) 8" where 9 gd- is a unit vector in Rd, and S denotes the unit d-dimensional
hypersphere. The generalized slice of probability measure μ4 with respect to ge is the one-dimensional
probability measure g0 44, which has the following density for all R.
/ 8(t go(x))dp,(x).
(t)
(4)
The generalized Wasssesesnr distance is then defined as
(SWa(uisH) = (1 Wa (go# (0
1
(5)
Note that for go(x) = 0 and Sw = Sd-1, the generalized sliced- Wasserstein distance is equivalent the Wasserseenn distance. Equation (5) the expected value of the Wasserstein distances
between slices of distributions and jíj-
distance include max (generalized) sliced- Wasserstein distance [38 40), in which the expected value in (5) is substituted with a maximum over Og i.e.
Extensions of the (generalized)
max-gSW2(14, p2)
max W2 W2 Pi; (1)) BE
subspace-robust Wasserstein distance 41 which generalizes the notion of slicing to projection onto subspaces, and the distributional sliced- Wasserstein distance [42] that proposes to replace the expectation with respect to the uniform distribution on Pg with non-uniform and learnable distribution.
From an algorithmic point of view, the expectation in 5 is approximated using Monte-Carlo integration, which results in an average of set of Wasserstein distances between random slices of d-dimensional measures. In practice, however, GSW distances only output good Monte-Carlo approximation using a large number slices, while max-GSW distances achieve similar results with only single slice, although at the cost of an optimization over 0.
PSWE: Pooling by Sliced Wasser-Wassestteinn Embedding
4.1 (Generalized) Sliced- Wasser-aassr Embedding
We propose a Euclidean embedding for probability measures, such that the weighted Euclidean distance between two embedded measures equivalent to the GSW distance between them. Consider set of probability measures J.M. with densities {p: bp For simplicity of notation, let n4f := go#t4 14 denote the slice of measure μ4 with respect to ge. Also, let μi denote a reference measure, with o representing its corresponding slice. Then, it is straightforward to show that the optimal transport map (i.e., Monge map) between μ and pf can be written as
- = F-1 F a

---

where as mentioned before, F. and FT respectively denote the quantile functions of μ and 12 o M Now, letting id denote the identity function, can write the so-called cumulative distribution transform (CDT) 433 of μ as
v =1 id.
(8)
For fixed 0, of satisfies the following conditions, the proof of which can found in the Supple- mentary Material:
Cl: The weighted 2-norm of of equals the Wasserstein distance between uf and M i.e..
|| 11 = (1 11
W2(
hence implying that || lug.2 = O. C2: the weighted & distance between us and of equals the Wasserstein distance between n4 and u., i...
|| 1111, We(m 1g).
Finally, the GSW distance between two measures, Jı; and pay. can be obtained
GSW2(p))))) (1 1|, ~/ 22 ( 10000
Based on (9). for probability measure μ4, the mapping to the embedding space obtained via (11)) = ( 0 E lt.).
4.2
Algorithmic Considerations
In this section, we introduce novel pooling algorithm, termed pooling by sliced- Wasserstein embedding (PSWE). Let X; {ar pi) denote input set with N; elements, and Xa {x" Po J.y denote the set of N samples from trainable reference set. Let θ, {0: ~ Un } denote set of I parameters sampled uniformly from Sz. Then, the empirical distribution of the 7th slice of p; be written ph Monge coupling between P. 20
Em= 5(t go, (=).)) To obtain vit need to calculate the and :th In what follows, we consider two scenarios:
1. When the input set and the reference set have the same cardinalities, i.e., N, = N. the Monge coupling (i.e., the discrete counterpart of the Monge map shown in 7 obtained by sorting X. == (go, brr)))= and X Let Tz denote the permutation indices (i.e., a argsort) obtained by sorting X Then, letting To denote the ordering that permutes the sorted set back to the original ordering based on sorting of elements in X the Monge coupling is obtained via Tı ma(mo [-]]] and the per-slice embedding is calculated as
10.0 = ge: (Ii,,r,
*(m1) -ge, (x
(10)
2. When the set cardinalities vary, the Monge coupling can be obtained via interpolation using 77. In our experiments, we use the Py PoTccc implementation of linear interpolation to evaluate F F-1 The per-slice embedding is calculated as
[ufi]
F-1 15;
Too
[1]
go, (4h)),
(11)
N
where F F, (x..) N4
To "" +1 assuming that the indices start from 0.
Note that, regardless of the cardinality of the input set, the per-slice embedding N-dimensional, i.e., tith N. The final embedding then defined as Φ(x4) = "/ ..... L4ll E N which satisfies GSW2(u4,14 ~ [[ي(())) o(4,) ||,,
(12)
where || 1|| denotes the Frobenius norm, and the approximation is due to the Monte-Carlo integral approximation with the slices.
httpp:://gii htt.:///w...nnalaaattttttrrrrena
5

---

Input: v (+6 M
Sliers Permutation Equivariant)
Ourputs sizens los, (r))
Mapa/Csupings Regalo Rayal
aud wird
Suub
M
M
Muni
R
Halt Set, fot Pollint
(ef + m200) fre [0.11
R
Trainable
Pooliing Sliced- -Wasserstein Embedding (PSWE) characterized
Permutation--mvariant Output
BOCKJ ~ (PTOree))
[e0x] - aZ,1, (aW(ma))
@ 4." (mx,6:3)) (
An overview of the proposed PSWE method. Each d-dimensional element in given input set X,, as well as each element in the trainable reference set X, is passed through multiple trainable then perform interpolation
Figure 1:
slicers {ge, 1 For each slicer, optimal transport maps that push the slicer output distributions of the reference set to the slicer output distributions of given set via 77. (10). and (TT). The resultant transport maps are then concatenated across all slices to derive the final set embedding.
the slicer outputs and derive the
4.3
On Projection Complexity of Sliced Distances
Given the high-dimensional nature of the problems of interest in machine learning, and the fact that samples often live on low-dimensional manifold, one requires large number of random projections, L, to obtain a good approximation of the GSW distance. This issue is related to the projection complexity of the sliced distances [38 40 Given the dependence
our pooling dimensionality on the number of slices, L, we would like to avoid using very large numbers of slices. Here, we devise a unique approach that ties our proposed embedding to metric learning. First, we note that ideas like max- max-GSW 440 38) or subspace-robust Wasserstein distance [4]] would not be practical in this setting, as the slicing parameters, I, are fixed for all probability measures and not chosen separately for each pairs of probability measures (/i, M45).
Given the training input sets, i.e., (x,) JMM and a reference set, Xo. we seek an optimal set of slices O that could be learned from the data alongside the other parameters in an end-to-end manner. This idea is related to [42] as it is similar to learning distribution over the unit hypersphere from which are sampling our L slices. The optimization O ties the PSWE framework to the field of metric learning, allowing us to find slices or, in other words, an embedding with a specific statistical characterization.
To put it all together, our pooling requires identifying: 1) the type of slicer ge R + R (e.g.. go(x) = 0T x), 2) the number of slices, I,, and 3) the number of elements in the reference set, N. Then, for an input set X; with N; elements, PSWE first slices the elements of the input and reference sets with respect to slicers ge, for E {1...., L}. Then, it sorts the sliced values (go, (xr) ) (x 1 and calculates or approximates the corresponding Monge couplings according to (10) or II respectively. Finally, PSWE calculates the per-slice embedding v." and returns ((x)) = 1 ..... v E x.. This procedure depicted in Figure [ as well as Algorithm Note that in proposed framework, the slicer parameters and the reference set elements all trainable parameters that are updated using backpropagation of gradients due to the objective function of interest.
and {ga,
5 Experimental Evaluation
We evaluate the proposed PSWE method variety of point cloud, graph, and image datasets as depicted Figure For comparison, we consider four different pooling methods: Global average pooling (GAP), global max pooling (MAX-evaluate on image classification only), Pooling by

---

Algorithm Pooling by Sliced Wasserstein Embedding
procedure PSWE(X, = {ar E )==))
Trainable parameters: Slicer parameters O2 E x. Reference elements Xg R Nxd
for 1orl=1
1=10 to L do
Calculate ge, (X,) == (go, (=)))== and go, (Xo) {90, [=)))==1
Calculate n; argsort(go, (X,)), m0 = argsort(go, (Xo)), and =u
N; N then
Calculate v according to (T0)
else
Calculate oht according to
return ((x)) [0? 04 E R X
(c)
Figure 2: We evaluate the performance of PSWE and other baseline pooling methods (a) 3D point
cloud classification from ModelNer40 dataset [44], (b) TUD graph classification datasets [45]. and
(c) image recognition NWPU-RESISCA, 46 and Places-Extra69 (77 datasets.
Multi-head Attention (PMA) 14] and Featurewise Sort Pooling (FSPool) [1]. In all the PSWE experiments, to ease the optimization process of reference elements, we optimize the reference clements at the output of the slicers rather than in the input space of the slicers. Morcover, for both PMA and PSWE, to reduce the embedding size, we use a similar weighting approach to that of FSPool, where the output embedding (which is of size N L and N x d for PSWE and PMA, respectively) is multiplied elementwise by a learnable weight matrix W of the same size, and the result is summed over the rows to derive final L-dim and d-dim embedding for PSWE and PMA, respectively. Further details on the experiments can be found in the Supplementary Material.
5.1 Point Cloud Processing
We consider the ModelNet40 dataset [44. consisting of 3-dimensional point clouds derived from triangular meshes of 12.31 CAD models belonging to 40 object categories. We sample 1024 points uniformly at random from each object as in [8] [9] and use the official split, with 9,843 training samples and 2,468 test samples. We consider two different backbones, namely multi-layer perceptron (MLP) and induced set attention block (ISAB) from the Set Transformer architecture [14]
Table shows the accuracy achieved by the proposed PSWE method using different numbers of slices (7. E (1,4, 4, 16, 64, 256, 1024]) and the baseline pooling methods GAP, PMA, and FSPool. As the table shows, for both backbone types, PSWE is able outperform other pooling methods for high-enough numbers of slices, Furthermore, it is noteworthy that while mean-pon mean-pooling does not perform well when using MLP backbone, performing message passing among the set elements using the ISAB backbone significantly boosts GAP's performance, suggesting that simple averaging of the per-element embeddings suffices to achieve a high performance level. This implies that there an inherent trade-off between the backbone and pooling complexity, and to maintain a high accuracy level, at least one
two components should complex enough. comparison the wall-clock training and testing times of PSWE and the baseline pooling methods on the ModelNer40 dataset, as well as experimental results on visualizing the closest and farthest samples to/from the trained reference for PSWE, can be found in the Supplementary Material.
7

---

PSWE
Backbone
PMA
MLP
57.8 + 0.5
86.6
0.6
8588 +
149 10
$2.9 2.1
774 =
0.4
83.9 + 0.6
86.3 0.5
86.9 1: 0.3
ISAB
86.6
87.3
324 144-34
86.2
86.9 4
0.4
Table I: Test classification accuracy (%) of the proposed PSWE method and the baseline pooling mechanisms on the ModelNet40 point cloud dataset using multi-layer perceptron (MLP) and induced set attention block (ISAB) [14] backbones.
Backbome
GAP
PMA
FSPool
L
L 16
L 64
206
1024
IMDB- M IMDB BB
GCN
69.5 d: 3.9
74.1 +: 5.3
73.5 3.7
72,6 i: 7.6
74.6 d: 5.8
17.3 d: 5.1
73.0 à:
72.7 +:
73.5 4: 6.1
73.4 = 3,5
+
72.4 6.9
71.3
74.0 +
70.9 +: 7.6
73.0 366
73.4 +
GIN
73,0 1:
70.0 8.0
73.4 6.1
73.8
72.5 1: 6.0
72.0 d:
72.3 4:
74.6 4: 7.0
68.8 1:
GCN
51.8 =: 4.2
50.1 + 2.8
51.1 =:: 3.4
44.2 4.9
90.7 + 4.6
30,8 =: 3.8
49.6 + 3.6
$1.4 4.2
50.2 1: 4.7
50.2 +
GIN
49.7 = 2.9
50. 3.0
50.8 5.3
44.6 2 4.4
49.1 + 2.7
48.0 + 6.4
50.6 + 2.6
$0.2 3.4
49.5 +
GCN
81.9 = 2.6
X2.2 + 2.5
84.0 2.8
77.5 4.5
$0.4 + 3.3
81.5 = 2.1
82.1 + 3.2
81.9 - 3.2
81.7 + 3.0
RDT
:3.0
GIN
81.2 d: 3.1
77.6 7.9
84.4 = 2.9
83.2 5.3
83.1 + 2.8
83.8 #: 2.8
84.6 +: 2.3
K3.9 -
83.7 + 1.7
ENZYMES PROTEINS
GCN
69.1 1 5.2
72.4 + 5.9
74.9 5.4
72.5 +: 3.9
73.3 1:: 5.3
73.3 2:
73.2 +: 6.1
72.8 1 6.0
73.9 1:
GAT
69.7 4.4
72.4 + 6.1
5.1
72.8
73.9 +: 4.6
74.4 4.4
13.7
GIN
69.8 =:: 6.5
72.3 3333::44 4.7
72.6 22.6:2244 4.5
71.3 1: 4.0
72.4 + 6.0
73.4 - 4.8
73.5 +
73.0 0: 5.0
74.9 +: 3.9
-
$2.1
+
31.8 -
32.5 +: 3.0
4.9
33.7 +
GAT
24.2 1:
28.8 1: 3.9
34.2
22.3 4.0
26.3 5.56
30.6 1:
34.6 03::::3
38.1 :1 5.5
34,9 1:
GIN
29.6 do 6.3
50.1 4:488
43.6 de 6.1
19.1 5.5
25.9 +
4.6
36.5 + 3.1
37.2 5.7
45.4 + 7.0
40.0 +: 6.0
Table 2: Cross-validation accuracy (%) of PSWE with different numbers of slices, as well as baseline pooling methods on different TUD graph classification tasks (45) using three backbones of GCN (17) GAT [18, and GIN [19]. The best performing pooling method in each row (i.e., (dataset, backbone) pair) is highlighted in bold.
5.2 Graph Classification
Next, we consider the prominent TUD benchmark 455 and evaluate the performance of the proposed method on five graph classification datasets, consisting of social network (IMDB-B, IMDB-M. REDDIT-B) and bio-informatics (ENZYMES, PROTEINS) datasets. For the former group of datasets, we use one-hot encoded degrees as initial node features, while for the latter group, we use the provided node labels as initial node features. We then pass the features, alongside the adjacency matrices, three popular graph neural network (GNN) backbones, namely Graph Convolutional Network (GCN) 17,, Graph Attention Network (GAT) [8]. and Graph Isomorphism Network (GIN) 19. Upon deriving the final node embeddings we treat them elements of a set and apply PSWE and the baseline pooling methods to derive fixed-size graph-level representation that is fed to a linear classifier.
given graph from GNN backbone,
Table 2 shows the resulting 10-fold cross-validation accuracies on different datasets, and using different backbone/pooling pairings, following the evaluation methodology used in the literature [19
50 [1]. As the table demonstrates, PSWE is able to perform similarly to or better than other pooling
methods on all datasets. Furthermore, the results show that the commonly used mean-pooling for
GNNs might not be the best choice, and more complex backbones might be needed to enhance the
classification performance. is important to note that in some scenarios, especially with smaller datasets (e.g., IMDB-B / ENZYMES) and more complex backbones (e.g., GIN), the performance of PSWE does not monotonically improve with the number of slices, L., which due the model becoming more complex and overfitting the training data.

---

Dataset
Backbone
MAX
GAP
FSPool
PSWE
16 Patches + MLP
71.0 +:
65.0
77.2
75.0
NWPU-RESISC45
ResNet18
89.3 + 0.5
91.4
90.7 0.3
90.6 0.5
16 16 Patches MLP
18.5 +: 2.2
29.1 + 0.1
35.4 1 1.8
35.2 0.2
Places-Extra69
ResNet18
57.3 + 0.1
58.6 0.0
57.3 + 1.8
58.3 + 0.0
Table 3: Image classification results (% test accuracy) the NWPU-RESISC45 and Places-Extra69 datasets using two backbone types coupled with MAX, GAP, FSPool, and PSWE pooling methods.
5.3 Image Recognition
Finally, we evaluate PSWE in the context of image recognition on two large-scale image datasets:
NWPU-RESISC45 [46,, which is remote sensing image scene classification dataset comprising a total of 31,500 images belonging to 45 different aerial scene classes, and Places-Extra69 [47]. which contains 98,721 training and 6,600 test images, belonging to 69 different scene categories. For processing the images, we consider two different backbone types:
16 16 Patches MLP: Inspired by the architecture used in the Vision Transformer (ViT) framework [52], we break the image into 256 patches, each flattened into a 16x 16x3 = 768- dimensional vector, pass each patch through shared multi-layer perceptron (MLP), add positional encoding to the MLP outputs, and treat the outputs as a set of 256 elements, each with 256 features.
ResNet18 [53] As an alternative, we pass the image through ResNet18, which is convolu- tional neural network backbone, mapping the input image into a 7 -i2-dimensional tensor. We treat this tensor set of 49 elements, each containing 512 features.
Table shows the test classification accuracy of PSWE, as compared to GAP, MAX, and FSPool on the two datasets using the two aforementioned backbones. For PSWE, we set the number of slices to T. 1024 for the 16 x 16 Patches + MI.P backhone, and T. = 1000 for the ResNet18 backbone. We did not include PMA results here as it performed significantly worse than other pooling types. As the table shows, PSWE performs on par with FSPool using both backbones, and significantly better than GAP with the simpler MLP-based backbone. This is consistent with our observation that more sophisticated pooling mechanisms can compensate the performance drop caused by simpler backbone architectures.
Conclusion
We introduced a novel method for permutation-invariant feature aggregation from set-structured data, called pooling sliced-Wassersse Wasserstein embedding (PSWE). Our method treats the elements of each input set as samples from a distribution, and derives a constant-size representation for the entire set based on the (generalized) sliced-Waa Wasserstein distance between the set elements and a reference set, whose elements are learned in an end-to-end fashion, alongside with the slicer parameters. We showed for
that our method derives an exact Euclidean embedding which is coometrically-interpeeaaa structured data. Moreover, we demonstrated, through experimental results, that our set embedding approach outperforms baseline pooling mechanisms on variety of supervised classification tasks on point cloud, graph, and image datasets. While our focus in this work was on deriving global representations for input samples (such as point clouds, graphs, and images), our method is not necessarily limited to global pooling. Indeed, our approach is generic mechanism for embedding an input set into a fixed-dimensional representation and, therefore, may also used for local pooling, which is an interesting direction for future work.

---

Acknowledgments and Disclosure of Funding
This material is based upon work supported by the United States Air Force under Contract No. FA8750-19 19-0--998- Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the United States Air Force and DARPA.
References
[1] Tony Jebara, Risi Kondor, and Andrew Howard. Probability product kernels. The Journal of Machine Learning Research, 5:819-844, 2004.
[2] Arthur Gretton, Karsten Borgwardt, Malte Rasch, Berhhard Schölkopf, and Alex Smola. A kernel method for the two-sample-ae prooble. Advances in neural information processing systems, 19:513 520, 2006.
[3] Oren Boiman, Eli Shechtman, and Michal Trani. In defense of nearest-neighhor based image classification. In 2008 IEEE Conference on Computer Vision and Puttern Recognition, pages 1-8, IEEE, 2008.
Barnabás Póczos, Liang Xiong, and Jeff Schneider. Nonparametric divergence estimation with applications to machine learning on distributions. Proceedings of the Twenty-Sevent Conference on Uncertainty in Artificial Intelligence, pages 599-608, 2011.
[5] Barnaáss Póczos and Jeff Schncider. Nonparametric estimation of conditional information and divergences. In Artificial Intelligence and Statistics, pages 914 923. PMLR, 2012.
[6] Krikamol Muandet, Kenji Fukumizu, Francesco Dinuzzo, and Berhhard Schölkopf. Learning from distributions via support measure machines, In Proceedings of the 25th International Conference on Neural Information Processing Systems- Volume 1, pages 10-18, 2012.
[7] Liang Xiong and Jeff Schneider. Learning from point sets with observational bias. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, pages 898 906, 2014.
[8] Soheil Kolouri, Yang Zou, and Gustavo K Rohde. Sliced- Wasserstein kernels for probability distributions. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 4876-4884, 2016.
[9] Matteo Togninalli, M Elisahetta Ghisu, Felipe Llinares-Lépez, Bastian Rieck, and Karsten M Borgwardt. Wasserstein weisfeiler-lehman graph kernels. NeurlPS, 2019.
[10] Konstantinos Skianis, Giannis Nikolentzos, Stratis Limnios, and Michalis Vazirgiannis. Rep the set: Neural networks for learning set representations. In International conference on artificial intelligence and statistics, pages 1410-1420. PMLR, 2020.
[11] Manzil Zaheer, Satwik Kottur, Siamak Ravanbhakhsh. Barnabás Póczos. Ruslan Salakhutdinov. and Alexander J Smola. Deep sets. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 340444444, 2017.
[12] Grégoire Mialon, Dexiong Chen, Alexandre d' Aspremont, and Julien Mairal. A trainable optimal transport embedding for feature aggregation and its relationship attention. In International Conference Learning Representations, 2021.
[13] Ryan Murphy, Balasubramaniam Srinivasan. Vinayak Rao, and Bruno Ribeiro. Janossy pooling: Learning deep permutation invariant functions variable-size inputs. In International Conference on Learning Representations, 2019.
[14] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, Whye Teh. Set transformer: sermutation-invarian neural networks. In International Conference on
A framework for attention Machine Learning, pages 3744-3753, PMLR, 2019.
[15] Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett. Fspool: Learning set representations with feature- wise sort pooling, In International Conference on Learning Representations, 2020.
[16] Soheil Kolouri, Navid Naderializadeh, Gustavo K. Rohde, and Heiko Hoffmann. Wasserstein embodding for graph learning, In International Conference on Learning Representations, 2021.
[17] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference Learning Representations (ICLR), 2017.
10

---

[18] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
[19] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations. 2019.
[20] Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings document distances. In International conference on machine learning, pages 957-966. PMLR, 2015.
[21] Gao Huang, Chuan Guo, Matt J Kusner, Sun, Fei Sha, and Kilian Q Weinherger. Supervised word mover's distance. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, pages 1962--870 Curran Associates, Inc., 2016.
[22] Chi Zhang, Yujun Cai. Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12203-12213 2020.
[23] Jincheng Li, Jiezhang Cao, Shuhai Zhang, Yanwu Xu, Jian Chen, and Mingkui Tan. Internal wasserstein distance for adversarial attack and defense, arXiv preprint arXiv:2/03.0759, 2021.
[24] Mauricio Delbracio, Hossein Talebi, and Peyman Milanfar. Projected distribution loss for image enhance- ment. arXiv preprint 20XV:22012.09289, 2020.
[25] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamoniy. Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 19(9):1853-1888 2017.
[26] Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European Conference Computer Vision (ECCV), pages 447-463, 2018
[27] David Alvarez Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. Advances Neural Information Processing Systems, 33, 2020.
[28] Yan Zhang, Jonathon Hare, and Adam Prügel-Benneth Learning representations sets through optimized permutations. In International Conference on Learning Representations, 2019.
[29] Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets, arXiv preprint X5v
1..3911 1,0391,, 2015.
[30] S Hamid Rezatofighi, Roman Kaskman, Farbod T Motlagh, Qinfeng Shi, Daniel Cremers, Laura Leal- Taixé, and Ian Reid. Deep perm-set net: Learn to predict sets with unknown permutation and cardinality using deep neural networks. arXiv preprint ftti.:v80000000 2018.
[31] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention. arXiv preprint arXiv:2103.03206. 2021.
[32] Tricu Trinh, Minh Minh-Thang Luong, and Quoc Le. Selfie: supe-supe pretraining for image embedding, arXiv preprint arXiv:1906.0244 2019.
[33] Cédrie Villani. Optimal transport: old and new, volume 338. Springer Science Business Media, 2008
[34] Yann Brenier. Polar factorization and monotone rearrangement vector-valued functions. Communications on pure and applied mathematics, 44(4):375-417, 1991
[35] Soheil Kolouri, Se Rim Park, Matthew Thorpe, Dejan Slepcev, and Gustavo K Rohde. Optimal mass transport: Signal processing and machine-learning applications. IEEE Signal Processing Magazine, 34(4):43-59, 2017.
[36] Julien Rabin, Gabriel Peyré, Julie Delon, and Mare Bernot. Wasserstein barycenter and its application texture mixing. In International Conference on Scale Space and Variational Methods in Computer Vision, pages 435-446. Springer, 2011.
[37] Nicolas Bonnotte, Unidimensional and evolution methods for optimal transportation. PhD thesis, Université Paris 11, France, 2013.
[38] Ishan Deshpande, Yuan- Yuan-Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander Schwing. Max-sliced wasserstein distance and its use for gans. In IEEE Conference Computer Vision and Pattern Recognition, 2019.
11

---

[39] Julien Rabin, Gabriel Peyré, Julie Delon, and Mare Bernot. Wasserstein barycenter and its application to texture mixing. In Scale Space and Variational Methods in Computer Vision, pages 435-446. Springer. 2012.
[40] Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. Generalized sliced wasserstein distances. Advances in Neural Information Processing Systems, pages 261-272. 2019.
[41] François-Pierre Paty and Marco Cuturi. Subspace robust wasserstein distances. In International Conference on Machine Learning, 2019.
[42] Khai Nguyen, Nhat Ho. Tung Pham, and Hung Bui. Distributional sliced-wasserstiin and applications generative modeling. International Conference on Learning Reprexentations, 2021.
[43] Se Rim Park, Soheil Kolouri, Shinjini Kundu, and Gustavo K Rohde. The cumulative distribution transform and linear pattern classification. Applied and Computational Harmonic Analysis, 45(3);616 641, 2018.
[44] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang. and Jianxiong Xiao. shapenets: A deep representation volumetric shapes. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[45] Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Gruph Representation Learning and Beyond (GRL+ 2020). 2020.
[46] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, (85 1883, 2017.
[47] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[48] Charles R Qi. Hao Su. Kaichun Mo. and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pages 652-660 660. 2017.
[49] Meng- Hao Guo, Jun- Xiong Cai, Zheng-Ning Liu, Tai- Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. arXiv preprint arXiv:2012.00 096688, 2020.
[50] Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1365-1374, 2015.
[51] Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. International conference on machine learning, pages 2014-2023 PMLR, 2016.
[52] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[53] Kaiming He, Xiangyu Zhang, Shaoqing Rea, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
12


Reviews:
[{"review_id": "zOKxBp-KlD", "reviewer": "Reviewer_xtA8", "summary": "The paper proposes a pooling mechanism for embedding sets of various sizes into a fixed-size representation with a novel and theoretically-grounded pooling approach. The proposed Pooling by Sliced-Wasserstein Embedding (PSWE) is simple to compute and permutation-invariant. This pooling provides an exact Euclidean embedding for the generalized sliced-Wasserstein distance. Experimental results show that this pooling can be used in different kinds of data to boost performance, when comparing with basic pooling mechanism on plain backbones.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n\tThe proposed Pooling by Sliced-Wasserstein Embedding is novel for permutation-invariant feature aggregation from set-structured data. The proposed method seems based on this following paper \n\nGeneralized Sliced Wasserstein Distances, arXiv 2019.\n\nAnd this work is an extension of the implementation/realization of a pooling module.\n\nQuality:\nStrengths:\n -  The theory is clearly defined and proven. Section 3 introduces the previous work on theorical background. Section 4 described the proposed pooling method based on Section 3 in dealing with the two considered scenarios and overall algorithm.\n\nWeaknesses:\n - The result of the Image Recognition experiment and the result of the PSWE pooling only improve the performance by a little when compared to Global Average Pooling (GAP).\n - For the comparison of basic pooling methods, I suggest the authors to add more methods such as maximum pooling that are frequently used for comparison (in Table 3).\n\nClarity:\nStrengths:\n - The paper is well-organized in introducing the basic theory and background to the description of the proposed method.\nWeaknesses\n - Some of the symbols in the paper is confusing and do not clearly explain those introduced in the previous papers. \n - It is hard to re-implement without more detail about the modification of the model in the Image Recognition, e.g. which layer is modified.\n - In Line 103, the symbol d, i were not explained which I guess is the channel size of the feature map and hidden layer number i.\n - In Line 159, the symbol f_i^θ was not explained.\n\nSignificance:\nThe result shows the proposed PSWE pooling obtains better performance in the Point Cloud Classification and Graph Classification task. However there is not much distinguishable improvement in the Image Classification task.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. See Checklist 1.(c).", "review_text": "", "extra_scores": {"ethical_concerns": "The proposed method is a general pooling technique, so I think there is no ethical concern involved.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "qU7FPd4hRMJ", "reviewer": "Reviewer_iGeu", "summary": "The paper proposes a method for learning how to aggregate a set of vector representations into a single representation for the whole set. This has applications in many tasks such as the common case of image recognition or representation where a 3D feature maps is pooled across the spatial dimensions, while other applications include point clouds and graph learning. The proposed approach is theoretically grounded and consists of an approximation of the 2-Wasserstein Distance. Experimental results contain experiments in 3 tasks: graph classification, point cloud classification, and image recognition.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- the proposed approaches has many applications - beyond the ones included in the experimental validation\n- this is a novel approach that is theoretically grounded\n- the fact that the final embeddings are not in the form of a matrix NxL but of a vector 1xL is mentioned only in the supplementary material. The matrix is multiplied with a learnable matrix, then, rows are summed and the embeddings are compared with Euclidean distance. I believe this information need to appear in the main paper. This should introduce performance loss (compared to Frobenious norm on the matrix embeddings). The performance of this is not mentioned in the paper. Despite the fact that this would be a more costly approach, it is useful to see what are the losses introduced by this last part of summing along rows. \n- In table 2, there is large performance drop in some of the cases for L=1024. This is not discussed in the paper. In my understanding, the larger the L the better the approximation. Is there some explanation for this?\n- is the trainable reference set explainable in any sense? This set of vectors lives in the same space as the input vectors in the considered sets. For instance, in the case of image recognition with 16x16 patches, similarity of each d dimensional vector in the reference with  patches from particular classes of the training set can be used to see association of these vectors with different classes. This as an illustrative example, but probably many more. \n\n-----\npost rebuttal: \nI am still positive about the paper. The authors have provided detailed explanations which are very satisfactory. It is true that in some of the cases the performance is very close to that of GAP which is a simple as it gets. In some of the experiments the improvement over GAP is very large, while in others non-existing - some elaborated explanation of this discrepancy is missing. This is an aspect to improve in the camera ready. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "not discussed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "oRQpxzAzd0", "reviewer": "Reviewer_vGLg", "summary": "In this paper, the authors propose a method to compare sets by using a probability measure between two embedded sets. This is achieved through the sliced Wasserstein distance (SWD) between their empirical distributions. The method provides theoretical guarantees to prove the validity of their proposed measures. The use of SWD for sets is demonstrated for point cloud classification, graph classification and basic image classification. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The use of SWD for learning architectures on sets is demonstrated using the idea of obtaining Euclidean embedding for the probability measures. The main ideas used are relying on cumulative distribution transform to obtain the distance measure. This then allows definition of the generalized sliced Wasserstein (GSW) to be used to measure the distance between two slices. The challenge then is to obtain slicer functions. These are obtained for the whole dataset using a reference set X_0. The procedure then is that given a dataset and a reference set, a number of slicers that are permutation equivariant are learned. These are used to obtain optimal transport maps between the slices. These transport maps are concatenated to obtain the final permutation invariant embeddings.\n\nPros:\n1) The use of probability measures to obtain pooled SW distance functions is novel and this is related to other pooling functions.\n2) The method is theoretically grounded using the CDT \n3) The method is demonstrated on a number of different settings ranging from point-cloud classification to image recognition\n\nCons:\n1) The paper would be improved by obtaining a more detailed ablation analysis of their method\n2) The method is focused on comparing different set based datasets for the various architectures such GAT, GCN using the proposed pooling methods and comparing against the other pooling functions. However, the method can also be seen as proposing a novel GSW distance metric. Thus comparison against other such methods such as the max-sliced Wasserstein distance and other GSW distances in their settings could also be useful. \n\nIn conclusion, I believe the work provides a novel enough contribution that is interesting and useful. I therefore have a positive view about the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The work is more mathematical in nature and is not expected to have a significant societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {"ethical_concerns": "The work uses publicly available datasets for set based evaluation and comparisons. Moreover, the contribution is mainly mathematical and is therefore not expected to have significant ethical concerns."}}, {"review_id": "3SlEB45fWIW", "reviewer": "Reviewer_SDvh", "summary": "\nSummary\n=======\n\nContext:\n--------\n\nThis paper tackles the deep learning setting where samples do not belong to a fixed dimensional space but\neach sample is a set of potentially different cardinality. Since samples are sets, constructing learning or embedding functions\nin this setting must be invariant to the order of their elements. The focus in this paper is on deep learning, specifically pooling layers\nfor set samples.\n\n\nContribution\n------------\n\nThe authors propose to model each set (data observation x_i with cardinality N_i) as samples (x_i) draw from some distribution \\mu_i. \nThen, the authors show how to construct an embedding (pooling) function f that takes a measure \\mu_i as input such that the euclidean distance\nin the embedding space approximates the Sliced Wasserstein distance between the measures (sets):\n\n                              \\| f(\\mu_i) - f(\\mu_j) \\| ~ SlidedWasserstein(\\mu_i, \\mu_j)\n\nThe output of the function f(\\mu_i) is a vector \\nu_i \\in R^[N, L] where L is the number of slices used and N is the cardinality of a reference\nset that is learned. The directions of the L slices are also learned during training. \n\nThe authors illustrate the performance of their pooling layer on various experiments showing  near state of the art performance specially for large L.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "review\n======\n\nGeneral assessment and main concerns\n-----------------------------------\nThis paper is very well written and illustrated. The idea of using the 1D Monge map as an embedding output to preserve the GSW distances\nin the output layer is brilliant, even though the experimental results are not significantly outperforming the baselines.\n\nMy main concerns are:\n\nA) Given that the experiments have all similar results, I was left unsatisfied with regards to how crucial the role played by OT is.\nPerforming controlled simulated experiments could have answered questions like this one (+ the comments below) by controlling for e.g the dimension\nof the elements of the set and computing the exact OT vs different values of L vs an MMD based network.\n\nB) The geometrical motivation behind the work (as mentioned in the abstract and introduction). I'm afraid I don't see how\n\"geometry\" plays any role here. While it certainly could in certain applications, the geometrical argument was not exploited nor motivated enough in this\npaper to be a convincing argument. It seems like the Wasserstein distance was taken merely for: (1) the ability to compare measures regardless of the number\nof samples; (2) the use of 1D slices to use fast argsort operations and have 1D couplings that preserve the distance in the embedding space. Given that The geometry here\n(the ground metric) was set to L2 regardless of the nature of the data. So i'm wondering whether using a better informed (or simply a different) cost could improve / degrade\n the obtained comparisons. \n\n\nSpecific comments\n-----------------\n\n1) The introduction should provide a formal statement of the context (each observation is a set with cardinality N_i ..); a permutation invariant function\nshould verify ... I found the first read of this paper quite difficult without going through the Deep Sets paper again. \n\n2) If I understand properly, the reference measure could be fixed to any arbitrary measure. Does learning it improve the performance of the model\nby a significant margin ?\n\n3) Since it is learned here for optimal results, could it be interpreted or visualized in the context of images for instance ? \n\n4) Performances being equal, how does the proposed method compare in terms of run time ?\n\nminor:\n5) The proposed method has an output in R^(N, L) which is likely to have more elements than the input set. Moreover, it learns\nan embedding that preservers GSW distance across sets. This is more likely to correspond to an entire network than a \"pooling\" \noperation. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers are positive about the work and also appreciated the rebuttal, where extra numerical experiments were promised, which I strongly encourage to add. Also an in-depth discussion of the variations in relative performances with respect to GAP seems important. As a small side note, the notion of SW was introduced in [38] and not [36, 37] as the end of Section 3.1 seems to imply (in particular [38] is not concerned with generalized SW). 
Decision: Accept (Poster)