Learning Prototype-oriented Set Representations for Meta-Learning 
Authors: Dan dan Guo
Long Tian
Minghe Zhang
Mingyuan Zhou
Hongyuan Zha
Venue: ICLR 2022 Poster (2022)
Keywords: Summary Networks
Distribution Matching
Optimal Transport
Few-shot Classification
Meta Generative Models

Abstract: Learning from set-structured data is a fundamental problem that has recently attracted increasing attention, where a series of summary networks are introduced to deal with the set input. In fact, many meta-learning problems can be treated as set-input tasks. Most existing summary networks aim to design different architectures for the input set in order to enforce permutation invariance. However, scant attention has been paid to the common cases where different sets in a meta distribution are closely related and share certain statistical properties. Viewing each set as a distribution over a set of global prototypes, this paper provides a novel prototype-oriented optimal transport (POT) framework to improve existing summary networks. To learn the distribution over the global prototypes, we minimize its regularized optimal transport distance to the set empirical distribution over data points, providing a natural unsupervised way to improve the summary network. Since our plug-and-play framework can be applied to many meta learning problems, we further instantiate it to the cases of few-shot classification and implicit meta generative modeling. Extensive experiments demonstrate that our framework significantly improves the existing summary networks on learning more powerful summary statistics from sets and can be successfully integrated into metric-based few-shot classification and generative modeling applications, providing a promising tool for addressing set-input and meta-learning problems.

Published as a conference paper at ICLR 2022
LEARNING PROTOTYPE-ORIENTED SET REPRESENTA- TIONS FOR META-
Dandan Guo ' Long Tian', Minghe Zhang', Mingyuan Zhou', Hongyuan Zha
'The Chinese University of Hong Kong, Shenzhen "Xidian University
*Georgia Institute of Technology "The University of Texas at Austin
guodandan cuhk. edu. en tianlongexian xxinngxxxii xingeexxiia...m edu. en mmhang888@atecch.ed migyuan. zhouêmccombs, utexas. edu thahy@cuhk.edu.c
ABSTRACT
Learning from set-structured data is fundamental problem that has recently at- tracted increasing attention, where a series of summary networks are introduced deal with the set input. In fact, many meta-learning problems can be treated as set-input tasks. Most existing summary networks aim to design different ar- chitectures for the input set order to enforce permutation invariance. However, scant attention has been paid to the common cases where different sets in a meta- distribution are closely related and share certain statistical properties. Viewing each set as distribution over set of global prototypes, this paper provides a novel prototype- orriented optimal transport (POT) framework to improve exist- ing summary networks. To learn the distribution over the global prototypes, we minimize its regularized optimal transport distance to the set empirical distribu- tion over data points, providing natural unsupervised way improve the mary network. Since our plug-and-play framework can be applied to many meta- learning problems. we further instantiate the cases of few-shot classification and implicit meta generative modeling. Extensive experiments demonstrate that our framework significantly improves the existing summary networks on learning more powerful summary statistics from sets and can be successfully integrated into metric-based few-shot classification and generative modeling applications, providing a promising tool for addressing set-input and meta-learning problems.
INTRODUCTION
Machine learning models, such as convolutional neural networks for images (He et al. 2016) and recurrent neural networks for sequential data (Sutskever et al. 2014), have achieved great success in taking advantage of the structure in the input space (Maron al. 2020). However, extending them to handle unstructured input in the form of sets, where a set can be defined as an unordered collections of elements, is not trivial and has recently attracted increasing attention (Jurewicz & Strømberg- Derezynski 2021). Set-input is relevant to a range problems, such as understanding a scene formed of set of objects (Eslami et al. 2016), classifying an object composed of set of 3D points (Qi et al. 2017), summarizing a document consisting of a set of words (Blei et al.) [2003] Zhou et al., 2016), and estimating summary statistics from set of data points for implicit generative models (Chen et al. 2021). Moreover, many meta-learning problems. which process different but related tasks, may also be viewed as set-input tasks (Lee al.) 2019). where an input corresponds to the training dataset of a single task. Therefore, we broaden the scope of set-related applications by including traditional sef-structured input problems and most meta-learning problems. Both of them aim to improve the quick adaptation ability for unseen sets, because of limited samples or the occurrence new categories for classification problems.
though the latter is more difficult
For set-input, the output of the model must not change if the elements of the input set are reordered, which entails permutation invariance of the model. To enforce this property, multiple researchers have recently focused designing different network architectures, which can be referred to summary network for compressing the set-structured data into a fixed-size output. For example, the prominent works of Zaheer al. (201)) and Edwards & Storkey (017) combined the standard

---

Published as a conference paper at ICLR 2022
feed-forwart neural networks with a set-pooling layer, which have been proven to be universal ap- proximators of continuous permutation invariant functions. [ee et al. (2019) further introduced Set Transformer encode and aggregate the features within the set using multi-head attention. [Maron et al. (2020) designed deep models and presented a principled approach to learn sets of symmetric elements. Despite the effectiveness and recent popularity of these works in set-input problems, there are several shortcomings for existing summary networks, which could hinder their applicability and further extensions: 1) The parameters of the summary network are typically optimized by task- specific loss function, which could limit the models' flexibility. 2) A desideratum of summary network is to extract set features, which have enough ability to represent the summary statistics of the input set and thus benefit the corresponding set-specific task; but for many existing summary networks, there is no clear evidence or constraint that the outputs of the summary network could de- scribe the set's summary statistics well. These limits still remain even with the recent more carefully designed summary networks, while sets with limited samples further exacerbate the problem.
To address the above shortcomings, we present novel and generic approach to improve the sum- mary networks for set-structured data and adapt them to meta-learning problems. Motivated by meta-learning that aims to extract transferable patterns useful for all related tasks, assume that there are K global prototypes (i.e., centers) among the collection of related sets, and each proto- type or center is encouraged capture the statistical information shared by those sets, similar the "topic" in topic modeling (Blei et al. [200] /hoo et al., 2016) or "dictionary atom" in dictionary learning (Aharon et al., 2006; Zhou al. 2009). Specifically, for the jth set, we consider it as a discrete distribution P3 over all the samples within the set (in data or feature space). At the same time, we also represent this set with another distribution 22 (in the same space with 12)), supported on K global prototypes with a K K-dimensional set representation hj. Since h measures the impor- tance of global prototypes for set j. it can be treated as the prototype proportion for summarizing the salient characteristics of set j. Moreover, the existing summary networks can be adopted to encode set j as h for their desired property of permutation invariance. In this way, we can formulate the learning of summary networks as the process of learning P, to be as close to Q, as possible, pro- cess facilitated by leveraging the optimal transport (OT) distance (Peyré & Cuturi 2019). Therefore, the global prototypes and summary network can be learned by jointly optimizing the task-specific loss and OT distance between P, and Q, in an end-to-end manner. We can refer to this method as prototype-oriented OT (POT) framework for meta-learning, which is applicable unsupervised and supervised tasks, such as set-input problems solved by summary networks, meta generation (Hong al. 2020c Antoniou al. 2017), metric-based few-shot classification (Snell et al. 2017). and learning statistics for approximate Bayesian computation (Chen et al., (2021). We
range of
note our construction has drawn inspirations from previous works that utilize transport based loss
between a set of objects and a set of prototypes (Tanwisuth et al. 2021 Wang et al. [2022). These
works mainly follow the bidirectional conditional transport framework of /Zhen & Zhou (022))
instead of the undirectional OT framework, and focus on different applications.
Since our plug-and-play framework can be applied to many meta-learning problems, this paper fur- ther instantiates it to the cases of metric-based few-shot classification and implicit meta generative modeling. We summarize our contributions as follows: (1) We formulate the learning of summary network as the distribution approximation problem by minimizing the distance between the distribu- tion over data points and another one over global prototypes. (2) We leverage the POT to measure the difference between the distributions for in joint learning algorithm.
(3) We apply our method to metric-based few-shot classification and construct implicit meta generative models, where a sum- mary network is used extract the summary statistics from set and optimized by the POT loss. Experiments on several meta-learning tasks demonstrate that introducing the POT loss into existing summary networks can extract more effective set representations for the corresponding tasks, which can also be integrated into existing few-shot classification and GAN frameworks, producing new way to learn the set' summary statistics applicable to many applications.
2 BACKGROUND
2.1 SUMMARY NETWORKS FOR SET-STRUCTURED INPUT
To deal with the set-structured input D, (æ,,1:N,) } and satisfy the permutation invariance in set, a remarkably simple but effective summary network is to perform pooling over embedding vectors extracted from the elements of set. More formally,
So (D₂)= = Fos (pool (fSen (2x,1) (..)) (IINN,) ())N,)))
(1)
2

---

training, Published as a conference paper at ICLR 2022
where fer (-) acts each element of a set and I0z (pool(.)) aggregates these encoded features and produces desired output, and Φ = {01, d2} denotes the parameters of the summary network. Most network architectures for set-structured data follow this structure; see more details from previous works (Lee et al. 2019; Zaheer et al. 2017: Edwards & Storkey 2017: Maron et al. 2020).
2.2 OPTIMAL TRANSPORT
Although OT has a rich theory, we limit our discussion to OT for discrete distributions and refer the reader to Peyré & Cuturi (2099) for more details. Let consider p and q as two discrete probability distributions on the arbitrary space X S RR, which can be formulated as p = =Ei= agox, and
9 Σy= uyd this case, a E E" and b E E", where Σx denotes the probability simplex of R. The OT distance between and is defined as
OT(a, b)
min (T, (a.(a.
(2)
where (:,-) means the Frobenius dot-product; C "2"0" is the transport cost function with element Rax denotes the doubly stochastic transport probability matrix such that bj: Th Tiy = a.). To relax the time-consuming problem when opti- mising the OT distance, Cuturi 2013) introduced the entropic regularization, H = Eij Tip In Tij.
Cy C(r,, M2): U(a,b) b) = E Tr >0
E
leading to the widely-used Sınkhorn algorithm for discrete OT problems.
3 PROPOSED FRAMEWORK
In meta-learning, given meta-distribution PM of tasks, the marginal distribution Py of task j is sam- pled from P. for j E J, where J denotes a finite set of indices. E.g., we can sample Pj from pM with probability 1 when PM is uniform over finite number of marginals. During direct access to the distribution of interest py is usually not available. Instead, we will observe a set of data points Dj - } ) which consists of N2 i.i.d. samples from pj over Rd. We can roughly treat the meta-learning problems as the set-input tasks, where dataset Dj from P/ corresponds to an input set. To learn more representative features from related but unseen sets in meta-learning problems, we adopt the summary network as the encoder to extract set representations and improve by introducing the OT loss and global prototypes, providing many applications. Besides, we also provide the applications to metric-based few-shot classification and implicit generative framework by assimilating the summary statistics. Below we describe our model detail.
3.1 LEARNING GLOBAL PROTOTYPES AND SET REPRESENTATION VIA OT
Given J sets from meta-distribution р.M can represent each set Dj from meta-distribution P. as an empirical distribution over N, samples on the original data space, formulated as
P3 N₂ 1 Σ 1=1 N, Jags Bas TR.
(3)
Since all sets (distributions) drawn from meta-distributee PM are closely related, it is reasonable assume that these share some statistical information. Motivated by dictionary learning, topic
modeling, and two recent prototype-oriented algorithms (Tanwisuth et al. 2021 Wang et al. 2022). we define the shared information the learnable global prototype matrix where K represents the number of global prototypes and Be denotes the distributed representation of the k-th prototype in the same space of the observed data points (e.g., "topic" in topic modeling).
[.3x) E gaxx
Given the prototype matrix B, each set can be represented with K K-dimensional weight vector h E Es (e.g., "topic proportion" topic modeling). where hisk means the weight of the prototype By for set Hence, we can represent set Dj with another distribution Q, prototypes Bik::
K k- - hykgg, B& Rd
-
(4)
where h2 is set representation for describing set j. Since set j can be represented as Q, and Pj. can learn set-specific representation h, and prototype matrix B by pushing Qj towards Pj:
N₂ K
OT(Р,,Q;) = -min (T def. Σ B,
Σ
Cak
(5)
3

---

Published as a conference paper at ICLR 2022
Global centers
Set 1
BL.K
Summary Network
h
0
Pool
Set j
9
0.2
0.4
0.4
Set J
Figure An overview our proposed framework, where "pool" operation including mean, sum, max or similar. We compute the representation hy for set j using the summary network in Equation [ which is the weight vector of the global prototypes (i.e., centers) B in the corresponding set.
where C E N is the transport cost matrix. In this paper, to measure the distance between 20
data point my in set j and prototype Bx. unless specified otherwise, we construct C as Cik = 1 -coos (z Ak), which provides an upper-bounded positive similarity metric. Besides, the transport
probability matrix T E N should satisfy IT b) == {T Tlx = a, T = =b} with Ti = T(zji, BB), where I E N and b [h E ΣK denote the respective probability
vectors for distribution P, in Equation and Qj in Equation 4
Since h; should be invariant to permutations of the samples in set J. we adopt a summary network to encode the set of N, points. For unsupervised tasks, taking the summary network in Equation as the example, we can directly add a Softmax activation function into S to enforce the simplex constraint in set representation ft,, denoted h2 Softmax(Ss(DD)) As shown in Fig. [ given sets, to learn the global prototype matrix B and summary network parameterized by ø, we adopt the entropic constraint [Cuturi 2013) and define the average OT loss for all training sets as
1 J
N Σ
N₂ >Σ
LPOT
min B. J
Σ
CaTT
-TalnTTix
min
1 B. T (OT.((P,))))))
(6)
where e is a hyper-parameter for entropic constraint. Algorithm I describes the workflow of the POT loss for improving summary network under unsupervised tasks. For supervised tasks, set is denoted as D, - (,,,,,N,,, yj where y, the ground-truth output determined by specific tasks. As h, is normalized weight vector, directly using it realize the corresponding task may be undesired. Denoting z, = ppoo (ffé, (æ11),... ..... for (HyN,) })). we project it to the following vectors:
hj = Ja(z)), Ú fx(=))
where h, and ÿJ are responsible for the POT and task-specific losses, respectively. Now the sum- mary network parameters Φ= {e, A, Φ1 and global prototypes B are learned by jointly optimizing the task-specific loss (computed by Ú, and y, and OT loss Equation ( In summary, minimiz- ing the POT loss defined by the prototype distribution Q, and empirical distribution P3 provides principled and unsupervised way to encourage the summary network to capture the set's summary statisties. Therefore, plug-and-play framework can integrate suite of summary networks and realize efficient learning from new sets for both unsupervised and supervised tasks.
3.2 APPLICATION TO METRIC- FEW-SHOT CLASSIFICATION
As a challenging meta-learning problem, few-shot classification has recently attracted increasing at- tention, where one representative method is metric-based few-shot classification algorithms. Taking the ProtoNet (Snell et al. 2017) as an example, we provide a simple but effective method to im- prove its classification performance with the help of POT and summary network, where we refer the reader to ProtoNet for more details. ProtoNet represents each class by computing an M-dimensional representation z; E IR M with an embedding function for Rd R where we adopt the same øi as the learnable parameters, following the feature extractor in summary network in Equation [] for simplicity. Formally, ProtoNet adopts the average pooling to aggregate the embedded features of the support points belonging to its class into vector Z,
|5 Σ (x,wnn)s, fo, (xj.). ProtoNet
4

---

Published as a conference paper at ICLR 2022
Algorithm The workflow of POT minimizing the OT distance between Pj and Qj.
Datasets Pr., batch size m., learning rate a, initial summary network parameters initial global prototype matrix B. cost function C7 and hyper-paare aeeer--rammterr c. while B, & has not converged do Randomly choose j from 1,2, .
Require:
Sample the real data {æt, h==1 from set j. which is denoted empirical distribution Pj in Equation 3
Compute the set representation h, =Softmax(S ) with summary network in Equation
Represent the Qj with global prototype matrix and statisties h in Equation 4
Compute the loss OT, (P3, Q2) between Pj and Q, with Sinkhorn algorithm in Equation 6
B <- B + agg, where gB <- Vn (PT.(P,, Q,)l: & <- o age, where go <- V (T. [T; Q,)];
end while
then compares the distance between query point for (xx) to the z, in the same embedding space. Motivated by the summary network, we further introduce a feed-forward network Id, to map the zj into the h used to define Qj distribution over B. Therefore, the functions Id, and fø, can be jointly optimized by minimizing the POT loss and the original classification loss in ProtoNet,
J
N
min V(di, $2, B) = LPOT + .r.B
CLS(UjG, 0zx)
(8)
where hy is used for computing the POT loss, N, the number of samples in class j. Dys the pre- dicted label for sample Ztc. conditioned on ž1:J and fe, (zzze), and CLS the classification loss. Only introducing matrix B and 962, whose parameters are usually negligible compared to før, our pro- posed method can benefit the metric- few-shot classification by enforcing the sor to learn more powerful representation 2j of each class and feature f0, (a)) of the query sample.
3.3 APPLICATION IMPLICIT META GENERATIVE MODELS
Considering implicit meta generative modeling is still a challenging but important task in meta- leaming, we further present how construct the model by introducing set representation h, as summary statisties, where we consider GAN-based implicit models. Specifically, given Pj PM. we aim to construct a parametrized pushforward (i.e., generator) of reference Gaussian distribution P. denoted as Po)-,,))Ep, to approximate the marginal distribution Pj. where o summarizes the parameters of pushforward. Since it is unaccessible to the distribution of interest Py, we replace pj with P, and use the summary network encode set D, into h, as discussed above, which is further fed into the generator serving as the conditional information, denoted as To; h2), z ~ To enforce the pushforward (2; h2) to fit the real distribution P, as well as possible, we introduce discriminator fw following the standard GAN (Goodfellow et al. (2014). Generally, our GAN-based model consists of three components. Summary network Sp(-) Tocuses on learning the summary statistics h by minimizing OT, (p.. The pushforward aims to push the combination of random noise vector and statistics h2 generate samples that resemble the ones from P.. where we simply adopt concatenation for h, and z although other choices are also available. Besides, fa tries to distinguish the "fake" samples from the "real" samples in set Dj. Therefore, we optimize the implicit meta generative model by defining the objective function as:
1 J
min max (BB 0,0, w) = LPOT + J Σ En-r, [log f (x)] + B,0.0
(9)
In addition to the standard GAN loss, we can also adopt the Wasserstein GAN (WGAN) of Arjovsky al. (2017) to approximate Pj- also flexibly to decide the input fed into fa- For example, following the conditional GAN (CGAN) of Mirza & Osindero (2014),
combine hy and data points (generated or real), where the critic can be denoted as Jw [z. h2) and (Te h2), h2), respectively. Since we Focus on fitting the meta-distribution with the help of the summary network and POT loss, we leave the problem-specific design of the generator, critic, and summary network as future work for considerable flexibility in architectures. In Appendix A provide the illustration of our proposed model in Fig. and detailed algorithm in Algorithm
4 RELATED WORK
Learning Summary Representation of Set-input. There are two lines for learning the set rep- resentation. The first line aims to design more powerful summary networks, which reviewed in
5

---

Published as a conference paper at ICLR 2022
Introduction and Section 2.1 and omitted here due to the limited space. The another line assumes a/some to-be-leamed reference set(s), and optimizes the distance between the original sets (or fea- tures of the observed data points) and the reference set(s) with OT other distance measures, to learn set representation. For example, RepSet (Skianis et al. 2020) computes some comparison costs between the input sets and some to-be-learned reference sets with network flow algorithm, such as bipartite matching. These costs are then used as set representation in a subsequent neural network. However, unlike our framework, RepSet does not allow unsupervised learning and mainly focuses on classification tasks. The Optimal Transport Kernel Embedding (OTKE) (Mialon et al., 202T] marries ideas from OT and kernel methods (Schölkopf et al., 2002), and aligns features of a given set to trainable reference distribution. Wasserstein Embedding for Graph Learning (WEGL) (Kolouri et al., 2021) also uses a similar idea to the linear Wasserstein embedding as a pooling oper- ator for learning from sets of features. To the best of our knowledge, both of them view the reference distribution as the barycenter and compute the set-specific representation by aggregating the features (embedded with kernel methods) in given set with adaptive weight, defined by the transport plan between the given set and the reference. Different from them, we assume J probability distribu- tions (rather than one reference distribution with uniform measure) over these shared prototypes by taking set representations h as the measures, to approximate the corresponding J empirical distributions, respectively. Then we naturally use the summary network as the encoder to compute hi., which can be jointly optimized with the shared prototypes by minimizing the POT loss in an unsupervised way. For a given set, we can directly compute its representation with summary network, avoiding iteratively optimizing the transport plan between the given set and learned refer- ence like OTKE and WEGL. These differences between the barycenter problem and ours, which are further described in Appendix [B] lead to different views of set representation learning and different frameworks as well. To learn compact representations for sequential data, Cherian & Aeron (2020) blend contrastive learning, adversarial learning, OT, and Riemannian geometry into one framework. However, our work directly minimises the POT cost between empirical distribution and the to-be- learned distribution, providing a laconic but effective way to learn set representation.
Metric-based few-shot classification methods. Our method has a close connection with metric- based few-shot classification algorithms. For example, MatchingNet (Vinyals et al. 2016) and Pro- toNet (Snell et al. 2017) learned to classify samples by computing distances to representatives of each class. Using attention mechanism over learned embedding of the support set to predict classes for the query set, MatchingNet (Vinyals et al. 2016) can be viewed as a weighted nearest- neighbor classifier applied within an embedding space. ProtoNet (Snell et al., 2017) takes class's prototype to be the mean of its support set in the learned embedding space, which further performs classification for an embedded query point by finding the nearest class prototype. Importantly, the global prototypes in our paper are shared among all sets, which is different from the specific proto- type for each class in ProtoNet but suitable for our case. Due to the flexibility of our method, we can project the average aggregated feature vector derived from the encoder in ProtoNet or MatchingNet, into h, by introducing a simple neural network, which can be jointly optimized with the encoder by minimizing the classification loss and POT loss. Our novelty is that the POT loss can be naturally used to improve the learning of encoder while largely maintaining existing model architectures or algorithms. Another recent work for learning multiple centers is infinite mixture prototypes (IMP) (Allen et al. (201)). which represents each class by a set of clusters and infers the number of clus- ters with Bayesian nonparametries. However, in work, the centers set-specific feature extracted from the summary network serves as the proportion of centers, where the centers and summary network can be jointly learned with the POT loss.
shared for all classes and
Meta GAN-based Models. As discussed by Hong et al. (2020a), meta GAN-based models can be roughly divided into optimization sitimizationnbasedd
and transformation- based methods. Clouâtre & Demers (2019) and Liang et al. (2020) integrated GANs with meta- meta-learming algorithms to real- ize the optimization- based methods, including aggostc meaa-nen (MAML) (Finn al., 2017) and Reptile (Nichol et al.) 2018). Hong et al.) (2020b c) fused multiple conditional images by combining matching procedure with GANs, providing fusion-based methods. For transformation based methods, Antomiou AAttooos,,nttoiouu et al. (2017) and Hong al. (2020a) combine only one image and the ran- dom noise into the generator to produce a slightly different image from the same category, without using the multiple images from same category. Besides, a recent work that connects existing sum- mary network with GAN is MetaGAN (Zhang et al. 2018), which feeds the output of the summary network into the generator and focuses on few-shot classification using MAML. The key differences

---

Published as a conference paper at ICLR 2022
of these models from ours is that develop POT to capture each sets' summary statistics, where we can flexibly choose the summary network, generator, and discriminator for specific tasks.
5 EXPERIMENTS
We conduct extensive experiments to evaluate the performance of our proposed POT in improving summary networks, few-shot generation, and few-shot classification. Unless specified otherwise, we set the weight of entropic constraint as e =00.1, , the maximum iteration number in Sinkhorn algorithm as 200, and adopt the Adam optimizer (Kingma Ba 2015) with learning rate 0.001. We repeat all experiments 5 times and report the mean and standard deviation on corresponding test datasets.
5.1 EXPERIMENTS ABOUT POT LOSS IN SUMMARY NETWORK
To evaluate the effectiveness of POT in improving the summary network, we conduct three tasks on two classical architectures: DeepSets (Zaheer et al. (017)) and Set Transformer (Lee et al. 2019), where the former uses standard feed-forward neural networks and the latter adopts the attention-based network architecture. For Set Transformer and DeepSets, the summary network is defined in Equation | and optimized by the task-specific loss; for Set Transformer(+POT) and DeepSets(+PPTT) the summary network, defined as Equations aadd] 7 is optimized by both the POT loss and task-specific loss. More experimental details provided in Appendix C
Amortized Clustering with Mixture of Gaussians (MoGs): We consider the task of maximum likelihood of MoGs with C components, denoted as P(x; 0) = > a.N (z Pes diag (o?) Given the dataset X = {rim} generated from the MoG, the goal is to train a neural network, which takes X as input set and outputs parameters
{Te. Me. 0e31. Each dataset contains n E [100, 500) points on a 2D plane, each of which is sampled from one of C' Gaussians. Table reports the test average likelihood of different models with varying C'E (4, where we K K 50 prototypes for all C. We observe that Set Transformer outperforms DeepSets largely, validating the effectiveness of attention mechanisms in this task. We note both Set Transformer(+POT) and (e))S)ts(+POTT) improve their baselines, showing that the POT loss can encourage the summary networks to learn more efficient summary statistics.
Here, we evaluate our method on the task of point cloud classification using the ModelNet40 (Chang al., 2015) dataset which consists of 3D objects from 40 different categories. By treating each object as point cloud, we represent it as a set of N vectors in R (x; y; z-coordinates). Table [ reports the classification accuracy, where we perform the experiments with varying N E {64, 1024). set K = 40 prototypes. Clearly, both DeepSets and Set Transformer can be improved by adding the POT loss. Notably, fewer points would lead to lower performance, where the POT loss plays more important role. Taking this task as the example, we further study our model's 's sensitivity to hyper-parameter in Fig. of Appendix
Point Cloud Classification:
C.5
Following Zaheer et al. (0017), we aim to compute the sum of a given set of digits, where we consider MNIST8m (Loosli et al. 2007), consisting of 8 million instances of 28 x 28 gcay stamps of digits in {O.....9 9}. By ran- domly sampling subset of maximum M 10 images from MNIST8m, we build N = 100k "sets" of training, where we denote the sum of digits in that set as the set-label. We construct 100k sets of test MNIST digits, where we vary the M starting from 10 all the way up 100. The output of the summary net- work is a scalar, predicting the sum of M digits. In this case, proto-
Sum of Digits:
0.9
Accuracy
0.8
Deplen
Depleni-OO)
Iandirnee
20
40
60
B0
100
Number digits add
Accuracy of digit sum- mation with image inputs, where all models are trained on tasks of length 10 at most and tested on examples of length up to 100.
Figure 2:
we adopt L1 as the task-specific loss and set K types. We show the accuracy of digit summation for different algorithms Fig. and find that the POT loss can enhance the summary networks to achieve better generalization. In this task, we also explore the convergence rate and the learned transport plan matrix of Sinkhorn algorithm in Fig. of the Appendix C.6
5.2 EXPERIMENTS ON FEW-SHOT CLASSIFICATION
To explore whether our proposed method can improve the metric-based few-shot classification, consider two commonly-used algorithms as the baselines, including ProtoNet (Snell et al.) 2017)
We adopt the point-cloud dataset directly from the authors of 7aheer et al. 2017

---

Published as a conference paper at ICLR 2022
Table 1: Test performance of different methods, where left table denotes the likelihood for MoG with varying
C' (number of components), oracle is the likelihood of true parameters for the test data, and right denotes the test accuracy for point cloud classification task with varying N (number of points).
Task
Test likelihood for MoG
Test accuracy for the point cloud classification
Algorithm
C=4
5
C=6
C=7
C
64
N=
256
N1512
N=1024
Oracle
-1,473
-1.600
-1.200
-1.946
-2058
-1.809
-1812
-1.97
-2.115
-2.261
79.14
82.31
84.62
83.74
86.83
DeepSets
=0.015
=0001
=0017
+0016
20.014
+0.035
#0.028
=0.337
+0045
+0,042
-1.723
-1.743
-1.861
-2.078
-2.214
79.91
83.65
85.22
86.25
86,93
DrepSens(+POT)
+0.015
+0.017
=0.012
+0.018
+0.015
+0.050
+0.000
-0.055
+0.067
+0.075
-1.501
-1.721
-18859
-2.003
-2.106
79.01
82.31
84.46
85.82
86,34
Set Transformer
+0.006
+0.006
+0.007
+00.007
4-0.007
+0.103
+0.117
40.114
40.122
Set Transformer++POT)
-1.486
-1.676
-1.828
-1.967
-2.084
80.00
83.32
85.64
86.51
86.84
1. 0.007
1. 0.007
0.006
100.07
1.0.007
100.111
1.0.13
-0.121
10.124
10.1115
and MatchNet (Vinyals et al. 2016). Denoting the feature extractor in each algorithm as fØr we consider several popular backbones, including ResNet10 and ResNet34 (He et al., 2016). Recalling the discussions in Section 3.2 to enforce fe, to learn more powerful image features, we additionally introduce matrix B and net fog and learn the model by minimizing the POT loss and classification errors. We perform the experiments on the CUB (Welinder et al. (0TO) and minilmageNet (Ravi & Larochelle 2016). As a fine-grained few-shot classification benchmark, CUB contains 200 different classes of birds with total of 11, 788 images of size 84 x 84 x 3, where we split the dataset into 100 base classes, 50 validation classes, and 50 novel classes following Chen et al. (2019). minilmageNet is derived from ILSVRC-12 dataset (Kussakovsky et al., 2015). consisting of 84 x 84 x images from 100 classes with 600 random samples in each class. We follow the splits used in previous work (Ravi & Larochelle 2016), which splits the dataset into 64 base classes, 16 validation classes, and 20 novel classes. Table 2 reports the 5way5shot and Sway 10shot classification results of different methods on minilmageNet and CUB. We see that introducing the POT loss and summary network can consistently improve over baseline classifiers, and the performance gain gradually increases with the development of number of network layers. This suggests that our proposed plug-and-play framework can be flexibly used to enhance the metric-based few-shot classification, without the requirement of designing complicated models on purpose.
Table 2: 5way5shot and 5way 10shot classification accuracy (%) on CUB and minilmageNet, respectively, based on 1000 random trials. Here, (-) is the p-value computed with two-sample t-test, and p-value with blue (red) color means the increase (reduce) of performance when introducing POT loss. Datasets
CUB
minilmageNNtt
ProtoNet(resnet10)
84,32 - 0.51
87.41 + 0.49
72.7410.63
78.14 1 0.56
ProtoNet(+ (resnet10)
84.44 1:0.51 (1e-7)
87.69 :0.53 (1e:33)
72.94 00666 (1e-12)
78.76 0.49 (1e:146)
ProtoNe((resne(34)
87.33 ::: 0.48
91.75 0.47
73.99 0.64
78.64 0.56
ProtoNet(+OT) (resnet34)
88.3411 0.46 (0.0)
92.17 0.48 (16-79)
75.151
0.63(1e-265)
79.05 -: 0.52 (1c-60)
MatchNet (resnet]U)
82.98 0.56
85,974 0.53
68.82 = 0.65
72.06 =: 0.54
MatchNet(= (resnet10)
83.64 1: 0.58 (1o-127)
86.02.1 0.56 (0:04)
68.95 0.62. (c-6)
71.94 4: 0.56 (Ic-6)
MatchNet (resmet34)
84.66 0,55
86.32 + 0.56
68.32 + 0.66
72.41 +: 0.63
MatchNet(=OT) (resmet34)
85.50 : 0.66 (1e-172)
86.75
0.61 (1e:37)
68.51 + 0.64 (1e-11)
71.98 = 0.59 (1e-53)
5.3
EXPERIMENTS ON FEW-SHOT GENERATION
Here, we consider few-shot generation task to investigate the effectiveness of our proposed implicit meta generative framework, where we consider CGAN (Mirza & Osindero 2014] and DAGAN (Antoniou et al., (017) as baselines for their ability to generate conditional samples. For CGAN- based models, we adopt summary network as the encoder to extract the feature vector h; from a given set Dj. which is further fed into the generator and discriminator as the conditional information. Since the original DAGAN only assimilates one image into the generator, in our framework, we replace the encoder in DAGAN with summary network to learn the set representation. For DAGAN- based models, we adopt the same way with the original DAGAN to construct the real/samples for the critic and explain here for clarity: we sample two sets (Dij, D2j from the same distribution or category; then we represent the real samples as the combination of Dış and Daj and the fake ones
as the combination of Dij and Dij from the generator (conditioned on Dıj). Different from our framework that separately optimizes the summary network using the OT loss, all other models for comparison in this paper jointly optimize the encoder (c.g., summary network) with the generator by the generator loss. Besides, for a fair comparison, we also consider introducing the additional reconstruction loss (mean square error, MSE) to optimize the generator and encoder in baselines. We
8

---

Published as a conference paper at ICLR 2022
consider the DeepSets the summary network for its simple architecture. For 3D natural images, we adopt the pretrained densenet (landola et al. 2014) to extract features from each data pints, and take the features the input to summary network. To evaluate the quality of generated samples, we adopt commonly used metric Fréchet Inception Distance (FID) (Heusel et al. (017). where we only report the FID score (Heusel et al., 2017) considering the notable performance gap between our model and the compared ones. We provide several examples on toy datasets to show the efficiency of our proposed model in Appendix E
Table 3: FID J. of images generated by different methods with varying unseen angles A on MNIST.
Algorithms
Aw-160
Am-120
Am-80
Aw.40
A=
Aw
Aw80
A=
A=160
227.94
203.59
211.74
231.63
228.35
222,87
195.30
202.69
202.35
CGAN
2.5
-2.11
-188
2.11
+1.44
+ 2,02
+1.58
+1.25
+1,16
225.56
201.09
209.11
222.54
223.61
220.18
193.75
200.12
201.13
CGAN+MSE
2.0.5
1.62
-0991
2.13
+1.55
+:147
+1.39
+1.08
2.35
213,30
193.42
196.56
217.28
210.27
206.44
181.05
190.19
191.22
CGAN+POT
+ 1.35
= 1.25
1.61
= 1.58
= 2.04
+1.78
+ 0.99
11.2
1,337
169.85
201.58
157.95
204.23
218.42
196.70
160.35
198.24
164.64
DAGAN
1,56
=1.98
=1 1.45
+2.31
+1.96
1,75
+1.57
+2.
1.88
169.77
200.12
155.87
202.16
215.12
194.61
159.08
197.48
163.23
DAGAN+MSE
+1.51
-2.10
-1.11
89
+-197
+2210
+1.74
4-2.05
+18.44
159.51
174.07
135.89
177.64
197.09
186.17
142.40
174.75
146.69
DAGAN+ POT
1.57
1.63
-1.17
11888
1:446
1.990
1:333
1:155
1.1.47
artificially transform each image in MNIST dataset (LeCun 1998) with 18 rotations 180 to 180 by 20 degrees), leading to 18 distributions characterized by angle A. We choose 9 interleaved distributions for training and the rest unseen distributions for testing. We consider the CGAN-based and DAGAN-based models, respectively. During the test stage, for each unseen distribution, we randomly sample 1000 real images and gen- erate 20 fake images based on every 20 real images and repeat this process 50 times (i.e., 1000 /0),, resulting 1000 generated samples for each method. We summarize the test performance in Table with varying A. We can find that our proposed framework allows for better generalization to related but unseen distributions at test time, indicating the POT loss can enforce the summary network to capture more salient characteristics.
Rotated MNIST: Following Wu et al. (2020),
We further consider few-shot image generation on Flowers (Nilshack & Zisser- man 2008) and Animal Faces (Deng et al.] (209)), where we follow seen/unseen split provided in Liu et al. (2019). Flowers dataset contains 8189 images of 102 categories, which are divided into 85 training seen and 17 testing unseen categories; Animal Faces dataset contains 117, 574 animal faces collected from 149 carnivorous animal categories, which are split into 119 training seen and 30 test- ing unseen categories. We present the example images generated by DAGAN and DAGAN(+POT) and network architectures in Appendix [F for the limited space, where we also compute the FID) scores with the similar way in Rotated MNIST experiment. We find that method achieves the lowest FID and has the ability to generate more realistic natural images compared with baselines. This indicates the summary network in our proposed framework can successfully capture the impor- tant summary statistics within the set, beneficial for the few-shot image generation.
Natural Images:
6 CONCLUSION
In this paper, present novel method to improve existing summary networks designed for set- structured input based on optimal transport, where a set is endowed with two distributions: one is the empirical distribution over the data points, and another is the distribution over the learnable global prototypes. Moreover, we use the summary network to encode input set as the prototype proportion (i.e., set representation) for global centers in corresponding set. To learn the distribution over global prototypes and summary network, we minimize the proritepe OT loss between two distributions in terms of the defined cost function. Only additionally introducing the acceptable parameters, our proposed model provides a natural and unsupervised way to improve the summary network. In addition to the set-input problems, our plug-and-play framework has shown appealing properties that can be applied to many meta-learning tasks, where we consider the cases of metric- based few-shot classification and implicit meta generative modeling. Extensive experiments have been conducted, showing that our proposed framework achieves state-of-the-art performance on both improving existing summary networks and meta-learning models for set-input problems. Due to the flexibility and simplicity of proposed framework, there are still some exciting extensions. For example, an interesting future work would be to apply our method into approximate Bayesian computation for posterior inference.

---

Published as a conference paper at ICLR 2022
REFERENCES
Michal Aharon, Michael Elad, and Alfred Bruckstein. K-SVD: An algorithm for designing over- complete dictionaries for sparse representation. IEEE Transactions on signal processing. 54(11): 4311-4322, 2006.
Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In International Conference on Machine Learning, pp- 232-241. PMLR, 2019.
Jason M. Altschuler and Enric Boix-Adserà. Wasserstein barycenters can be computed in polyno- mial time in fixed dimension. J. Mach. Learn. Res., 22:44:1 2021.
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint 1.11:01711.04555 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
David M Blei, Andrew Y Ng. and Michael Jordan. Latent Dirichlet allocation. Journal of machine Learning research, 30an):993-1022, 2003.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Yanzhi Chen, Dinghuai Zhang, Michael U. Gutmann, Aaron C. Courville, and Zhanxing Zhu. Neu- ral approximate sufficient statistics for implicit models. In 9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Anoop Cherian and Shuchin Aeron. Representation learning via wwversarially-conrraaa optimal transport. In International Conference on Machine Learning, volume 119, pp. 1820-1830, 2020.
Louis Clouâtre and Marc Demers. Figr: Few-shot image generation with reptile. arXiv preprint
19Xivv1901.02.0 2019.
Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. Advances in neural information processing systems, 26:2292-2003 2300, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEE Computer Society, 2009.
Harrison Edwards and Amos Storkey. Towards a neural statistician. 5th International Confer- ence on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems, 29:3225-22222 3233, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pp. 1126-1135. PMLR, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. volume 27, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
10

---

Published as a conference paper at ICLR 2022
Yan Hong, Li Niu, Jianfu Zhang, Jing Liang, and Liqing Zhang, Deltagan: Towards diverse few-shot image generation with sample-specifie delta. arXiv preprint arXiv:2009.08753, 2020a.
Yan Hong,
Li Niu, Jianfu Zhang, and Liqing Zhang. Matchinggan: Matching- braching--bsedd few-shot image generation. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6.
IEEE, 2020b.
Yan Hong, Li Niu, Jianfu Zhang, Weijie Zhao, Chen Fu, and Liqing Zhang. F2gan: Fusing-and- filling gan for few-shot image generation. In Proceedings of the 28th ACM International Confer- ence on Multimedia, pp. 2535-2543, 2020c.
Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kilian Q Weinberger. Condensenet: An efficient densenet using learned group convolutions. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 2752-2761, 2018.
Forrest landola, Matt Moskewicz, Sergey Karayev, Ross Girshick, Trevor Darrell, and Kurt Keutzer. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint arXiv:1404. 1869, 2014.
Mateusz Jurewiez and Leon Strömbeerg-errzzynski Set-to-sequence methods in machine learning: review. arXiv preprint arXiv:2103.09650 2021.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In 3rd Interna- tional Conference on Learning Representations, 2015.
Soheil Kolouri, Navid Naderializadeh, Gustavo K. Rohde, and Heiko Hoffmann. Wasserstein em- bedding for graph learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmist// 1998.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: framework for attention-based permutation-invarrt neural networks. In Interna- tional Conference on Machine Learning, pp. 3744-3753, PMLR, 2019.
Weixin Liang, Zixuan Liu, and Can Liu. Dawson: A domain adaptive few shot generation frame- work. arXiv preprint arXiv:2001. 1,0576, 2020.
Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In 2019 IEEF/CVE International Conference Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 10550-10559, IEEE, 2019.
Gaëlle Loosli, Stéphane Canu, and Léon Bottou. Training invariant support vector machines using selective sampling. Large scale kernel machines, 2, 2007.
Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning sets of symmetric elements. In International Conference on Machine Learning, pp. 6734-6744, PMLR, 2020.
Grégoire Mialon, Dexiong Chen, Alexandre d' Aspremont, and Julien Mairal. A trainable optimal transport embedding for feature aggregation and its relationship to attention. In ICLR 2021 -he Ninth International Conference on Learning Representations, 2021.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. CoRR, abss/411..176 2014.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint 1803v 2018.
Maria-Eleen Eleiaa--eaaaa Nilsback and Andrew Zisserman. Automated flower classification over large num- ber of classes. In Sixth Indian Conference Computer Vision, Graphics & Image Processing, ICVGIP 2008, Bhubaneswar, India, 16-19 December 2008, pp. 722-729. IEEE Computer Soci- ety, 2008
11

---

Published as a conference paper at ICLR 2022
Gabriel Peyré and Marco Cuturi. Computational optimal transport. Found. Trends Mach. Learn., (5-6):355 607, 2019.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3)2111252 2015.
Bernhard Schölkopf, Alexander Smola, Francis Bach, et al. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002.
Konstantinos Skianis, Giannis Nikolentzos, Stratis Limnios, and Michalis Vazirgiannis. Rep the set: Neural networks for learning set representations. In International conference artificial intelligence and statistics, pp. 1410-1420 1420.. PMLR, 2020.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp.
2017.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.
Korawat Tanwisuth, Xinjie Fan, Huangjie Zheng, Shujian Zhang, Hao Zhang, Bo Chen, and Mingyuan Zhou. A prototype-oriented framework for unsupervised domain adaptation. Advances in Neural Information Processing Systems, 34, 2021.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3630-3638, 2016
Dongsheng Wang, Dandan Guo, He Zhao, Huangjie Zheng, Korawat Tanwisuth, Bo Chen, and Mingyuan Zhou. Representing mixtures of word embeddings with mixtures of topic embeddings. arXiv preprint arXiv:2203.01577 2022.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
Mike Wu, Kristy Choi, Noah Goodman, and Stefano Ermon. Meta-amortized variational inference and learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp- 6404-6412, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. Deep sets. In NeurIPS, pp. 3391-3401, 2017.
Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. Metagan: An adversarial approach to few-shot learning. NeurlPS, 2:8, 2018.
Huangjie Zheng and Mingyuan Zhou. Exploiting chain rule and Bayes' theorem to compare proba- bility distributions. Advances in Neural Information Processing Systems, 34, 2021.
Mingyuan Zhou, Haojun Chen, Lu Ren, Guillermo Sapiro, Lawrence Carin, and John Paisley. Non- parametric Bayesian dictionary learning for sparse image representations. Advances in neural information processing systems, 22, 2009.
Mingyuan Zhou, Yulai Cong, and Bo Chen. Augmentable gamma belief networks. The Journal of Machine Learning Research, 17(1):5656- 2016.
12

---

Published as a conference paper at ICLR 2022
N2 P3
K
Baya E R
Q
Σ
hg Ag Rd.
N
POT
P
Dj
Summary net
hi
Generator
So(1),1NN))
To (z; hj)
Real samples
Fake samples
Discriminator
f (x)
Figure 3: The overview of our proposed implicit meta generative framework, where we sample the j-th distribution from training sets, feed the data points into the summary network, generate the fake samples with the random noise and summary output as the input, discriminate the real/fake samples.
Algorithm The workflow of our proposed implicit meta generative framework.
Require: Datasets Di.,, initial discriminator parameters w, initial generator parameters 0, initial summary network parameters ø, initial matrix B, the cost function C, the number of critic itera- tions per generator iteration Meritic. the batch size m, learning rate a and the paper-parameter
while B.B e has not converged do Randomly choose from 1, 2, ..,
for t 1, ..................... .eritic do
Sample the real data set D, = { from j-th empirical distribution Pj: Embed the observed batch data into the statisties h, Softmax(S&(D,))
Sample a batch of prior samples { )m from p(z);
9w f- - Σm [log (x)) + log(1 fw (Ta h)))
<- + age
end for:
Represent the Q2 with global prototype matrix B and representation/statiiiss hj
Compute the loss OT, (Py. Q2) between P, and Qj with Sinkhorn algorithm in Equation
@B Vn [OT, (P;, Q;)];
B - B + ogg::
I+ V (P,
Φ + o + aga: see Algorithm ! for more details;
Sample batch of latent variables (1) ji=1 p(2);
m log (1 fn (To h,)))]:
90
0 - 0 + agei
end while
A ALGORITHM AND ILLUSTRATION OF OUR PROPOSED MODEL
The pseudo code for the implicit meta generative modeling is provided in Algorithm
13

---

Published as a conference paper at ICLR 2022
B THE DIFFERENCE BETWEEN OUR MODEL AND BARYCENTER PROBLEM
In this section, we clarify the difference between Wasserstein barycenter and our method. Specif- ically, for j-th distribution, we denoted P3 as its empirical distribution consisting of Nj samples, expressed as P, = Ei= 1 625,: Lijs E R Notably, aj = Ny. E Σ, represents the probability Ni measure for distribution Pj.
For another thing. we can represent Pj with another to-be-learned distribution Qj. defined as ===Ek hgx Bp E 12d. Here b, [hyz]] E > is the probability measure for distribution Qj. which can be computed using summary network (a and serves as the representation for set J. And 3R is the k-th prototype in the same space of the observed data points, which is the k-th column of B E Rxxd a learnable global prototype matrix. To optimize the f1.K and the summary network Sş for computing hj. we minimize the average OT loss (between Qj and Pj) for all training sets. We rewrite the Equation (6) here for convenience:
K ΣΣ
N, K
J
Lor min B
1
Σ
CATIAN
TwInT'
min B.
(OT, (P).
J
j=1
j=1
Usually, this equation can also be represented:
1 Σ (OT, (a,, (OT, B.
min
(10)
In terms of Wasserstein barycenter, we adopt the same notations for consistency. Following (Altschulrr & Boix- 2021), given J empirical distributions PI.J and their respective proba- bility measures |ai,..... a. supported on Rd and vector X E E', their corresponding Wasserstein barycenter can be viewed as another distribution Q, i.e., Q = Σx= m268, Br. E IRd where Br is the k-th column of B, m = [mx] E ΣK is the probability measure for distribution Q. Then we can learn the barycenter (i.e., B and m) by minimizing
3
J Σ j=1
min Bm
Σ j=1
(P, (P,
= =min- B,m J
W, m)
(II)
where above -(:-)) denotes the squared 2- Wasserstein distance. By comparing the equation 11 and equation 10 we can find that our model learns a Q2 to approximate P, for each distribution but "barycenter" problem learns shared Q the barycenter for all Pı.j- Therefore, after minimizing the average loss for all training sets with Equation (6). we can use the probability measure bj = directly map the test
[hya] (i.e., h2) to represent the empirical distribution Py. Especially, set to its representation by using the summary network. However, since the probability measure m in "barycenter" is shared by all distributions, it can not represent specific set. Therefore, to achieve the set representation, it might need to first compute the transport plan between test set and the barycenter Q and then aggregate the data points (or features) within the test set by taking the transport plan as the weight. Therefore, our model produces more intuitive solution to learn the set representation, which can take full advantage of the existing summary networks and provides a promising tool for addressing set-input and meta-learning problems.
C EXPERIMENTAL SETTINGS ABOUT INTRODUCING POT LOSS INTO THE SUMMARY NETWORKS
C.I DETAILS FOR AMORTIZED CLUSTERING WITH MIXTURES OF GAUSSIANS
We generate the 2D toy datasets following the Lee et al. 2019 where we additionally vary the C (the number of components) from to 8. Below, we present the detailed generation process about the toy datasets:
Specify the number of components C' for 2D toy dataset. 2. Generate the number of data points, n ~ Uniform [100, 500).
Sample the mean vector for components.
Hed Uniform( -4,4), c 1..... C, d= 1,2
14

---

Published as a conference paper at ICLR 2022
4. Sample the cluster labels.
π Dir (1)). 23 ~ Categorical(T)) ,,,4 1....,,
5. Generate data from spherical Gaussian.
I3 ~ N (H., (0.3) I....,
C.2 DETAILS ABOUT SET- WWWET-RRRSOOOORRRRRARR AND DREPSETS-BASED ARCHITECTURES USED MOGS EXPERIMENTS
DeepSets In terms of the DeepSets, the fe, in summary network contains 3 cermmutationequivvrian layers with 256 channels followed by mean-pooling over the set structure. Then the resulting vector representation 2j of the set is then fed to fully connected layer with 512 units followed by linear layer 512 x C(1 + * 2), where C denotes the number components. We use ELU activation at all layers. To introduce the POT loss into the DeepSets, further feed the 2j into a fully nected layer with 512 units followed by a 50-way softmax unit and also introduce the global matrix B E 2Kx50
Set Transformer To perform the MoGi experiments, we adopt the same architecture for Set Trans- former following [ee et al. (2019), whose parameters are reported in Table To introduce the POT loss, we also add the two fully connected layers with 256 units on the resulting vector 2j followed by a 50-way softmax unit, and a global prototype matrix B E 2
C.3 DETAILS ABOUT SET-TRANSFORMEE BASED AND DEEPSETS-BASEE BASED ARCHITECTURES USED IN SUM DIGITS
DeepSets Following the official code in Zaheer et al. 2017). we adopt the default architecture implement the DeepSets, where we first project the image into a 128-dimensional vector with three convolutional layers and apply summary network on the 128-dimensional vectors. To build DeepSets(+POT), we take the set representation 2, after sum-pooling in summary network as the input and introduce fully connected layer with 128 units followed by 10-way softmax unit, and global prototype matrix B E R128x10
Set Transformer For Set Transformer, we follow the similar structure used in MoGs experiments, where we also project the image with three convolutional layers and output a scalar. To build Set Transformer(+POT), we take the representation z, after sum-pooling in summary network as the input and introduce fully connected layer with 128 units followed by a 10-way softmax unit, and a center matrix B E R 12
C.4 DETAILS ABOUT SET-TRANTE TRANSFORMER-BASS AND DEEPSETS-BASEL ARCHITECTURES USED IN POINT CLOUD CLASSIFICATION
DeepSets For original DeepSets, we adopt the same architecture with Zaheer al. (2017). In specific, the for in summary network contains 3 followed by max- max-ppolingg over the set structure. Then the resulting vector representation z, of the set is then fed to fully connected layer with 256 units followed by 40-way softmax unit. We use Tanh activation at all layers and dropout on the layers after set-max-pooling (i.e., two dropout operations) with 50% dropout rate. To introduce the POT loss into the DeepSets, we further feed the zj into a fully connected layer with 256 units followed by a 40-way softmax unit, with 70% dropout rate. Besides, we additionally introduce the center matrix
layers with 256 channels
3x40
Set Transformer We also adopt the same architecture to implement the Set Transformer, where we summarize the parameters Table 5 following Lee et al. (2019). improve the Set Transformer with POT loss, we also introduce fully connected layer with 256 units followed by 40-way softmax unit, with 90% dropout rate, and a center matrix E 3x
C.5 PARAMETER SENSITIVITY
In the previous experiments, fix the value of 0.1, controlling the weight of the entropic regularisation in the Sinkhorn algorithm. Notably, unless specified otherwise. we specify the con- struction of C as Ci.. = 1 cos (2 (0,,,,. Bx)) Therefore, the cost function provides upper-bounded
15

---

Published as a conference paper at ICLR 2022
Table 4: Detailed architectures of Set Transformer used in the MoGs experiments, cited from Lee al.
2019
where denotes the number of components.
Encoder
Decoder
FFF FC(128, ReLU)
SAB
ISAB
Pooling
PMA
SAB(128, 4)
ISABm( 28, 4)
mean
PMA4(128, 4)
FC(12, ReLU)
SAB(128, 4)
1S28 4)
FC(12, ReLU)
SAB(128, 4)
FCT128, ReL.)
FCT128, ReL.)
FC(128, ReLU)
FCT128, ReLU)
21+2-(++222))
FC (1 (1+2-2)))
Table 5: Detailed architectures of Set Transformer used in the point cloud classification experiments, cited
from Lee et al. (2019).
Encoder
Decoder
FFF
ISAB
Pooling
PMA
FC(256, ReLU)
(266 4)
max
Or05out(O.5)
FC(256, ReLU)
12A6 4)
(1)))
PMAI(256. 4)
FC(256, ReLU)
FCT256, ReLU)
Dropout(0.5)
FC(256.) )
D10)
FC(40,)
FC(40.)
70
DeepSets(+POT) DeepSets
Accuracy (%)
64
62
60
0.01
0.050.10
0.501.00
10.00
Hyper-parameter in Sinkhorn algorithm
Figure 4: Parameter sensitivity of DP)) on point cloud classification task, with varying e, where cach object is represented set of N 20 vectors.
positive similarity metric, making the e has the corresponding reasonable range prior knowledge. e. We consider the point cloud classification task and each object is represented as a set of N = 20 vectors. As shown in Fig. 4 we report the performance of Dee)Sets(+POT) on point cloud classification task with varying e, where DeepSets serves as the baseline. It can be seen that our model is robust to the Besides, all the results of (O))) with different e are superior than that of DeeepSets, indicating the effectiveness of our method. By fine-tuning for each dataset in each task, might obtain better results than those reported in our experiments. However, we aim to validate our method instead of exhaustively tuning this hyper-parameter and thus set 0.1, which can achieve the acceptable result.
Here, we study our DeepSets(+POT)'s sensitivity
C.6 CONVERGENCE RATE OF SINKHORN ALGORITHM
In this paper, we the maximum iteration number as Itermax 200 in Sinkhorn algorithm for all experiments. As shown in Fig. 5) we visualize the convergence rate of Sinkhorn algorithm, where we consider the task about "sum of digits" (Oee The upper figure shows the convergence rate of Sinkhorn algorithm. The bottom figure visualizes the transport plan matrix with varying iterations. We find that the 200 iterations are typically enough for Sinkhorn algorithm and we can learn sparse transport plan matrix T when the algorithm converge. Notably, the transport plan matrix needs to satisfy two marginal constraints, defined by the probability measures of two distributions, respectively. Recall that the empirical distribution has unchanged uniform probability measure, so the learned transport plan matrix is dense for the N, observed samples. In terms of another distribution, its probability measure is the set-specific representation, weighting the
16

---

Published as a conference paper at ICLR 2022
Itern
Figure 5: Top: the convergence rate of Sinkhorn for c(z, y) = cosine(r, y). and = 0.1, as measured in term of marginal constraint violation >>>>
[ug 6+1 - ulyl where is the iteration index and u is the scaling 2019) for more details. Bottom: evolution the transport plan computed at iteration of Sinkhorn's 's iterations.
Ny
variable; please see page 67 Peyré
Cuturi
matrix T = diag u (4) K diag
(<)
importance of K shared centers for corresponding set. Therefore, it is reasonable that transport plan matrix is sparse for K centers.
D EXPERIMENTAL SETTINGS ABOUT FEW-SHOT CLASSIFICATION
Denote the prototype for set j (computed by fo, in few-shot classification Cj. We consider two backbones for fez. including ResNet10 and ResNet34, which produce the 512-dimensional c,. To improve the metric-based few-shot classification with our framework, taking the e input, further construct the 96.. Specifically, we introduce fully connected network with architecture as 512 > 256 units with ReLU function followed by a X-way (X=64 for CUB, and X=128 for minilmageNet) Softmax function and center matrix B E IR5 5 We conduct 10000 tasks of the training set DDr train the model while 1000 tasks of the test set Die to evaluate the learned model. And D,, 0 Die G. We run 60 epochs to train the model on CUB and minilmageNet. The model trained using Adam optimizer with default settings (learning rate 1e - 3, B = (0.9, 0.999), and le 8) on one Nvidia Geforce RTX3090 GPU.
E ADDITIONAL EXPERIMENTAL RESULTS ON FEW-SHOT GENERATION ABOUT TOY DATASETS
We test our algorithm through a series of synthetic data sets and realistic data sets. For synthetic datasets, we set To fw and S as fully connected neural networks, where To fo have 4 hidden layers and fo, and 96. (we adopt DeepSets) have 3 hidden layers. Each layer has 200 nodes, and the activation function is chosen as RELU, where we adopt the softmax in the final layer.
Normal distribution on 2D toy data: We first consider the 2D normal case, where the training data contains 10K sets and each set contains 100 data points from N(u, Σ). We sample the mean, variance, and covariance from U[-5, 5]. (1,1 2). and U|-0.5, 0.5], respectively. Fig. 6 shows the real (gray points) and generated samples (red points) by different models given unseen test sets, where we only consider CGAN- -bseed methods for the simple toy data. We find that our model (third column) can improve the resistance to mode collapse compared with CGAN+MSE (second column) and better the unseen test distributions than CGAN (first column). This result indicates the POT loss spur the summary network to capture desired statistics for unseen distributions.
17

---

Published as a conference paper at ICLR 2022
Figure 6: Examples of few-shot generation for two dimensional Gaussian distributions, where we visualize the real samples from the true distributions (gray points) and generated samples (red points) by different models (from first third column: CGAN, CGAN+MSE and proposed CGAN+POT), where also plot the contour each Gaussian distribution.
In this case, generate another collection of synthetic 1- D datasets based on Gaussian parametric family, where the means and variances are sampled from 1] and U[0.5, respectively. The training data contains 10K sets each containing 50 samples. We also visualize the pdfs of gaussian distributions with randomly means and variance Fig 7 which are used sample test data, and show the 500 data points generated by the push- forward. For this experiment, we set the dimension of and summary vector as 2.
One-dimensional Gaussian distributions:
Multi-family distribution on 1D toy data: To validate if our proposed model can capture many types of distributional families simultaneously, we construct collection of synthetic 1-D datasets each containing 100 samples from either an Exponential, Gaussian or Laplacian distribution with equal probability. For Gaussian and Laplacian distributions, means and variances are sampled from U[-1.1] 1] and (00.5, 2] respectively; for Exponential distributions, rates are sampled from (00..5 5, 2]. Fig. visualizes the pdfss of six one-dimensional test distributions with different means and variances and the generated data points. It is interesting to observe that the generated data points can fit the corresponding pdf well, indicating
model generalize to different distributions with varying parameters. Besides, our model performs slightly worse on the Exponential distributions, perhaps attributing to the fact that it the only non-symmetric distribution.
F DETAILS ABOUT NATURAL IMAGE GENERATIONS AND THE RESULTS
We use NenseNet proposed by Huang al. (2018) the backbone of summary network, then a pooling operation is conducted as Zaheer et al. (017) does. And a 2-layer fully connected network with ReLU activation function finally employed to embed the 4096-D visual features into the corresponding 512-D set representations hy. As for conditional generator (conditioned on set representations
well Gaussian noise), we introduce 2-layer embedding network [100->600-100 100] with LeakyReLU activation function to embed the input noise n. Besides, we use a 5-layer deconvolution network [Conv Trooppsaaspos + 100, 512, 4, 1, 0) ConvTrans- p5se2d(512, 256, 4, 2, I) Transpose2d(512. 128, 4, 2, Transpose2d 3. 4, 2. 1)], where = 64, 128 for oxford and animal face datasets respectively with BatchNorm along channels and ReLU activation function to deconvolute the con- catenated noise embeddings and set representations cat(In. ([n, h,]) as fake output images 2 fake : x33644664 Finally, 5-layer discriminator network [Conv2d(2*3, 64, 4, 2, 1) 128, 4, 2, D))-Conv2((128, 256, 4, 2, 1)==Con2d(256, 512, 4, 2, 1) 1)= 1, 4, 1, 0)] with BatchNorm as well as LeakyReLU activation function at the first fourth deconvolutional layers and the last layer without BatchNorm while with sigmoid activation funtion to distinguish the true or fake generated images. We present the generated results in Figure
64, 4,
2.
18

---

Published as a conference paper at ICLR 2022
Cenerated Sumples vs For of Gaussian Disributio
Generated Samples FDF Gaundin Detribution
Generated Samples vs PDF of Gaussion Darribution
0.7
0.6
0.5
0.4
0.3-
0.3
0.2
0.2
0.1-
0.1
0.1
-2
-4
Generated Sangle es vs PDF Gauddian Dkariibttio
Gomerated Samples PDP of Gausslan Diatibutionn
Generated Samples vs PDF Gaustion Distr Distribtion
0.5
0.40
0.4
0.30
0.3
0.25
0.2
0.15
0.1
0.1
0.05
-4
-2
0
-4
-2
2
-4
-2
0
Figure 7: Examples of few-shot generation for one dimensional Gaussian distributions, where we visualize the generated samples by our GAN+POT (conditioned on the test samples from the unseen distribution) and the PDF of the unseen true distribution.
Samples Laplacian Oktribation
Genernaa PDR Coussin
Generated
0.5
0.30
0.30
0.4
0.25
0.25
0.20
0.15
0.15
0.2
0.30
0.10
-6
-4
-8
-6
-4
-2
-12.5-10.0-75
-30
2.5
5.0
Savples FDT Expanential Distibution
Generated Samples as PDF of Exponential Distrbution
Sarpples Gaussion
0.7
10
1.2
1.0
0.8
0.8
0.6
0.4
0.4
0.2
-2
0
2
4
8
8
10
0.0
4
10
0.0
-6
-2
Figure 8: Examples of few-shot generation for multi-distribution where we visualize the generated samples (green) by our GAN+POT (conditioned on the test samples from the unseen distribution) and the PDF (red) of the unseen true distribution.
19

---

Published as a conference paper at ICLR 2022
Roal images Animal Face
(b) images generated by DAGAN
(c) Fake images generated by DAGAN-OT Above: Animal Face Below: Oxford flower
Above:
Above: Animal Face
Below: Oxford flower
Below: Oxford flower
Figure 9: Examples of few-shot generation for natural images, where the images generated by DAGAN (second column) and DAGAN+POT (third column) conditioned on different categories on Oxford(flower) and animal face datasets. The FID scores of DAGAN and DAGAN+POT Oxford are 97.25 and 91.78, respectively. The FIDs of DAGAN and DAGAN+POT on animal face are 139. 14 and 131.51.
20


Reviews:
[{"review_id": "uwxlGm-O7XY", "reviewer": "Reviewer_AAZ1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1) This paper provides a promising research direction in using set-structured data on various downstream tasks. Especially, the paradigm using well-learnt set-representations as an conditional input deserves more explorations to tackle the limitation of few-shot generation problems.\n\n2) The proposed algorithm is well-motivated, and is effective to be implemented for real-world applications. In particular, it is easy to combine the summary network with the majority of existing metric-based meta-learning models and generative models.\n\n3) The experimental study is extensively conducted on various tasks related to set-structured data, and the improvement is consistent over different tasks. This shows the effectiveness of the proposed plug-and-play framework.\n\nWeakness\n1) The main idea of this paper that uses optimal transport in learning representation, has been widely studied such as https://openreview.net/pdf?id=ZK6vTvb84s, https://arxiv.org/pdf/2007.05840.pdf. It would be better to clarify what the contribution of this paper is and what the major difference is in contrast to the existing literature of optimal transport for representation learning.", "summary_of_the_paper": "This paper proposed an improved training of summary network using optimal transport based auxiliary loss. The summary network with negligibly increased parameters show much better performance on amortized clustering, point-cloud classification, sum of digits, few-shot classification, and few-shot generation tasks.", "summary_of_the_review": "This paper is well-motivated and definitely effective for real-world meta-learning applications, however, the authors should discuss the literature of representation learning using optimal transport.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eNtPO2u9y3N", "reviewer": "Reviewer_tuTq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern is on the novelty of this work. Indeed, it seems to me that some ideas presented here can be found in previous papers on set representation learning that are not mentioned. In [1], prototypes against which OT is computed are learned (potentially jointly with an element-wise embedding) with or without supervision, although the outputs used in practice seem to differ. [2] presents a similar method in the context of graph representation. It is therefore difficult to me to understand the real contributions of this paper: it would be great if the authors could comment on the similarities and differences between their method and these ones.\n\nPros:\n- The paper is well-motivated and mostly clear.\n- Experiments are varied and seems sound.\n- In particular, the proposed framework seems to improve existing methods for set representations (DeepSets and Set Transformer).\n\nCons:\n- I have concerns on the novelty of this work.\n\nQuestions and remarks:\n- Equation (6) seems to be an instance of Wasserstein barycenter: could you comment on this?\n- It could be worth to look at this other paper on set representation in the related work [3] which has a different approach to the current and previous work yet also similar ideas.\n- Could the authors elaborate on the computational complexity of computing the transport plans? How many Sinkhorn iterations do you use?\n- How sensitive are you results with respect to the entropic regularization parameter $\\epsilon$ ?\n\n------------------\n\n[1]  A Trainable Optimal Transport Embedding for Feature Aggregation and its Relationship to Attention (Grégoire Mialon, Dexiong Chen, Alexandre d'Aspremont, Julien Mairal)\n\n[2] Wasserstein Embedding for Graph Learning (Soheil Kolouri, Navid Naderializadeh, Gustavo K. Rohde, Heiko Hoffmann)\n\n[3] Rep the Set: Neural Networks for Learning Set Representations (Konstantinos Skianis, Giannis Nikolentzos, Stratis Limnios, Michalis Vazirgiannis)", "summary_of_the_paper": " This work proposes a method to improve set representation, and applies it in the context of meta-learning. More precisely, the method consists in jointly learning a summary network and prototypes using an optimal transport loss: the prototypes and summary network output should minimize the sum of the optimal transport costs for all the sets of the meta dataset. Experiments are conducted in the context of few-shot learning and other tasks used to evaluate summary networks.", "summary_of_the_review": "The paper seems sound but I have concerns on the novelty: I am willing to raise my score if the authors clarify their contribution w.r.t [1] and [2].", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dZ06YICChP", "reviewer": "Reviewer_Nrfc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThe intuition of the work is clear and the approach is reasonable a priori. This makes it easy to follow the motivations and illustrations, and lowers the barrier to the community adopting the presented techniques.\n\nThe proposed method is simple and does not require complicated architectural or training adaptations to improve the models to which it is applied.\n\nThe experiments are extensive, varied and compelling.\n\nThe related work section is mostly good and does a good job of comparing the contained approach with existing work (although see the minor comments.)\n\n**Weaknesses**\n\nThe main weakness is in the presentation of the manuscript. I think these problem can be addressed by the authors within the rebuttal period and my score is based on the expectation that this happens; if the authors do not improve the technical aspects of the presentation I will lower my score.\n\n1. (Technical) All the tables should include errors. Currently the tables 1 and 3 do not include errors. The table captions should also include details of the number of repetitions used to calculate the mean and the error that is being reported (standard deviation or standard error on the mean). An indication should be made when results are within error.\n\n2. A meta-statistic would be useful for Table 2. In many cases the original model and the +OT model performances are within error but overall 14 of the 16 head-to-head comparisons results in the OT model having the higher mean. It is unlikely that the addition of the OT loss does not improve performance given such a high number of head-to-head (e.g. in a simple binomial model with $p=0.5$ and 16 trials, the probability of 14 or greater successes is <0.03).\n\n3. (Non-technical) Figure 2 would be improved by using a better colour scheme and different markers for each model to make it easier to distinguish the different models in black and white or for a colour blind person. This is probably an access issue which is why I have included it in the weaknesses rather than the minor comments, and it is in the authors' interest to improve the presentation of their results in any case.\n\n4. (Non-technical) The last sentence of page 2 reads _‘Despite their effectiveness, there is no clear evidence that the output of these summary networks could describe a set’s summary statistics well, which have been proven crucial for the set-structured inference problems (Chen et al., 2021)’_. If X is necessary for Y, and Z has Y, then Z has X. If describing a set’s summary statistics well is crucial for set-structured inference problems, and summary networks are effective at set-structured inference problems, then the output of summary networks describe a set’s summary statistics well. Either effectiveness is clear evidence or describing a set’s summary statistics well is not crucial for these problems. This sentence should be revised.\n\n5. There is no supplementary code provided which means I cannot verify the experimental claims, this has resulted in a lower confidence score in my review.\n\n**Minor comments and suggestions**\n\nThe following comments are minor and potentially subjective so feel free to ignore them. My score does not depend on these comments being acted on.\n- Infinite Mixture Prototypes for Few-Shot Learning Allen et al ICML 2019 seems like a relevant reference that could be discussed in the related work section on developments to metric based approaches. The approaches differ in fundamental ways, and may be complimentary, but both have the concept of learning multiple centres and it may help to position this work to discuss the ideas presented here in contrast with those presented by Allen et al.\n- Algorithm 1 is in the Appendix and that should be stated in the text when it is referenced (e.g. page 4)\n- Algorithm 1 should be included in the main text.\n- 'as good as possible' on page 5 should be 'as well as possible'\n- The related work section has many sentences that should be revised:\n  - 'Different from them that focus on...'\n  - 'where most related work to ours is...'\n  - Different from these studies usually compute class' prototypes...'\n- Figure 3 is not particularly compelling and takes up a lot of space that could be better used by presenting Algorithm 1.", "summary_of_the_paper": "This work introduces a straightforward development for set representation learning in the meta-learning context based on the intuition that the sets encountered in real-world meta-learning tasks tend to have common attributes, as illustrated in Figure 1. The idea is to jointly learn these common attributes $\\beta_{[1:K]}$, referred to as 'global centres' or 'global prototypes', and the parameters of a summary network using an Optimal Transport derived loss function that compares the empirical set distribution $P_j$ with a set summary distribution over the global centres/prototypes, $Q_j=\\sum_{k=1}^K h_{jk}\\delta_{\\beta_k}$. The idea is elegant in its simplicity and effectiveness.", "summary_of_the_review": "In summary, I think this is a well motivated and elegant proposal that has been shown to be effective in a variety of experimental settings. I found the authors' claims to be well supported and I cannot find technical fault with their work that would support rejection, so I will recommend acceptance. The lack of supplementary code reduces my confidence in the review as I can only assess the claims as they are presented. The main issue with the work is its presentation which may be easily improved within the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6bJ1FWoZIB9", "reviewer": "Reviewer_TfNJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "‌‌ **‌Originality**\n\nThe paper is original and proposes an Optimal Transport (OT) based algorithm for improving existing summary networks for learning from set-structured data.\n\n**Quality**\n\nThe paper is technically sound, however, the novelty is a little bit limited. The proposed approach doesn't improve upon the pre-existing summary networks architecture.\n\n**Clarity**\n\nThe paper is clear and easy to follow. Algorithm 1 should be moved to the main body rather than the appendix.\n\n**Significance**\n\nThe work is significant, but novelty is a bit limited.\n\n**Limitations**\n\n- The paper lacks theoretical analysis for the proposed approach.\n- Error bars are missing and not reported in the results.\n\n\n**Questions to Authors**", "summary_of_the_paper": "The paper provides an optimal transport (OT) based algorithm for improving existing summary networks for learning from set-structured data. The proposed approach views each set as a distribution over a set of global prototypes. To learn the distribution over the global prototypes, the proposed approach minimizes the OT distance to the set's empirical distribution over data points. Empirical results demonstrate that the proposed framework improves upon the existing summary network approaches as well as metric-based few-shot classification and generative modeling applications.\n", "summary_of_the_review": "‌‌ The paper provides an optimal transport (OT) based algorithm for improving existing summary networks for learning from set-structured data. The paper is original, and  technically sound, however, the novelty is a little bit limited. The proposed approach doesn't improve beyond the pre-existing summary networks architecture. Empirical results demonstrate that the proposed framework improves upon the existing summary network approaches as well as metric-based few-shot classification and generative modeling applications. The paper lacks theoretical analysis for the proposed approach, and the error bars are not reported for the empirical results.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The proposed method for set representation learning with an application to mete learning is well-motivated and reasonable. Reviewers' original concerns about novelty and technical presentation have been well explained and addressed in the revision. If some theoretical analysis can be provided regarding the proposed method, it would make this work stronger.

In summary, a positive recommendation is given here.
Decision: Accept (Poster)