Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding
Authors: Bruno Andreis
Jeffrey Ryan Willette
Juho Lee
Sung Ju Hwang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Mini-Batch Consistency
Set Encoding
Point Clouds

Abstract: Most existing set encoding algorithms operate under the implicit assumption that all the set elements are accessible, and that there are ample computational and memory resources to load the set into memory during training and inference.  However, both assumptions fail when the set is excessively large such that it is impossible to load all set elements into memory, or when data arrives in a stream. To tackle such practical challenges in large-scale set encoding, the general set-function constraints of permutation invariance and equivariance are not sufficient. We introduce a new property termed Mini-Batch Consistency (MBC) that is required for large scale mini-batch set encoding. Additionally, we present a scalable and efficient attention-based set encoding mechanism that is amenable to mini-batch processing of sets, and capable of updating set representations as data arrives. The proposed method adheres to the required symmetries of invariance and equivariance as well as maintaining MBC for any partition of the input set. We perform extensive experiments and show that our method is computationally efficient and results in rich set encoding representations for set-structured data.

Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding
Andreis Bruno', / Jeffrey Ryan Willette', Juho Lee Sung Ju Hwangh?
KAIST 1 South Korea
AITRICS 2 South Korea
fandries, jwillette, juholee, sjhwange2)
Abstract
Most existing set encoding algorithms operate under the implicit assumption that all the set elements are accessible, and that there are ample computational and memory resources to load the set into memory during training and inference. However, both assumptions fail when the set excessively large such that it impossible to load all set elements into memory. when data arrives in stream. To tackle such practical challenges in large-scale set encoding, the general set-function constraints of permutation invariance and equivariance are not sufficient. We introduce a new property termed Mini-Batch Consistency (MBC) that is required for large scale mini-batch encoding. Additionally, present scalable and efficient attention- set encoding mechanism that is amenable to mini-batch processing of sets, and capable of updating representations as data arrives. The proposed method adheres to the required symmetries of invariance and equivariance as well maintaining MBC for any partition of the input set. We perform extensive experiments and show that our method is computationally efficient and results in rich set encoding representations for set-structured data.
Introduction
Recent interest in neural network architectures that operate sets [14, 6] has garnered momentum given that many problems in machine learning can be reformulated as learning functions on sets. 7]. classification, prediction [8], and set extension can all be cast in this framework of learning functions over sets. Given that sets have no explicit structure on the elements, such functions are required conform to symmetric properties such as permutation invariance or equivariance for consistent processing.
Problems such as point cloud classification [13], image reconstruction
A defining property of many practical functions over sets involves encoding of the input set to vector representation, the set encoding. This set encoding is then used for downstream tasks such as reconstruction classification. In DeepSets [14], eem-decomposable family of functions is derived for a class neural network architectures that encode a given set to such a representation. However, the simplicity of the functions derived in DeepSets makes ineffective for modeling pairwise interactions between the elements of the sets. Set Transformers [6] remedy this by using Transformers [12) to model higher order interactions resulting in rich and expressive set representations.
In all these works, there an implicit assumption in the experimental setup that the cardinality of the set is manageable, and that there are ample computational resources available for processing all the elements during the set encoding process. However in real-world applications such as large scale point cloud classification and data intensive applications in particle physics, the set size be extremely large. In such cases, even if one has access to a set encoding function that is linear w.r.t. the number of elements in the set, it is still impossible encode such sets in single batch since may not be able to load the full set into memory. Current encoding methods deal with this issue by sampling
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Deep Sets
Set Transformer
g
Slot Set Encoder
Figure 1: Mini-Batch Consistent Set Encoding. Each figure shows the processing of (left) mini-batch encoding of {21, ) E S, and (right) {6),,5} E S. Deep Sets: can he made to encede batches consistently but cannot model pairwise interactions. Set Transformer: self-attention prevents mini-batch consistent encoding. Slot Set Encoder: Attention w.r.t. parameterized slots allows for both attention and mini- baiih consistency.
subsets of the full set (as data preprocessing step) and encoding them to representative vector, as done in the ModelNet40 [13] experiments in [14, 6]. However, this results in the loss of information about the full set, which is undesirable when huge amounts of monetary resources are invested in obtaining data such as in the Large Hadron Collider (LHC) experiments, and hence is imperative that the data is exploited for maximum performance on the given task. As such, it is necessary to have a set encoder that can take advantage of the full data, in a computationally efficient way.
To this end, when the set is too large to fit in the memory, or given as stream of data, we propose to iteratively encode over multiple rounds, by randomly partitioning it, encoding each random subset (mini-batch) with the set encoder at each round, and aggregating all the encodings. However, naive mini-batel set encoding could compromise essential properties of the set encoder such as permutation invariance and equivariance. To use concrete example, in Figure 2 show the performance of Set Transformer (in Negative Log-Likelihood) on an image reconstruction task. By processing the full set (in this case concatenations of pixel and coordinate values), the model performs as expected (SerTransformrr in Figure 2). However when we instead partition the set elements into mini-batches, independently encode each batch and aggregate them to obtain single set encoding, model as shown in Figure 2 (Set' Sherrvranformerr (MiniBatch) in Figure 2). This degradation stems from the attention module in Set Transformer where mini-batch encoding and aggregation results in mini-batch inconsistent set encoding.
-22
aethonderee
-34
-3.5
-3.
-1.
-12
Figure 2: Performance degradation (CelehA image reconstruction) when eval- uating Set Transformer [6] a minibatch vs. non- minibatch setting.
observe performance degradation in the
In this work, identify and formalize key property, necessary for mini batch encoding of sets. In what follows, we present the necessary constraints required for MBC and propose slot based attentive set encoder, the Slot Set Encoder (SSE) which is MBC. SSE largely alleviates the performance degradation in mini-batch encoding of large sets. We depict MBC and non-MBC set encoders in Figure 1.
Consistency (MBC), that is
Our contributions in this work are as follows:
We identify and formalize key property of set encoding functions called Mini-Bater Consistency (MBC), which requires that mini-batch encoding of a set is provably equal to the encoding of the full set (in Section 2.2).
We present the first attention based set encoder that satisfies MBC, Our method can efficiently scale arbitrary set sizes by removing the dependence of through the use of slots which frees the transformer from requiring to process the full set in a single pass (in Section 2.3).
the transformer
We perform extensive experiments various tasks such as image reconstruction, point cloud classification and dataset encoding where we demonstrate that SSE significantly outperforms the relevant baselines (Section 3).

---

2 Mini-Batch Consistent Set Encoding with Slot Set Encoders
2.1 Preliminaries
Suppose we are given a dataset of sets D = f.X,...,, Xx} where [D] = N and each X, E R where ny is the number of elements in X; and each element x:!") (that is, the jth element of X)) is represented by d dimensional tensor. For each X,, the ordering of the n, individual elements considered arbitrary and could be any permutation of the indices. We further assume that both n, and d can be large enough such that processing an instance X; is prohibitive both in terms of memory and computation. As an example, the processing task could be encoding each element X; to representative vector to be used for downstream tasks such as set reconstruction, set classification or set prediction. Current set encoding methods such as Zaheer et al. [14] and Lee al. [6] deal with such large sets by sampling a smaller subset of the original set as preprocessing step. In essence, they approximate the full set with a random subset which can be undesirable when we want to exploit all the data at our disposal for maximum performance. In this section, we present a method for encoding arbitrarily sized set data via a mini-batch encoding scheme. Our method can then iteratively obtain the full set representation. Our
encode subsets of X4 and aggregate the subset-encodings method is Mini-Bane Batch Consistent, invariant to the order of the subsets, and invariant to permutations on the set elements. For clarity, when say set X; is mini-batch processed, we mean that random partition function can be applied to X, to obtain proper subsets of X; The subsets, which we call mini-batches, are then processed independently and aggregated to obtain representation for X;.
2.2 Mini-Batch Consistent Set Encoding
We consider a single element X, E D with n, elements each with dimensions. We truncate notation and write X,, nu X and n respectively since we present our algorithm for a single sample here. Our ultimate goal is to encode the set X to representation Z Rd' where d' is the dimension of the encoding vector. We assume that both d and n are large such that it is not feasible to process X as whole. To get around this computational problem, of X where the mini-bated samples are with respect to the number of elements n. Specifically, we partition X such that X can be written as finite union X X1 U U Ux and each X; in this partition has n4 = | elements and p is the total number of partitions. We further assume that X is partitioned such that we can efficiently process given mini-batch X, in the partition. Each partition X, is encoded with set encoding function f(X,) to obtain Z;. We then define the set encoding problem as follows:
propose perform mini-batch encoding
== =((((),)), ((X)), f(Xp))
(1)
where g(-) is an aggregation function that takes individual Z,'s and combines them to obtain 2. We further require that the functions f and g are Mini-Batch Consistent.
Property 2.1 (Mini-Batch Consistency). Let X Raxd be partitioned such that X Xi U X2
...UXp and f : Raa + Ed be a set encoding function such that f = Z. Given an aggregation Consistent if and only if
function {Z E Idd 13 + Rdd, and f are
(U((X))..,,
(()
Property 2.1 ensures that no information is lost by encoding the full set in independent and that the aggregation function 9 guarantees the same output as encoding the full set in a single batch. Additionally, Property 2.1 is general and places no restriction on the form of the functions and g except that when used together, MBC must be satisfied. In what follows, we define an attention based set encoding function ff together with aggregation functions that satisfy Property 2.
2.3 Slot Set Encoder
In this section, we provide formulation for an attention based set encoding function f in Equation I which utilizes slots [8]. Given an input set X E R we begin by sampling random initialization for K slots S E I xxte where h is the dimension of each slot. Specifically,
S ~ Nu, diag(o)) 18 KA
(2)
3

---

where μ E IR' 1x and a E IR 1xh are learnable parameters. Optionally, instead of sampling random initialization for the K slots, we can instead designate deterministic learnable parameters Sx E R xxh An advantage of random initialization over learnable parameters is that at test time, we can increase or decrease K for the randomly initialized model. In the ablation studies in section 3. 1.1.1, we explore the performance gains that result from the choice of slot initialization. The key design choice in the slot set encoder is that unlike attention based set encoding methods like Lee et al. [6]. we compute attention over 5 (no sell-attention) instead of computing it over the n elements of X (self-attention).
This allows us to remove the dependence of the set encoding function on n, alleviating the requirement of processing the entire set at once, and allowing for MBC encoding of partitions of X since the same slots are used to encode all mini-batches, Specifically, we compute dot product attention between S and X according to the following formulation:
attn.,y == (Mi.) where M == 1 k(X) q(S)" E TR vd
(3)
where k and q are linear projections of X and to a common dimension d and a is the sigmoid activation function. There is a key difference in our attention formulation in that we use sigmoid instead of softmax to construct the attention matrix. Using a softmax, even when computed over S, breaks MBC it requires batch dependent normalization term which will vary across different partitions of X. By using the sigmoid function, we are able to free our model of this constraint and satisfy MBC. Finally, we weight the inputs based on the computed attention matrix to obtain the final slot set encoding $ for X:
$ WT. x(X) Kxd where W = Σ aftn,
attna
(4)
where o also a linear projection applied to X and W E Bank are the weights computed over slots instead of elements. The normalization constant in W occurs over the slot dimension K, and only contains a linear combination of single n, E and slots S and therefore has no dependence on other elements within the partition X,, The Slot Set Encoder is fully described in Algorithm 1.
The Slot Set Encoder in Algorithm is functionally composable over partitions of the input set X. More concretely, for given slot initialization and any partition of X,
f(X) =
f(xp))
(5)
Where the aggregation function g is chosen to satisfy Property 2.1. This is convenient since allows us to define g from the following set of operators: g E {mean, sum, max, min]. We note that to satisfy MBC, we have chosen g to be an associative function but it is entirely possible to formulate a non-associative g and corresponding f that satisfies MBC and we leave that for future work.
Proposition 2.1. For given input set X E Ruxd and slot initialization S E R xxxd the functions f and g as defined in Algorithm are Mini-Batch Consistent for any partition of X and hence satisfy Property 2.1.
Proposition 2.2. Let X E Rexd and S E Rxd be an input set and slot initializaaton respectively. Additionally, let SSE(X, S) be the output of Algorithm 1, and T E Raxn and ag E RR be arbitrary permutation matrices. Then,
SSE( X, 5) = ns SSE(X, (X
Proofs of Proposition 2.1 & 2.2 involve showing that each component in Algorithm I are both MBC, and permutation invariant or equivariant. Details can be found in the Appendix with discussions on why the baselines fail satisfy MBC.
2.4 Hierarchical Slot Set Encoder
The Slot Set Encoder can encode any given set X E Raxd to a d dimensional vector representation. In the mini- Mini-batch setting, all partitions are independently encoded and aggregated using the aggregation function g. In many practical applications, it is useful to model pairwise interactions between the elements in the given set since not all elements contribute equally to the set representation. Indeed this is the key observation of Lee al. [6] which models such pairwise and higher order interactions between set elements and obtain significant performance gains over Zaheer et al. [14] which assumes

---

that all elements contribute equally to the set encoding. In the Slot Set Encoder, we would like be able to model such pairwise and higher order interactions. However satisfy Mini-Batch Consistency, we cannot model interactions between set elements given that the Slot Set Encoder removes all dependencies on n. Hence, we propose model interactions among slots via a stack of Slot Set Encoders. Specifically, instead of using a Slot Set Encoder with K = 1, we stack 7 Slot Set Encoders each with K; slots and set Kr = for the final encoder. Concretely, if we let SSE be an instance of the Slot Set Encoder defined in Algorithm I, then we can define a series of such functions, SSE,..... SSEr. Then for a set X, a composition these Slot Set Encoders is valid set encoding function that satisfies all the requirements outlined in Section 2.3, such that:
f(X)= ..E (SEE (X)))
(6)
This gives hierarchy of Slot Set Encoders capable of modeling pairwise and higher order interactions between slots. The Hierarchical Slot Set Encoder is analogous to stacking multiple Induced Set Attention blocks followed by final Pooling MultiHead Attention used in Set Transformer [6]. The major difference that our model still remains MBC, a property violated by Set Transformer.
2.5 Approximate Mini-Batch Training of Mini-Bateh Consistent Set Encoders
Set encoding mechanisms such Zaheer al. [14] and Lee et al. [6] require that gradient steps taken with respect to the full set. However, in the Mini-Batch Consistent setting described so far, this is not feasible given large set sizes or constraints on computational re- sources. A solution this problem train on partitions of sets sampled at each iteration of the optimization process. We verify this approach empirically and show that the Slot Set Encoder presented so far can be trained on partitions of sets and still generalize to the full set at test time. these experiments (Figure 3b), train set encoders on subsets of sets sampled at each optimization iteration and per- form inference on the full set. Specifically, some iteration f, we sample a mini-batch of size B x ň d, where B is the batch size and ñ is the cardinality of partition of set with elements (ñ < n). For this empirical analysis, we use the CelebA dataset [7] where the pixels in an image forms set and the task is to reconstruct the image using the set encoding of few pixels given as context points.
Algorithm Slot Set Encoder.
Partitioned input X (X1,..., X p) Initialized slots R Kx
aggregation function
1: Input: X [X,,, X2, Xp). RR
Output: $ S Kxd
Initialize
4: S LayerNorm(S)) for i = to p do
6: Compute at att,, 8) using Equation 3 Compute (Xi, attn;) using Equation 4 $ $= S.)
9: end for
10: return $
3 Experiments
We evaluate our model on the ImageNet [2], CelebA [7], MNIST [5] and ModelNet40 [13] datasets. Details these datasets can be found in the Appendix. In these experiments, all sets arrive in streaming setting in a mini-batch fashion where we only get subsets (batches) of the full set at an instance. Once the mini-batch is encoded, we only have access to the set encoding vector (we do not keep the original set elements since it requires further storage) which must be updated as more batches of the set arrive. We provide further implementation details in the Appendix.
3.1 Image Reconstruction
In this task, we are given a set of pixels (each 3 dimensional) with their corresponding coordinate points (each dimensional). termed context values, and the task is to reconstruct the full image given the coordinates of the pixels we wish to predict using only the context points. The context points are 5 dimensional vectors formed by concatenating the pixels and their corresponding coordinates. We use Conditional Neural Processes [3], CNP, for this task. CNP has a set encoding layer for compressing the context points into single vector representation that then passed to a decoder for constructing the full image. Here, our goal is to demonstrate that the Slot Set Encoder is proper set encoding
5

---

-07
DeapSete
SefTunatorer
-14
SeepSete
Satthanloome
0.50
Dreptens
SerTranitorme
SelScinnodeer
-35
5 steceonee
0.25
-1.
0.00
-0.8
-9.25
-1.0
-18
-0.50
-32
-3.75
-3.9
-3.4
Deeplett
-1.00
-2.0
SetT-amitr met
ShotSerEricoder
200
400
900
610
1000
1000
2000
300G
4000
200
200
3100
400
500
50
100
150
200
Set Size
(a) NLL on CelebA
(b) Minii-Batch Testing (L) (c) Miii-BBtchee Testing (S) (d) Convergence Plot
Figure 3: (a) NLL for different set sizes (full sets, mini-bateh processing is used here) on the image reconstruction task. (b) Mini-Bater Testing of the reconstruction model (1000 set sized model in Figure 3a) trained on 1000 pixels and evaluated on large (L) set sizes 1000-4000 pixels processing mini-batches of size 1000 at time. (c) Mini-Batch Testing of reconstruction (sdel as in Figure 3b) on smaller (S) set sizes of 100-500 pixels processing mini-batches of size 100 time. (d) Convergence plots for the considered Set Encoders the image reconstruction task with the Slot Set Encoder trained according to Section 2.5.
function capable of learning rich set representations. We sample 200, 400, and 500 context points for images of size 32 32 and 1000 context points for images of size 64 x 64. As baselines, use the mean set encoding mechanism of Zaheer et al. [14] and the Pooling MultiHead Attention (PMA) Blocks of Lee et al. [6]. We compare this with variant of the Slot Set Encoder with random slot initialization. All the methods encode the given set to a 64 dimensional representative vector and are trained for 200 epochs. Additionally, for fair comparison, we only use single Slot Set Encoder in our models as well as single PMA block for Set Transformers.
The results for these experiments are presented in Figure 3a where it can be seen that the Slot Set Encoder model learns richer set representations resulting in lower negative log-likelihood when trained on full sets. Additionally in Figure 3b we demonstrate the scalability of the Slot Set Encoder where models initially trained on 1000 context points (in Figure 3a) are used to encode larger sets of context points (2000- 4000) at test time. Since Set Transformer is not Mini-Batch Consistent, it cannot make use of the additional context points as it violates Property 2.1 hence in the constraints of the experiments,
use 1000 context points for Set Transformer. DeepSets and Slot Set Encoders can utilize the additional data but the representation obtained by the Slot Set Encoder is richer in its representation power resulting in significant performance gains test time as well better generalization to larger set sizes than was used during training. In Section 3.1.1 we further discuss Figure 3b in the context of constrained resources and streaming data.
3.1.1 Ablation
The Slot Set Encoder presented so far has various components and evaluation modes that can have an impact on the richness of the resulting set representation. In this subsection, we perform extensive ablation studies to explore how these various components contribute to the performance of the encoder.
Constrained Resources and Streaming Data In order to demonstrate the utility of the models that satisfy Property 2. we simulate cases where the computational resources are limited in terms of memory. Specifically, we assume that our computational budget allows us to only compute the set representation for 1000 context points at time. Under these constraints, the Set Transformer can only be used to encode 1000 context points since the Softmax attention layer requires all the set elements to be in memory. For MBC DeepSets, and Slot Set Encoder, and iteratively update the set representation until the full set has been encoded. Additionally, this setting is akin to the case of streaming data where the set elements arrive or are obtained irregular intervals and hence the set encoding representation must be incrementally updated.
can mini-batcher of of size 1000
We present the results of this experiment for an input image with 4096 pixels with 1000 context points randomly selected and encoded at each time step. From Figure 3b it can be seen that for both DeepSets and the Slot Set Encoder, the more batches we encode, the better the performance. However, the Slot Set Encoder significantly outperforms DeepSets due to it's ability to model interactions between the set elements and slots. Additionally, it can be seen that the MBC models get better as

---

-1.65
-02
Setsatrcotin))
-1.00
-24
0.800
+1.75
-3.00
mein
0.855
1.65
DeepSets dbrcodd
mas
-3.0
0.45
-2.00
-1.4
-2.65
-3.2
1000
2000
3000
4000
200
400
600
810
1000
himber llteracciins
100
2.00
300
400
500
(a) Choice of
(b) Slot Initialization
(c) Hierarchical Slots
(d)
Testing
Figure 4: (a) Effects of the aggregation function 9 model performance. (b) Effects of using random and deterministic slot initialization. (c) Effects of stacking multiple hierarchics of the Slot Set Encoder. (d) Accuracy mini-batch testing MBC models on the ModelNet40 dataset.
more data arrives justifying our initial motivation for making use of all the available set elements instead of using a randomly sampled subset (in the preprocessing stage) in place of the full set.
Mini-Batch Training of Mini-Batch Consistent Set Encoders In Section 2.5, we proposed to train the Slot Set Encoder by approximating the full training procedure on partitions of the set. As we highlighted, under memory constraint and large set size assumptions, we cannot take gradient steps with respect to the full set. To demonstrate that this mini-batch training generalizes when the full set is processed at inference time, we train model on 1000 elements and test it on larger set sizes. This result demonstrated Figure 3b on the CelebA dataset where the model performance improves as larger sets processed. In Figure 3c, take the same model and evaluate it on smaller set sizes of 100-500 context points and compare it to both DeepSets and Set Transformers where again we find that the Slot Set Encoder perform significantly better. Additionally, Figure 3d shows that the models trained in this fashion can achieve a faster convergence rate. We find that for the problems we consider in this work, this mini-batch training is sufficient and generalizes at inference time.
Choice of Aggregation Function In Section 2.3, we stated that we choose the aggregation function g from the set (mean, sum, max, min). On the CelebA experiments, we explore the effects of making such a choice for the model with random slot initialization. In Figure 4a, we take a model trained on sets with cardinality 1000 and evaluate it on larger sets at test time. As can be inferred, the aggregation functions mean and sum consistently outperform the min and max for this task. This is intuitive since in the image reconstruction task, aggregation function that considers the contribution of all pixels is necessary. In Section 3.2, we use a model with max as the aggregation function for Point Cloud Classification where it performs better than the other options. We find that the choice of the aggregation function g is very much informed by the task at hand. We observe a similar trend for the Slot Set Encoder with deterministic slot initialization and report this result in the Appendix.
Deterministic (D) vs Random (R) Slot Initialization For slot initialization, we have the choice of using deterministic or random (with learnable and a) slots. In this choice, we find in almost all our experiments that random initialization performs better across all the tasks we investigated. This trend can be observed in Figure 4b where using random slot initialization results in lower negative bog-likelihood. To explain this behaviour, we reiterate that slots attend to different portions of the input set and hence the usage of random initialization encourages the model to attend to wider coverage of the input sets during training as opposed to deterministic slot initialization.
Number of Slots When we define slots S E Kx we must decide the number of slots K to use and we explore the effects of this choice on model performance here. We fix a model that encodes an input set to 64 dimensional vector and train multiple models with varying K (specifically, we test for = 1, 1232, 64, 128, 256) and we observe that using very small or large number of slots can have a negative effect on the performance of the model for both deterministic and random slot initialization. For this specific model, we find that selecting K between 32 and 128 performs best. Additionally, we find that this parameter is also very dependent on the task. More details are provided in the Appendix.
Dimension of Slots Additionally, for slots S E kxh, we can choose the dimension of each slot arbitrarily since the projection layer will eventually reduce it to h. Similarly to the experiments on the number of slots, we fix model that encodes sets to 64 dimensions using single Slot Set Encoder and vary the dimensions of the slot keeping K 1. We experiment with = 32, 64, 128, 256 and find that this parameter has significant impact on the performance of the model. Specifically,
7

---

Table 1: Test Accuracy for Point Cloud Classification with varying set sizes.
Model
100
200
500
1000
5000
DS
0.8428 +:2 0.0027
0.8518 = 0.0032
0.8574 1 0.0017
0.8528
0.0020
0.8883
0.0008
ST
0.8258 + 0.0042
0.8443 = 0.0023
0.8545 + 0.0019
0.8511
0.0018
0.8604
0.0009
SSE(D)
0.8556 + 0.0048
0.8633 0.0038
0.8710 0.0040
0.8720 +: 0.0009
0.8784
0.0023
SSE(R)
0.8532 + 0.0009
0.8667 = 0.0028
0.8699 + 0.0020
0.8743 0.0021
0.8853
0.0012
find that for slots with random initialization, the dimension of the slots can have a negative impact if not chosen properly. For deterministic initialization, increasing the slot dimension generally results in better performance. In our experiments involving randomly initialized slots, we train multiple models with varying h and pick the best one. The results discussed here can be found in the Appendix.
Hierarchical Slot Set Encoder Finally, we explore stacking multiple layers of Slot Set Encoders based on the hierarchical formulation presented in Section 2.4. We start with a single Slot Set Encoder with K- and h 32 and for each hierarchy, we double the slot dimension. For this experiment, we find that stacking multiple Slot Set Encoders for the tasks we consider provides marginal performance gains and can result in performance degradation when the hierarchy gets very deep. This is shown in Figure 4c where the performance of the model degrades after 4 hierarchies.
3.2 Point Cloud Classification
0.3
Doopstts SkadedEncddr
Using the ModelNet40 dataset, we train models that encode an input (point cloud) to a single set representation that is used to classify the object as one of 40 classes. We follow the experimental settings of Zaheer et al. [14] and Lee et al. [6] where for point clouds with cardinality less than 5000, all the compared methods encode the point cloud 256 dimensional vector and 512 for point clouds with 5000 points or more. Set Transformer and DeepSets use proprietary version of the ModelNet40 dataset however in our experiments, we use the raw ModelNet40 dataset which can sample an arbitrary number of points, and sample points for each instance randomly at every iteration.
085
Eanc
207
073
0.55
0.2
200
403
900 800
1930
Auwuy
Figure 5: Accuracy vs. 'way' in the centroid pre- diction task. As the prob- lem becomes harder, Slot Set Encoder outper- forms DeepSets.
The ModelNet40 dataset is difficult to classify when the number of ele- ments in the set is relatively small. This is shown in Table 1 where we test the two variants of our model with both deterministic (D) and random (R) slot initialization. The Slot Set Encoder models outperform DeepSets and Set Transformer. As the number of points increase to 5000, DeepSets and the Slot Set Encoder model achieve similar performance as expected. From the Table 1, it can be seen that the two competing models are DeepSets and our models and hence we check the generalization of both models the Mini-Bater testing setting where a model trained on sets with 100 elements is tested on 200, 300, 400 and 500 elements. As can be seen from Figure 4d, although DeepSets is Mini-Batch Consistent, it experiences performance degradation when the set size grows on the ModelNet40 dataset. Conversely, the Slot Set Encoder based model gets better as the set size increases at test time showing that it generalizes better than DeepSets with Mini-Batch training (Section 2.5).
3.3 Cluster Centroid Prediction
In order to show the effectiveness of Mini-Batch Consistent set encoding on sets of high cardinality, we experiment with Prototypical Networks [11] for classification on the ImageNer and MNIST datasets (see the Appendix for MNIST results). We use a pretrained ResNer50 backbone with the final layer removed, and train the set-encoding layers for DecpSets/SSE in the latent feature space. We train for 20-way classification with 2 support and 2 query instances. Each support set is encoded to a 128 dimensional vector (centroid), and we then train the model to minimize the Euclidean distance between the query instances and the corresponding class centroid. There is no decoder in this experiment, as we only utilize the pooled representation for inference. We investigate the effects of testing on both lower and higher "way' (5-1000 way) problems (Figure 5) and different encoded support set sizes (Figure 6). Given the large size of the set, we compare our model with DeepSets, as is the only other MBC set encoder. In Figure 6 we plot accuracy vs. encoded set size for various ways'. In both the I and 20 way tasks, our SSE and Deepsets show similar performance, we attribute
8

---

0.36
0.44
0.92
0.65
0.30
0.62
3.34
0.32
0.48
0..22
20.00
0.50
0.86
0.34
0..5
0.36
0.82
040
0.34
0.66
Deepdiets
SotSstEncoper
0.82
Deepdeets
SctSetEncocer
0.76
0.76
Deepdiet
SctSetEnceder
0:35
poepdets SetSeiEnsede
10 50
100 500 1000
12 50
100 500
1000
10 50
100 500 1000
10 100 500 1000 Incaded
(a) 5 5-way
(b) 10-way
(c) 20 w0w
(d) 1000-way
Figure 6: Accuracy vs. Encoded set sizes for different n-way classification problems ImageNet. Our Slot Set Encoder performs best on n-way problems which differ from what was seen during training (trained on 20-way). 1-way provides singleton set, and therefore is about equal for both models.
this to the fact that way classification provides minimally informative singleton sets, and 20-way classification is the setting which the model was trained. The Slot Set Encoder generalizes better on the other 10 and 1000 way experiments which differ from the training setting.
4 Related Work
Set Encoding In DeepSets, Zaheer et al. [14] show that neural architectures for set data are required to obey the permutation invariance/equivariance symmetry and can be formulated as the following sum-cccomposition two functions, fxx) = EiE o(x,) where & is a feature extractor and f is some non-linear activation function such as ReLU, Additionally, instead of the sum operator, min, max and mean also satisfy the imposed constraints. While there exist previous works [9] on invariant pooling methods, DeepSets provided universal model for constructing such neural networks. However, the architectures proposed in DeepSets are simple and some cases lack expressivity. Set Transformer [6] solves this issue by formulating set compatible version of the Transformer [12] which is capable of modeling pairwise interactions among the input set. Additionally, Lee et al. [6] reduce the complexity of Transformers from O(n²) to ((m)) by introducing inducing points that serve as queries to reduce the size of the input set before self-attention. Other methods that tackle the set encoding problem include Murphy et al. [10] and Zhang al. [15]. However all of these methods require processing all elements of the input set at once and hence cannot be used in the MBC setting.
Slot Attention Locatello al. [8] introduced a slot attention mechanism to learn object-centric representations. Slots are variables which adapt and bind to objects (sets of features) to form non-stationary, input-dependent representations. The slot attention mechanism works by iteratively updating an initialized set of slots with a GRU [1] to adapt them to the current inputs. Through the adaptation process, the slots compete for sets of similar features which compose objects by specializing their weights to achieve a higher activation value in the softmax of the attention matrix. While the idea of slots provided initial inspiration for work, they encoding like DeepSets and break MBC for a couple of reasons: 1) The GRU iteratively updates the slots, which depends heavily the elements. 2) the Softmax normalization layer depends all elements in the set and 3) updates on hidden states in the model also break MBC.
not designed for set
5 Conclusion
In this work, we have identified key limitation of set encoding methods when applied to large scale streaming set encoding. Additionally, we identified key property, Mini-Batch Consistency. that is required to guarantee that set encoding methods are amenable to mini- batch processing of set. We further detailed the formulation of an instance of an attentive set encoding mechanism dubbed Slot Set Encoder that respects the symmetries of permutation invariancefequivariane. Mini-Batch Consistent and can be trained with mini-batch approximation of the full set gradients.
is
We demonstrated the utility of SSE with extensive experiments and ablation studies and verified that

---

SSE is capable of learning rich set representations while satisfying MBC. An interesting direction of future research would be use more sophisticated methods to approximate the mini mini-batch training routine (in Section 2.5) with respect to the full set without having to consider all elements in the set.
6 Acknowledgment
This work was supported by the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF-2018R) ASA 1059921), the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2020-0-00155) the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-202 116666555), the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the
Korea governmen((MSTI) 550.2021-0-02000 Artificial Intelligence Innovation Hub), the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea (MV)) 20--00007000 Artificial Intelligence Graduate School Program(KAIST) and the HPC Support Project, supported by the Ministry of Science and ICT and NIPA.
References
[1] K. Cho, B. van Merrienbner, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using encoder-decoder for statistical machine translation, 2014.
[2] Deng, W. Dong, Socher, L..J. Li, Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages -48-255. leee, 2009.
[3] Garnelo, D. Rosenbaum, Maddison, Ramalho, Saxton, Shanahan, Tch, J. Rezende, and S. Eslami. Conditional neural processes. arXiv preprint arXiv: 1807.01613, 2018.
[4] H. Kim, A. Mnih, J. Schwarz, M. Garnelo, A. Eslami, D. Rosenbaum, O. Vinyals, and Y. W. Teh. Attentive neural processes. arXiv preprint 190..56,, 2019.
[5] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner: tradient-based learning applied to document recognition. Proceedings of the IEEE, (6:127) 222744 1998.
[6] Lee, Lee, J. Kim, A. Kosiorek, S. Choi, and W. Teh. Set transformer: framework for attention- based cormutation-ivvaiian neural networks. In International Conference on Machine Learning, pages 3744-3753. PMLR, 2019.
[7] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[8] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and
T. Kipf. Object-eentric learning with slot attention. arXiv preprint arXiv:2006. 15055, 2020.
[9] D. Lopez-Paz, R. Nishihara, Chintala, Scholkopf, and Bottou. Discovering causal signals images.
In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 6979-6987, 2017.
[10] R. L. Murphy, B. Srinivasan, V. Rao, and B. Ribeiro. Janossy pooling: Learning deep permutation invariant functions for variable size inputs. arXiv preprint arXiv:1811.01900 2018
[11] J. Snell, K. Swersky, and R. S. Zemel. Prototypical networks for few-shot learning. arXiv preprint 1:X3vv1703.05///1 2017.
[12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neurul information processing systems, 30:5998-6008, 2017.
[13] Z. Wu, Song, A. Khosla, F. Yu, L.. Zhang, X. Tang, and J. Xiao. 3d shapenets: deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[14] M. Zaheer, S. Kottur, S. Ravanbakhsh, Poczos, R. R. Salakhutdinov. and A. Smola. Deep sets. In Advances neural information processing systems, pages 3391-3401. 2017.
[15] Y. Zhang, J. Hare, and A. Prügel-Bennett Fspool: Learning set representations with featurewise sort pooling, arXiv preprint arXiv: 1906. 02785,022255220 2019.
10


Reviews:
[{"review_id": "eQ56H-LeSXh", "reviewer": "Reviewer_P7K7", "summary": "This paper addresses the problem of set encoding when the dimensionality and/or cardinality of the set it too large to fit into memory to feed into an encoder or when set elements arrive in a stream. This paper defines a property, mini-batch consistency, for set encoding functions that encode the set in non-overlapping subsets (mini-batches) then aggregate the mini-batch encodings to achieve the set encoding. If obeying mini-batch consistency, the encoder achieves the same set encoding as if the encoding function was applied directly to the entire set. Finally, the paper proposes an encoding function, Slot Set Encoder, to satisfy mini-batch consistency.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem formulation is interesting and appears to be novel. However, the specific benefits of the proposed MBC method, SSE are under motivated in the empirical evaluation.\nAs compared to Deep Sets [1] which also satisfies mini-batch consistency, SSE’s claimed improvement is modeling set element interactions by proxy of modeling the interaction between slots. However hit seems the using deeper hierarchies hurts performance or has no effect on performance compared to number of hierarchies T=1. Since the number of slots in the T-th hierarchy is set to K_T=1, then SSE is also using one slot and one hierarchy. Thus, slots and hierarchies proposed in the model don’t get exploited or properly motivated. Indeed most performance measures evaluated in experiments are comparable between SSE and Deep Sets, with the exception of generalization to larger sets at test time (Fig 3b) where SSE significantly outperforms Deep Sets.\n\n[1] Zaheer, Manzil, et al. \"Deep Sets.\" Advances in Neural Information Processing Systems 30 (2017).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations with regards to the motivation for SSE as discussed above are not addressed in the paper, ie are slots and hierarchies actually useful in the proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "YJ6OvUM9kaG", "reviewer": "Reviewer_s6Je", "summary": "The authors proposes the use of attention based set encoder which can scale to arbitrary set sizes. Attention in Transformers has the quadratic dependence on number of positions (i.e. cardinality of the set). The proposed method makes the problem easier by enabling the use of slots while preserving a property which method defines as mini-batch consistency. The proposed method is invariant to the order of the subsets, and invariant to permutations on the set elements.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Proposed method: The paper proposes the  Slot Set Encoder that respects the symmetries of permutation invariance / equivariance. The basic idea is to represent K slots (can be randomly initialized or the initial representation can be learned). Now, the paper proposes to use the cross-attention between the resulting K slots and over partitions of the input set X\n\nExperiments: The paper evaluates the proposed model model on various datasets  (MNIST, CeleA, ImageNet,  ModelNet40) datasets. The paper compares to various baselines like DeepSets and  Set Transformer, and achieves superior results as compared to both the methods. \n\nPros:\n- The paper is very well written.\n- The authors compared to the relevant baselines. The reviewer particularly liked the ablations done in section 3.1.1 that evaluates the proposed model by restricting the memory, showing the importance of aggregating function, and the effect of using random or learned initializations of the slots, effect of the number and dimensionality of the slots. \n-  Its interesting that the paper also explores the use of hierarchical set encoding, and shows the result that the performance decreases as we increase the number of layers which intuitively makes sense. \n\nCons:\n- Some of the most relevant work is missing in the current version of the paper. Their are different papers [1, 2] which uses a similar idea (but for different problems). The basic idea is same: learn a encoding function for the set of slots to reduce the computational complexity.  [1] tries it in the context of transformers/modular RNNs. The representation of different positions/RNNs act as a set containing different tensors. The idea is to learn a latent bottleneck consisting of a set of slots. Then there's cross-attention between these set of slots and the original representation of different positions. Thus reducing the complexity of attention from quadratic to linear in the number of positions. \n\n- The idea of cross-attention in the way used in the paper is first used in RIMs [3]. The contribution of Slot attention was to make it iterative. \n\nRelevant Literature work:\n- [1] Coordination Among Neural Modules Through a Shared Global Workspace, https://arxiv.org/abs/2103.01197\n- [2] Perceiver: General Perception with Iterative Attention, https://arxiv.org/abs/2103.03206\n- [3] RIMs, https://arxiv.org/abs/1909.10893\n\n=========================================================\n\nAfter rebuttal:\n\nI thank the authors for their response. I will keep my original score. Good results in the hierarchical setup, or just carefully constructed experiments for showing improved results would be an interesting area to explore. \n\n\n\n\nQuestions for the authors:\n- The reviewer is pretty happy to increase my score, if the relevant work is appropriately referenced. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations adequately addressed. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "VxS-GHreyqe", "reviewer": "Reviewer_9cx8", "summary": "The paper addresses the problem of set encoding when the set is too large to fit in the memory, or given as a stream of data. The proposed method randomly partitions the set,  encodes each random subset and then aggregates all the encodings. The key contribution of the proposed method is that the aggregation of the encodings is required to be equal to the encoding of the full set and it satisfies permutation invariance and equivariance. This property is formally defined and named as Mini-Batch Consistent Set Encoding in the paper. Compared to DeepSets, which also satisfy Mini-Batch Consistent Set Encoding, the proposed method can model pairwise interactions, while DeepSets lack this property. The encoding is implemented using previously introduced slot attention [8] (the attention is done over “slots” instead of self-attention) with the two differences: softmax is replaced with sigmoid in constructing the attention matrix and GRU update of slots is omitted here. The experiments compare the proposed method with two previous methods for set encoding in three tasks showing some improvement.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have two main problems with the paper. One is the technical novelty of this paper, which in my opinion is very modest which is extending previous methods for set encoding by parts with the slot attention (utilized from another previous work). The  Hierarchical Slot Set Encoder that is the most novel part of the paper (in my opinion) doesn’t seem to show much improvement.\nMy second problem is with the clarity of the paper. First, it assumes prior knowledge about slots. It would be very helpful to give some intuitive explanation about the mechanism that is central to this paper. Second, I was confused about the use of word “partitions”. First, I thought that it creates multiple partitions of the set as a part of the algorithm, but then I realized that the authors use the word “partitions” to refer to sub-sets of the same partition. \nAlso I cannot agree with the claim (in the contributions part of the paper) that experiments are extensive. All the experiments are contrived, if there is a real need in the proposed method, it would be more convincing to include an experiment in which the conditions assumed in the paper are satisfied. The motivation was the data from the LHS, but the experiments show only standard vision applications.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "8LxmQIzHdz9", "reviewer": "Reviewer_U9Cv", "summary": "This paper introduces a new concept called “mini-batch consistency” (MBC), which is presented as necessary for efficiently training large-scale set encoding models using mini-batch processing of sets.  MBC is appropriate when the cardinality of the sets in the data is very high, and thus all set elements cannot be loaded into memory, or when the data arrives in a stream.  MBC adheres to the required properties of invariance and equivariance for set encoding models.  A MBC-based set encoding mechanism is also proposed, called the slot set encoder.  Extensive experimental results are provided, which demonstrate that the proposed method is scalable and generally outperforms competing approaches.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The MBC property, along with the proposed slot set encoding mechanisms, appear to be technically sound and novel.  The paper is well written, with a clear presentation of the proposed approaches.  Scalable mini-batch training of set encoding models is a relevant problem, and the contributions in this paper are significant, and others will likely build on these ideas in future work.  The theoretical contributions (Propositions 1 and 2) are sound, and the proofs of both propositions are straightforward and appear to be correct.  The experimental results are convincing, and clearly demonstrate that the proposed slot set encoder approaches outperform competing approaches when trained on mini-batches of sets.\n\nA few additional remarks:\n- It would be helpful for the authors to include a comparison of the memory consumption (or space complexity) for their slot set encoder approach vs. competing approaches.  Such an analysis would make the memory consumption advantages of their approach clear, particularly for large datasets where mini-batch training has significant advantages.\n- It is somewhat surprising that the hierarchical slot set encoder approach gives only marginal gains in predictive performance at best, as shown in Figure 4c.  Do the authors have an explanation for this behavior? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have included a short, but adequate, discussion of the limitations of their work in Section 5 and Appendix A.1.  Potential negative societal impacts are adequately discussed in Appendix A.1.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "5hYD1Yh9Gly", "reviewer": "Reviewer_DaP6", "summary": "This paper studies the problem of set encoding in the setting where the cardinality of the set is prohibitively large or unbounded. For example, this is the case when dealing with streaming data. A property called Mini-Batch Consistency (MBC) is defined, and it is argued that set encoding algorithms must have the MBC property to learn useful representations of the entire set when partitions of the set are being processed incrementally. A slot-based algorithm that is MBC is proposed. It models interactions between the inputs and slots to learn strong representations. The algorithm, called the Slot Set Encoder (SSE), is compared against Deep Sets and the Set Transformer on a variety of relevant tasks. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n--------------\n- This task has yet to be considered in the set representation learning community.\n- The method, SSE, offers a unique capability of processing partitions of sets incrementally while also learning strong representations with input-to-slot attention.\n- However, the Introduction and Related Work noticeably are missing some recent papers on set encoding, for example, [1-3], that would help frame the proposed SSE better within the wider context of set encoding methods. These should be discussed in a revised version.\n \n\nQuality\n--------------\n- I believe the MBC property, the SSE algorithm, and Propositions 1 and 2 are for the most part technically sound, and the experimental results verify the authors’ claims.\n- The output of Algorithm 1 as described is $\\hat{S}$, which is a *set* with cardinality $K$. However, as stated explicitly in L160 “The Slot Set Encoder can encode any given set $X \\in \\mathbb{R}^{n \\times d}$ to a $\\hat{d}$ dimensional vector representation”. And, when checking the provided code, I saw that $\\hat{S}$ is initialized to zeros and additively updated, and then $g \\in \\{ \\texttt{mean}, \\texttt{sum}, \\texttt{max}, \\texttt{min} \\}$ is applied to the final $\\hat{S}$ to reduce the slot dimension $K$ to 1, so that the output of Algorithm 1 is a vector. The Algorithm 1 description in the paper should be fixed to match the code. \n- The paper does a good job of explaining the distinction between the pairwise interactions modeled by Set Transformer’s attention and the input-to-slot attention that SSE learns. I would have liked to see visualizations of what that attention (the $W_{i,j}$) learns in the experiments section. \n- Since the inability to model pairwise interactions between set elements could potentially affect the usefulness of this technique, I would have liked to see this highlighted more prominently in the limitations in the Conclusion. \n \n\nClarity\n--------\n- The paper is easy to read and follow.\n- Error bars are missing from the line plots and should be added to show robustness across multiple training runs.\n \n\nSignificance \n----------------\n- I think this work represents a good first attempt at solving the problem of scalable set encoding. \n- It does not provide a complete solution to the problem, since the modeling of pairwise interactions between set elements is essential for most domains where set encoding is the prevalent technique, such as in point cloud processing.\n- I think this is fine, though, as this raises interesting questions such as whether there are ways to build set encoding algorithms that are guaranteed to be \"approximately\" MBC and can model \"some amount\" of pairwise interactions between set elements.\n \n\nReferences\n---------------\n1. Murphy, Ryan L., et al. \"Janossy pooling: Learning deep permutation-invariant functions for variable-size inputs.\" arXiv preprint arXiv:1811.01900 (2018).\n2. Zhang, Yan, Jonathon Hare, and Adam Prügel-Bennett. \"FSPool: Learning set representations with featurewise sort pooling.\" arXiv preprint arXiv:1906.02795 (2019).\n3. Huang, Qian, et al. \"Better set representations for relational reasoning.\" Advances in Neural Information Processing Systems 33 (2020).\n\nUPDATE POST-REBUTTAL: As explained in my response to the author's rebuttal, my comments have been addressed and therefore I am happy to maintain my initial score of 7.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The key limitation, which is the inability to model pairwise interactions between set elements, is not mentioned in the Conclusion section. This should be remedied. As mentioned above, I believe this represents a promising direction for an extension to this work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have reached a consensus that the paper merits publication, even though some reviewers still rate the paper as borderline. The author responses have done a good job of addressing the reviewers' concerns. I agree with this assessment, and am recommending that this paper should be accepted.
Decision: Accept (Poster)