Set Norm and Equivariant Skip Connections: Putting the Deep in Deep Sets
Authors: Lily H Zhang
Veronica Tozzo
John M. Higgins
Rajesh Ranganath
Venue: ICLR 2022 Submitted (2022)
Keywords: deep learning
permutation invariance
normalization
residual connections

Abstract: Permutation invariant neural networks are a promising tool for predictive modeling of set data. We show, however, that existing architectures struggle to perform well when they are deep. In this work, we address this issue for the two most widely used permutation invariant networks, Deep Sets and its transformer analogue Set Transformer. We take inspiration from previous efforts to scale neural network architectures by incorporating normalization layers and skip connections that work for sets. First, we motivate and develop set norm, a normalization tailored for sets. Then, we employ equivariant residual connections and introduce the ``clean path principle'' for their placement. With these changes, our many-layer Deep Sets++ and Set Transformer++ models reach comparable or better performance than their original counterparts on a diverse suite of tasks, from point cloud classification to regression on sets of images. We additionally introduce Flow-RBC, a new single-cell dataset and real-world application of permutation invariant prediction. On this task, our new models outperform existing methods as well as a clinical baseline. We open-source our data and code here: link-omitted-for-anonymity.

Under review as a conference paper at ICLR 2022
SET NORM AND EQUIVARIANT SKIP CONNECTIONS: PUTTING THE DEEP IN DEEP SETS
Anonymous authors Paper under double-blind review
ABSTRACT
Permutation invariant neural networks are promising tool for predictive modeling of set data. We show, however, that existing architectures struggle to perform well when they are deep. In this work, we mathematically and empirically analyze normalization layers and residual connections the context of deep permutation invariant neural networks. We develop set norm (SN), normalization tailored for sets, and introduce the "clean path principle" for equivariant residual connections alongside novel benefit of such connections, the reduction of information loss. Based on our analysis, we propose Deep Sets++ and Set Transformer++, deep models that reach comparable or better performance than their original counterparts on a diverse suite of tasks. We additionally introduce Flow-RBC, new single- cell dataset and real-world application of permutation invariant prediction. We open-source our data and code here: link- tel--omiiteefooo foateie-fo---ooymiiy
INTRODUCTION
Many real-world tasks involve predictions on sets or multisets, from point cloud classification (Guo al., 2020; Wu al., 2015; Qi et al., 2017a) to the prediction of health outcomes from single-cell data (Regev et al., 2017; Lähnemann et al., 2020; Liu al., 2021;
Yuan et al., 2017). In the context of health prediction among other applications, there is the need for reliable methods that can be applied by application-baser practitioners without the additional requirement of engineering models for every application task.
Accuracy
0.6
layers
50 layers
50 layers In
0.2
An important property of models applied to input sets is permutation invariance: for any permutation of the instances in the input set, the model prediction stays the same. Deep Sets (Zaheer et al., 2017) and Set Transformer (Lee et al., 2019) are two general-purpose deep permutation invariant models. Such models have been proven to be universal approximators of permutation invariant functions under the right conditions (Zaheer et al., 2017; Wagstaff et al., 2019). In practice, however, the architectures are tailored to specific tasks to achieve good performance.
Epochs
(a) Deep Sets
Accuracy
0.6
0.4
layers
16 layers
layers
0.0
In this work, we explore a more general approach to achieving good performance: making networks deeper, strategy which has yielded benefit across many tasks (He et al., 2016b; Wang et al., 2019b). We first note that naively stacking more layers on both the Deep Sets and Set Transformer architectures hurts performance. Consider Figure 1. Deep Sets 50 layers (panel (a)) has significantly worse performance than Deep Sets 3 layers due to the problem of vanishing gradients. Residual connections are often used to combat vanishing gradients, but even the Set Transformer architecture which consists of residual connections has worse performance at a higher depth (panel (b)). Another strategy often used in combination with residual connections in deep architectures is the use of normalization layers, but the use of layer norm does not improve or deteriorates performance of both architectures at high depths, one with residual connections (Set Transformer) and without (Deep Sets). In fact, the Set Transformer
Epochs
(b) Set Transformer
Figure 1: Deep versions of ex- isting models perform worse.

---

Under review as a conference paper at ICLR 2022
paper (Lee et al., 2019) mentions layer norm component of their architecture but turns it off in actual implementation.
In this work, tackle the challenge of designing deep permutation invariant networks with the aim of providing general purpose architectures.
Based on real- applications, we formalize desiderata for normalization layers when inputs are sets and systematically analyze options based on unified framework. We introduce set norm, aimed at optimizing the desiderata we introduce. Set norm is a general, easy-to- tampleemmm normalization that can be applied on any architecture. We empirically demonstrate its benefit with ablation studies (Section 3, Section 5).
We consider settings for residual connections which prevent vanishing gradients and maintain permutation equivariance. Through extensive experiments, we show that the best setting is that of equivariant residual connections (ERC) coupled the "clean residual path" principle (Section 4). To the best of our knowledge, we are the first to analyze this principle across architectural designs and to suggest new benefit of certain residual connections: alleviation of information loss (Section 5).
We develop Deep Sets++ (DS++) and Set Transformer++ (ST++), leveraging set norm and ERC. We demonstrate that both models yield comparable or improved performance over their existing counterparts across a variety of tasks (Section 5). Among other results, these new architectures yield better accuracy point cloud classification than the task-specific architectures proposed in the original Deep Sets and Set Transformer papers. The fact that set norm and equivariant residual connections yield improvements on different architectures suggests that they may also be able to benefit other permutation invariant architectures in the future.
Given growing interest in single-cell analysis (Regev et al., 2017) and the need for methods which can extract signal from single-cell data (Lähnemann et al., 2020), we introduce novel clinical single-cell dataset for prediction on sets called Flow-RBC. The dataset consists of red blood cell (RBC) measurements and hematocrit levels (i.e. the fraction of blood volume occupied by RBCs) for 100,000+ patients. The size and presence of prediction target (hematocrit) makes this dataset unique, even among single-cell datasets in established repositories like the Human Cell Atlas (Regev et al., 2017). Flow-RBC not only provides a new real-world benchmark for methodological innovation but also a potential source of information for biomedical science (Section 5).
2 BACKGROUND
Let S be the number of samples in a set. A function f: + y is permutation invariant if any permutation of the input set results in the same output: f(xx) = f(x). A function x + y permutation equivariant if, for any permutation
the outputs are permuted accordingly: a(Tx) = TO Tо((x) (x). A sum/max-decome decloppoabeeeeee function f X + y one which can be expressed via an equivariant encoder a X + 2 that maps each input sample in X' to a latent space 2, a sum or max aggregation over the sample dimension of the encoder output, and an unconstrained decoder p: Z +
f(x) p (I Ex(x))
An architecture is a universal approximator of permutation invariant functions if and only if it is sun/max-decomposin with sufficient conditions on the latent space dimension (Wagstaff' al., 2019). Both the Deep Sets and Set Transformer architectures can be written in the form of Equation (1). Concretely, Deep Sets architecture consists of an equivariant encoder made up of feedforward layers, sum or max aggregation, and a decoder also made up of feedforward layers (Zaheer et al., 2017). The Set Transformer encoder consists of multi-attention blocks called inducing point attention blocks (Lee et al., 2019), variant of the original transformer block (Vaswani et al., 2017) modified to handle large set sizes. The aggregation is learned via attention, and the decoder consists of transformer blocks with self-attention.
Both architectures are permutation invariant since they consist of a permutation equivariant encoder and permutation invariant aggregation over the outputs of the encoder. In order maintain permutation invariance of the architectures upon modification, one needs to ensure that changes to the encoder preserve permutation equivariance, and changes to the aggregation maintain permutation
2

---

M. Under review as a conference paper at ICLR 2022
invariance. In the next two sections, discuss how to incorporate normalization layers and residual connections while maintaining permutation invariance overall.
3 NORMALIZATION FOR SET DATA
Incorporating normalization layers into the permutation equivariant encoders requires careful consid- eration, not all normalization layers appropriate to use. We study normalization layers which consist of two operations, standardization and transformation. This setting captures most common normalizations (loffe & Szegedy, 2015; Ba et al., 2016; Ulyanov et al., 2016).
Let a E BxSxD be the activation before the normalization operation, where B is the size of the batch, S is the number of samples in set (the tensor is pero-padded to the largest sample size), and D is the feature dimension. First, the activations are standardized based on a setting L which defines which dimensions utilize separate statistics. For instance, I, {B, S} denotes that each set in batch and each sample in set gets its own mean and standard deviation for standardization, which are repeated over the D dimension so that μy (a), oz (a) R Bx match the original activation tensor for elementwise subtraction and division. A standardization operation can be defined as:
a (1
TL (a)
Next, the standardized activations are transformed through learned parameters which differ only a setting of dimensions M. For instance, MM = {D} denotes that each feature is transformed by a different scale and bias, which shared across the sets in the batch and samples in the sets. Let 7 Mr 3 E BxSxD denote the learned parameters and O represent elementwise multiplication, any transformation operation can be defined as:
aM a о
(3)
We can now summarize wide array of normalization possibilities simply via choice of L and M.
3.1 NORMALIZATION DESIDERATA
We formally introduce the desiderata for a general-purpose normalization layer for prediction on sets. In particular:
The normalization must be permutation equivariant.
(ii) The computation of the statistics for standardization (14r (a) should not be affected
by the presence of sets of varying sizes.
(iii) The normalization should not deteriorate performance due removal of information that is useful for prediction.
Item (ii) is meant to address the concern that real-world set imputs are often not all the same size, c.g. not all point cloud objects are made up of the same number of points, and not all patients will have the same number of cells in blood sample. If the presence of different set sizes forces a choice for application-specife implementation of normalization layer, then such normalization is not general-purpon Item (iii) refers to the fact that the standardization operation in normalization layer incurs loss of information, as activations which were once of varying means and variances across certain dimensions are now forced to be zero mean and unit variance. For real-valued sets in particular, this can greatly limit predictive performance, e.g. the removed means and variances would help in the output prediction. As an example, the mean cell volume of a blood sample can be indicative of patient health, and normalization layer that makes impossible to distinguish between patients' mean cell volumes will restrict performance across wide variety of health outcomes predictions.
3.2 THE ONLY SUITABLE TRANSFORMATION FOR SETS IS ON THE FEATURE DIMENSION.
Only normalization layers adopting a per feature transformation, i.. M = (D), meet the permutation equivariance desiderata (i), as per sample transformations break permutation equivariance and per
3

---

Under review as a conference paper at ICLR 2022
(i) Feature Norm
(ii) Layer Norm
(iii) Set Norm
We can represent normalization layers by the dimensions they perform standardization (blue) and the dimensions they perform transformation (yellow). Green indicates overlap of yellow and blue.
Figure 2:
set transformations cause the prediction output to change when an element's position in the batch changes (see Appendix B for proofs).
Proposition 1, Per feature transformation is the only setting which maintains permutation equivari- ance and prediction behavior which is agnostic batch position.
Thus, to systematically compare normalizations, we only need to consider the settings for L.
3.3 EXISTING NORMALIZATIONS DO NOT MEET ALL DESIDERATA.
We now consider possible settings for standardization: 0. [B) {s}, (DD).BBB)). (B, (S, D}. We first note that 7. = {S} and T, {S, D) destroy permutation equivariance (see Appendix B, for proof). Of the remaining settings, we note that two already exist in the literature. The first, which we call feature norm, is used in the permutation invariant PointNet and PointNet++ architectures for point cloud classification (Qi et al., 2017a:b) and performs a per feature standardization followed by per feature transformation, i.e. L {D), .M (D) (see Figure 2 (i)). The second is layer norm, used in Set Transformer, which applies standardization individually to each set and each sample and transformation separately per feature, i.e. L S), M {D) (see Figure 2(ii)).
Both these normalization layers have drawbacks. Feature norm calculates statistics over the batch and consequently may exhibit batch effects: sets in the batch no longer independent, and performance can be heavily impacted by batch size (Hoffer et al., 2017). Additionally. like in batch norm, in feature norm sets standardized differently at training and test time.' Moreover, in the context of sets of varying sizes, each possible implementation of feature norm (e.g. computing statistics over the batch, averaging statistics computed on each set separately) will lead to each set (or samples in the set) having a different impact in the computation of the statistics. Therefore, the use of feature norm requires careful consideration, especially in applications having as input real-valued sets with wide ranges of both set sizes and first moments. A similar property holds for normalizations which calculate statistics over the batch (B & L) such as L = O-
Layer norm, on the other hand, removes information which may be particularly useful for set prediction tasks. By performing standardization on each sample in each set separately, layer norm maps two samples whose activations differ in only scale and bias to the same output. The inability to distinguish such samples after layer norm can hurt prediction, as we see in Section 5.3.
Proposition 2. Applying layer norm in its most common placement (after linear projection, before non-linearity) removes mean and variance information from each sample.
The use of layer norm can present a representation issue similar to the oversmoothing problem in Graph Convolutional Networks (Li et al., 2018), where nodes (samples) get mapped to the same value as the number of layers in the network increase and neighborhoods of nodes become similar in representation.
'eettrre norm is implemented as batch norm on transposed tensor. Thus, test hatches are standardized by running mean and variance while training batches standardized by their hatch statisties.
4

---

Under review as a conference paper at ICLR 2022
X
weight
muftread
weisht
4
multreedd
nom
arteption
weight
od2ioon
weight
norm
mgom
ReLU
weght
weight
adiition
Diccttion
ReLU
addhon
norm
(a) Non-clean path for Deep Sets
(b) Clean path for Deep Sets
(c) Non- cle-clean path for Set Transformer
(d) Clean path for Set Transformer
Figure 3:
(Non-) clean path visual representation for Deep Sets and Set Transformer. In (c), weight* also part of the attention computation.
3.4 SET NORM
We introduce normalization layer which cxhibits reasonable trade-off over the three desiderata. Our proposed normalization, set norm (SN), is a per set standardization, per feature transformation, Le. L (B), .M (D), Figure 2(ii)):
(N(aasd)
σ
1bb
7d Bds
μbb DD
Ghads ob
S
abad jbb
SN is permutation equivariant (see Appendix B for proof) and, since is per set standardization. it naturally handles sets of different sizes It also avoids batch effects that may arise when using feature norm while resulting in less information loss than layer norm. In fact, the per set standardization of set norm, i.c. L {B), removes the least information of any standardization which acts on each set separately (see Appendix B for proof). It is for this reason that propose norm another per set standardization, e.g. per set per feature.
Proposition 3. Among all standardizations which are applied separately per set, the per set stan- dardization of set norm results in minimal loss of information.
Experimental results (Section 5) confirm our hypothesis that set norm is generally preferable to feature norm and layer norm. However, set norm does remove some information, namely the global mean and variance of each set (as does any standardization with B E .). In the next section, we discuss how alleviate this information loss through the of residual connections.
RESIDUAL CONNECTIONS FOR SET DATA
Residual connections can be incorporated into an equivariant encoder two ways: either adding each sample input to its corresponding output (equivariant residual connection) adding an aggregated function over all the samples (aggregated residual connection).
An equivariant residual connection (ERC) residual connection that acts on every sample. Let be an equivariant function where X = y Rd i.e. f IR Mxd + I Mxd A function which adds single sample equivariant residual connections to any equivariant function is also permutation equivariant:
((xx) f(Tx) xx T + mx(x).
5

---

Under review as a conference paper at ICLR 2022
An aggregated residual connection (ARC) is residual connection that sums permutation -nnuuuattiioonvaiiantt aggregated function of the samples (e.g. sum, mean, max). function 9 which adds aggregated equivariant residual connections to any equivariant function f is also permutation equivariant:
((Tx)= - (Txx + (x.............................. Xs) - fx( + pool(x)......................... Xs) Tg(x).
One major benefit of residual connections is that of addressing vanishing gradients. We show in Appendix F that ERCs and ARCs (mean and max) offer this benefit. While sum aggregation does as well, we see empirically that such networks often explode in their outputs and therefore do not report results on
4.1 THE CLEAN PATH PRINCIPLE
There are many existing arrangements of residual pipelines (He al., 2016b;a; Vaswani et al., 2017; 2018). We hypothesize that the best way to implement residual connections is to leave clean path from input to output without other operations in between; we call this the clean residual path principle, where the residual path is the shortest path from input to output of block (see Figure for illustration). Our hypothesis aligns well with previous literature on non permutation invariant architectures showing how the presence of certain operations between skip connections could yield undesirable effects (He et al., 2016a; Wang et al., 2019a; Xiong et al., 2020) (see Appendix C for more details). Concretely, propose the use of the pipeline in Figure 3(b) over the pipeline in Figure 3()) for Sets, as well as the pipeline in Figure 3(d) over the one in Figure 3(c) for SeiTransformer (He al., 20;b;;;; Klein et al., 2017; Vaswani et al., 2018: Xiong al., 2020) for all possible implementation of residual connections that respect permutation invariance.
We additionally hypothesize further benefit of the clean path principle, i.e. skip connections alleviate information loss due to normalization. While this is true in general, for applications involving images or text (the bulk of the DNN literature) information loss normalization may be less important than specific data invariances. For prediction on real-valued sets like in distribution regression, on the other hand, this information is fundamental for good performance.
4.2 RESIDUALS CAN ALLEVIATE LOSS OF INFORMATION FROM NORMALIZATION
In specific tasks such as distribution regression, moments are often crucial to reach good performance. Applying normalization techniques that rely standardization leads to the loss of information on the first two moments along some dimension. Clean path equivariant and aggregated residual connections will alleviate this issue as they help reintroduce such information.
Consider set norm placed before the nonlinearity in Deep Sets, e.g. f(x) = relu(SN(xW") for single layer. If two sets scaled and translated versions of each other, i.e. y = without residual connections, the network with set norm will map both inputs to the same output, i.c. fx) fly). On the other hand, Deep Sets with a residual connection (both aggregated or cquivariant) will result in different outputs: (x)
then
+ f(x) # g(y). Consider now layer norm in the same placement. If two samples in a set are mapped the same output, an ERC will pass each sample forward, alleviating information loss, while ARCs will only pass the same aggregated function forward. Therefore, ERCs can alleviate information loss to a greater extent than ARCs can. This difference can have large implications for performance Moreover, ARCs could lead to similar representations across sample activations the aggregated activations become larger in magnitude than the single sample ones they are added to. This resembles the concept of oversmoothing for graph neural networks (Li et al., 2018).
some tasks (see Section 5.3).
5 EXPERIMENTS
We run our experiment on tasks with diverse inputs (point cloud, continuous, images) and outputs (regression, classification). On these tasks we compare the original architectures with normalization and residual connection choices described in the previous sections. For all experiments, decoder and aggregation to their original versions, focusing on the effect of the encoder changes. Since the decoder unconstrained network, we expect general architectural principles non-specific
fix the

---

Under review as a conference paper at ICLR 2022
Table 1: Adding set norm and feature norm improves performance of Deep Sets (50 layers), while adding layer norm does not.
Hematocrit (MSE)
Point Cloud (CE)
MNIST Var (MSE)
Normal Var (MSE)
no norm
2 0.0014
3.6090 0.0003
5,5545 + 0.0014
8.0228 0.0017
layer norm
25.8748
0.0018
3.6186
0.0001
5,5651 + 0.0006
7.9512 0.0026
feature norm
25.8749
0.0018
0.9940 0.0045
2.2629 0.2715
0.3965 0.0059
norm
25.8749
0.0018
1.5423 0.0862
0.2578 + 0.0031
5.4131 0.0081
Table 2: The setting of Deep Sets which performs best overall uses set norm with a clean residual path (highlighted gray). First row is the original Deep Sets architecture.
Path none
Residual type
Norm
Hematocrit (MSE)
Point Cloud (CF)
Mnist Var (MSE)
Normal Var (MSE)
none
Rone
25.8791 + 0.0014
3.6090 :: 0.0003
5,5545 + 0.0014
8.0228 = 0.0017
non-clean path
cquivariant
layer norm
19,0063 0.0137
0.5982
0.0064
0.3528 1 0.0063
0.1641 0.0386
fiaature
19.9801 0.0862
0.6541
0.0022
0.3371 + 0.0059
0.4762
0.0832
set norm
19.3146 + 0.0409
0.6055
0.0007
0.3421
+ 0.0022
0.2711 = 0.0813
clean path
equivariant
layer norm
19.5236
0.0132
0.65224 0.0072
0,3529
0.1641
#0.0386
feuture norm
19.3917 + 0.0685
0.7148 + 0.0164
0.3368
+-0.0449
0.1162 = 0.0177
set norm
19.2118
0.0762
0,7096
0.0049
0.3441 00.0036
0.0320
0.0118
cleun path
layer_norm
21,12477
0.0303
0.8656
010099
1,1732 + 0.0259
3.6164
0,13997
feature noom
18,9233 1 0.0638
0.9773
1 0.0288
1.1296 1. 0.0222
1.9861 = 0,8488
set_norm
19.3462
0.0260
D.8885
1,2808
0,8287
0.1818
clean path
layer
21,9485
1.00181
0,9637
10.004
1,3373 1:00114
0.7133
000339
feature_norm
noomm
21.0755
+-0.0400
0.9092 +-0.2488
1.4285 4-0.0174
1.0224 0.0956
set_norm
19,8171
+000266
0.8758 +:000196
1.3798 +0.0162
0.9084 000000
to sets to be applicable, and because of our focus is scaling architectures to large depths, we do not modify the aggregation, which itself is typically a single block.
5.1 ARCHITECTURES
To incorporate normalization layers and residual connections to Deep Sets and Set Transformer, we modify the equivariant encoder to consist of residual blocks (See Figure 3). For the clean path version of Deep Sets, we precede the first residual block with linear layer and no bias and place normalization-rell-weeee
sequence after the final residual block, following (He et al., 2016a). When we use equivariant clean path residuals with set norm, we called the architecture Deep Sets++ or Set Transformer++. The clean path Set Transformer block architecture resembles that of the Pre-LN Transformer (Klein et al., 2017; Vaswani et al., 2018).
5.2 DATASETS & TASKS
We use four main datasets for ablations (Hematocrit, Point Cloud, Mnist Var and Normal Var) and one (CelebA) for validation of the models.
Hematocrit Regression from Blood Cell Cytometry Data (Hematocrit), The prediction of hematocrit (the fraction of blood volume occupied be RBCs) from set of individual RBC mea- surements (volume and hemoglobin) aims to answer an open biological and clinical question: are characteristies of individual cells predictive of their prevalence in the blood? On this task, increases in performance provide exciting scientific signal which can help lead to deeper under- standing of RBC production and regulation.? 2 The dataset consists of 98240 train and 23104 test volume/hemoglobi distribution coupled with their hematocrit levels. We select the first visit for given patient such that each patient only appears once in the dataset, and there is no patient overlap between train and test. We subsample for each distribution to 1,000 cells. See Appendix A for complete details about the dataset.
Point Cloud Classification (Point Cloud). Following Zaheer et al. (2017); Lee al. (2019), we use the ModelNet40 dataset (Wu et al., 2015) (9840 train and 2468 test clouds), randomly sample 1,000 points per set, and standardize each object to have mean zero and unit variance. We report ablation results as entropy loss to facilitate the readability of the tables, i.e. lower is better.
2 As an analogy, imagine we learned for the first time that the distribution of individual leaf colors predicted the number of leaves on tree. This insight would provide strong evidence for scientists to look for an underlying mechanism for this relationship, e.g. understanding how deciduous trees change in the fall.
7

---

Under review as a conference paper at ICLR 2022
Table 3: The setting of Set Transformer which performs best overall uses set norm with a clean residual path (highlighted gray). First row is the original Set Transformer.
Path
Residual type
Norm
Hematocris (MSE)
Point Cloud (CF)
MNIST Var (MSE)
Normal Var (MSE)
Improved tasks
now-eleaa puth equivariani
18.7982 +: 0.0086
0.9217 0.0119
6,2663 10.3307
0,0015 0.0001
noo clean path equivariant
layer norm
19,0904 = 0,1003
0.0052
10063 1,009
0.0597 0.0137
174
feature norm
19,4968 1: 0.1442
0.8251
10.0025
0.4043 1. 0.0078
0.0884 1: 0.0092
2/4
set sorm
19.0521 =0.0288
1.9167
1 0,4880
0.4064 1: 0.0147
0.0627 1 0.0217
1/1
clean path
equisariont
layer
18, 08.85774
0.0583
0.6656
0.0148
0.6383 + 0.0020
0,0083
0.0015
2/4
feanure norm
19,1967
0.0330
0.0141
0.7946 40:0065
0.0224
2/4
set norm
18.6883
1: 0.0238
0.6280
0.0098
0.7921 1: 0.0006
0.006S 1: 0.000S
3/4
clean path
mean
layer poor
19,8620 = 0.09963
1:3745
0.0079
1,2562 + D.061
0,1123 # E.0155
2/4
feature_norm
19.3714
0,1143
18169
0.1407
setutom
19.6945
0.1066
0.811 0.0453
1.6273-4 + 0.0335
0.0512 0.0179
2/4
Table 4: Deep Sets++ and Set Transformer++ outperform their Deep Sets and Set Transformer counterparts at 50 and 16 layers respectively.
Hematocrit (MSE)
Point Cloud (CBs
MNIST Var (MSE)
Normal Var (MSE)
CelebA (accuracy)
Seep (3 layers
19.1257
0.0361
0.7357 + 0.0119
0,4520 + 0,0111
0,0417
0.0074
0.3808 0.0016
Deep Sets (50 layens)
25.8791 1 0.0014
3.6090 1 0.0003
5,5545
0.0014
8,0228
10.0017
0.1005
0.0000
DeepSets ++ (3 layers)
19.5882
0.0555
0.6703 + 0.0093
0,5895
0.0114
0.0707
0.0326
0,5730
0.0016
Deep Sets++ (50 layers)
19,2118
0.0762
0,7096 1 0.0049
0.3441
10.00/36
0.0118 0.0320
0.0134 0.5763
Set Transformer (2 layers)
18.8750
0.0058
0.7487 + 0.0381
0.6151
0.0072
0.0016
0.0005
0,1292
0.0012
Set Transformer (16 layens)
18.7436
0.0148
0.9217 + 0.0119
6.2663
0.33377
0,0015
0.0001
0.4570
0.0540
Sei + layers)
18.9223
0.0273
0.6366 1 0.0004
1.1525 0,0158
0,0050
0.0008
0.6533
0.0012
Set Transfomerr++ (16 layers)
18.7258
0.0342
0.6280 + 0.0098
0.7921 = 0.0006
0.0068 = 0.0008
0.6587 : 0.0016
Variance Prediction, Image Data (MNIST Var). We implement empirical variance regression on MNIST digits as proxy for real-world tasks with sets of images, e.g. prediction on blood smear histopathology slides. We sample 10 images uniformly from the training set and use the empirical variance of the digits as a label. Test set and training set images size is 50,000 sets, and test set size is 1,000 sets. We represent each image as ID vector.
non-overlapping. Training set
Empirical Variance Prediction, Real Data (Normal Var). Each set is a collection of samples (sample size 1000) from a univariate normal distribution. Means are drawn uniformly in [-10, 10], and variances are drawn uniformly in [0, 10]. The target for each set is the empirical variance of the samples (regression task) in the set. Training set size is 10,000 sets, and test set size is 1,000 sets.
Supervised set anomaly detection (CelebA). Following Zaheer et al. (2017), we construct 18,000 sets of 10 CelebA images (each 64x66) where one does not belong. For each set, we randomly select two attributes out of the 40 total and select 9 images which share both attributes and one which does not exhibit either. Train and test do not contain the same individuals.
Results are reported in Mean Squared Error (MSE) on the test set for the regression experiments and in cross entropy loss (CE) for point cloud classification, averaged over three seeds. We fix all hyperparameters, including epochs, and use the model at the end of training for evaluation. We notice no signs of overfitting from the loss curves. For further details, see Appendix D.
5.3 MAIN RESULTS
Set norm has better performances than feature norm and layer norm. We compare how feature norm, layer norm, and norm affect performance of deep Deep Sets model (50 encoder layers) in Table 1. In general, set norm and feature norm both improve results compared to no normalization while layer norm often does not. We hypothesize that the problem primarily information loss and explore in Appendix E.4 why layer norm may be suitable for categorical text data and not for real-valued sets. A similar phenomenon seems to exist for set norm on Normal Var, but we show next that we can ameliorate the effect with residual connections. Feature norm greatly improves performance on Point Cloud, which we speculate is due to specific invariances that hold for clouds of points. We provide an experiment in Appendix E with additional evidence to support this hypothesis. While feature norm performs well in Table 1 in particular, set norm tends to perform better overall once other considerations such residual connections are taken into account (Table 2, Table 3). Additionally, set norm does not require additional considerations under sets of varying sizes. This makes it the best choice for general purpose architecture.
8

---

Under review as a conference paper at ICLR 2022
Equivariant residual connections following the clean path principle have best performance overall. Experiments in Table and Table 3 confirm that clean path ERCs pipelines generally yield the best performance across set tasks, with ERCs performing better than ARCs both for Deep Sets (Table 2) and Set Transformer (Table 3). The effect of clean path ERCs is especially evident on Deep Sets with set norm on Normal Var: the clean path residual (3e-2) especially improves performance over the non-clean path residual (le-1) and no residual (5.4), testament to the ability of a clean path residual pipeline to make up for the loss of global scale information by the set norm operation. Clean path ARCs perform worse than both non-clean and clean path ERCs. We hypothesize that this may be due to a problem similar to oversmoothing. Note that table 3 does not have max aggregation results as at least one seed failed for all experiments due to problems in training.
Deep Sets++ and Set Transformer++ outperform existing architectures. Table shows that deep DS++ and ST++ generally yield better or comparable performance to deep and shallow Deep Sets and Set Transformer. We note that depth plays a role in the improved performance, as shallow +S++ and ST++ do not improve results in general. When deep, Deep Sets++ consistently improves over Deep Sets, sometimes with orders of magnitude improvement in test loss. Set Tranformer++ improves performance Point Cloud and MNIST Var and reaches comparable performance on Hematocrit and Normal Var. In Appendix E, we show on an official point cloud benchmark repository (Goyal al., 2021a) that DS++ and ST++ without any modifications outperform versions of Deep Sets and Set Transformer tailored for point cloud classification (87% vs 86% accuracy). On the unseen set anomaly task (CelebA), both DS++ and ST++ perform significantly better than their counterparts. On Hematocrit, results surpass a model that uses the clinical baseline as well as prediction from moments computed on the distribution. The best performing model is ST++ with 18.73 MSE, 28% error reduction over the clinical bascline (25.85 MSE).
6 RELATED WORK
There exist several permutation invariant neural architectures in the literature, including Deep Sets (Zaheer et al., 2017) and Set Transformer (Lee et al., 2019) for general-
sets and PointNet (Qi al., 2017a) and PointNet++ (Qi et al., 2017b) specialized for point cloud data. Previous efforts to design residual connections (He et al., 201 Veit et al., 2016; Yao et al., 2020) or normalization layers (loffe& & Szegedy, 2015; Ba et al., 2016; Santurkar et al., 2018: Ghorbani et al., 2019; Luo et al., 2019; Xiong et al., 2020; Cai et al., 2021) have often been motivated by particular applications; our work is the first to design residual connections and normalization layers for prediction on sets in general. Efforts to build deep neural architectures exist in computer vision (He et al., 2016b), natural language (Wang et al., 2019b), and graphs (Li et al., 2019; Chen et al., 2020; Zhao & Akoglu, 2019). While graph convolutional networks also encode permutation eguivariance, in this paper address distributional real-valued data as well as sets of high-dimensional inputs where no external information about graph structure is available.
7 CONCLUSION
We introduce set norm (SN) and equivariant residual connections (ERCs) and illustrate the benefits of both in comparison to other choices of normalization layers and residual pipelines. Using set norm and equivariant residual connections, we developed Deep Sets++ and Set Transformer++, deep permutation invariant architectures which improve upon the current state-of-the-ar models. Such architectures are general-purpose architectures and the first permutation invariant architectures of their depth that show good performance on a variety of tasks without task-specific architectural changes. Interestingly, between Deep Sets++ and Set Transformer++, there is not best choice in architecture across tasks, so the choice for which to use may often come down to computational resources and time. Set Transformer++ with 16 layers has more expensive computational and memory requirements than Deep Sets++ with 50 layers, so Deep Sets++ may be preferred under limited resources. Lastly, we introduced new open-source dataset, FBCw-RBC, which provides a real-world application of permutation invariant prediction in clinical science. With data from over 100,000 patients, including an external clinical measurement (i.e. hematocrit), Flow-RBC's is uniquely suited among single-cell datasets to be a benchmark for deep predictive modeling on sets. We believe our new models and dataset have the potential to motivate future work and applications of prediction sets.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jimmy Ba, J. Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, (bbs/1607.06450 2016.
Tianle Cai, Shengjie Luo, Keyulu Xu, Di He, Yan Liu, and Liwei Wang. Graphnorm: A principled
approach to accelerating graph neural network training. In ICML, 2021.
M. Chen, Orhan Firat, Ankur Bapna, M. Johnson, Wolfgang Macherey, George F. Foster, Llion Jones, Niki Parmar, M. Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances in neural machine translation. In ACI., 2018
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning, pp. 1725-1735, PMLR, 2020.
B. Ghorbani, Shankar Krishman, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In ICML, 2019.
Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with simple and effective baseline. In ICML, 2021a.
Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with a simple and effective baseline. International Conference on Machine Learning, 2021b.
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: A survey. IEEE transactions on pattern analysis and machine intelligence, PP, 2020.
Neil Harris, Jolanta Kunicka, and Alexander Kratz. The advia 2120 hematology system: flow cytometry-based analysis blood and body fluids in the routine hematology laboratory, Laboratory Hematology, 11(1):47-61 61, 2005.
Kaiming He, Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. ArXiv, abs//6003.500000 201 6a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pp. 770-778, 2016b.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- 2017.
tion gap in large batch training of neural networks. ArXiv,
S. loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ArXiv, 502.033677
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. Opennmt: Open-source toolkit for neural machine translation. ArXiv, abs/1 7007028100 2017.
David Lähhemann, Johannes Köster, Ewa Szezurek, Davis J. McCarthy, Stephanie C. Hicks, Mark D. Robinson, Catalina A. Vallejos, Kieran R. Campbell, Niko Beerenwinkel, Ahmed Mahfouz, Luca Pinello, Pavel Skums, Alexandros Stamatakis, Camille Stephan- Otto Attolini, Samuel Aparicio, Jasmijn A. Baaijens, Marleen Balvert, Buys de Barbanson, Antonio Cappuccio, Giacomo Corleone, Bas E. Dutilh, Maria Florescu, Victor Guryev, Rens Holmer, Katharina Jahn, Thamar Jessurun Lobo, Emma M. Keizer, Indu Khatri, Szymon M. Kietbasa, Jan O. Korbel, Alexey M. Kozlov, Tzu-Hao Kuo, Boudewijn P. F. Lelieveldt, Ion I. Māndoiu, John C. Marioni, Tobias Marschall, Felix Mölder, Amir Niknejad, Lukasz Raczkowski, Marcel J. T. Reinders, Jeroen de Ridder, Saliba, Antonios Somarakis, Oliver Stegle, Fabian Joachim Theis, Huan Yang. Alex Zelikovsky, Alice Carolyn McHardy, Benjamin J. Raphael, Sohrab P. Shah, and Alexander Schönhuth. Eleven grand challenges in single-cell data science. Genome Biology, 21, 2020.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Y. Teh. Set transformer: A framework for attention-based permutation-invartee neural networks. In ICML. 2019.
10

---

Under review as a conference paper at ICLR 2022
Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go deep as cnns? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9267-9276, 2019.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for
semi-supervised learning. In Thirly-Second AAAI conference artificial intelligence, 2018.
Jiajia Liu, Zhiwei Fan, Weiling Zhao, and Xiaobo Zhou. Machine intelligence in single-cell data analysis: Advances and new challenges. Frontiers in Genetics, 12, 2021.
Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. ArXiv, abs/ 8809.0086, 2019.
Richard A McPherson, MD Msc, and Matthew R Pincus. Henry's clinical diagnosis and management by laboratory methods E-book. Elsevier Health Sciences, 2021.
C. Qi. Hao Su, Kaichun Mo, and L. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77-85, 2017a.
C. Qi, L. Yi, Hao Su, and Guibas. +ointnet++: Deep hierarchical feature learning on point sets in metric space. In NIPS, 2017b.
Aviv Regev, Sarah A Teichmann, Eric S Lander, Ido Amit, Christophe Benoist, Ewan Birney, Bernd Bodenmiller, Peter Campbell, Piero Carninci, Menna Clatworthy, et al. Science forum: the human cell atlas. elife, 6:e27041, 2017.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? In NeurlPS, 2018
DH Tycko, MH Metz, EA Epstein, and Grinbaum. Flow-cytometric light scattering measurement of red blood cell volume and hemoglobin concentration. Applied optics, 24(9):1355-1365 1985.
Dmitry Ulyanov, Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. ArXiv, abs/1607.0800 2016.
Ashish Vaswani. Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/1706.03362,, 2017.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, François Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam M. Shazeer, and Jakob Uszkoreit. Tensor2tensor for neural machine translation. In AMTA, 2018.
Andreas Veit, Michael Wilber, and Serge J. Belongic. Residual networks behave like ensembles of relatively shallow networks. In NIPS, 2016.
E. Wagstaff, Fuchs, Martin Engelcke, Posner, and Michael A. Osborne. On the limitations of representing functions sets. In ICML, 2019.
Qiang Wang. Bei Li. Tong Xiao, Jingbo Zhu, Changliang Li. Derek Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In ACI., 2019a.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. arXiv preprint 1xXv:1906.01789 2019b.
Zhirong Wu, Shuran Song, A. Khosla, F. Yu, Linguang Zhang, Xiaoou Tang, and J. Xiao. 3d shapenets: A deep representation for volumetric shapes. 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1912-1920, 2015.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie- Yan Liu. On layer normalization in the transformer architecture. ArXiv, abs/2002.04745.. 2020.
11

---

Under review as a conference paper at ICLR 2022
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W. Mahoney. Pyhessian: Neural networks through the lens the hessian. 2020 IEEE International Conference on Big Data (Big Data), pp. 581-590, 2020.
Guocheng Yuan, Long Cai, Michael Elowitz, Tariq Enver, Guoping Fan, Guoji Guo, Rafael A. Irizarry, Peter V. Kharchenko, Junhyong Kim, Stuart H. Orkin, John Quackenbush, Assieh Saa- datpour, Timm Schroeder, Ramesh A. Shivdasani, and Itay Tirosh. Challenges and emerging directions in single-cell analysis. Genome Biology, 18, 2017.
M. Zaheer, Satwik Kottur, Siamak Ravanbakhsh, B. Póczos, R. Salakhutdinov, and Alex Smola. Deep sets. In NeurIPS, 2017.
Lingxiao Zhao and Leman Akoglu. Pairnrrm: Tackling oversmoothing in gnns. In International Conference Learning Representations, 2019.
12


Reviews:
[{"review_id": "xX2ZcOPkPMR", "reviewer": "Reviewer_Dj6B", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I appreciate the effort to tackle the challenge of designing deep networks. Indeed,  I feel that it is an important research question. \nOverall the paper is well written and easy to follow. The formulation of the method is presented in a general form which allows the reader to easily understand the core details. \nI appreciate the effort to introduce an additional benchmark, which will be publicly available to the community.\n\nHowever, I do have some concerns regarding the current version of the paper, detailed next.\n\nThe authors state that “existing architectures struggle to perform well when they are deep”.  However, I am not convinced that the claim is True as existing architectures do incorporate normalization layers. Moreover, I think the authors should consider refining the original question to  two separated questions:\n\n1) Do normalization and skip connections help networks in learning ? regardless of the choice of how many layers are there.\n2) Is a deep architecture a good inductive bias for some tasks involving sets?\n\nFigure 1 does not answer the first question as it does not apply normalization+skip layers to standard architectures.  It also does not fully answer the second question as it does not evaluate on different depths of networks. To my understanding, the main implication from Figure 1 is that with 50 layers and without normalization the learning performs poorly, which is not novel.\n\nMoreover, existing architectures already incorporate normalization layers. For example, in the pointnet [1] paper, which is a similar architecture to deepsets,  they use normalization layers (feature normalization). Do they improve/stable/disimprove with respect to network depth? Is there a difference between set normalization to feature normalization? Moreover, the results in table 2 do not show a clear advantage for the set norm over the feature norm.\n\n*Relation to graphs*\n\nThe current paper focuses on sets. Note that there is extensive literature on the more general settings of deep permutation invariant networks, which is the case of graph neural networks. For example, what is the relation between the proposed set norm and the one suggested in [2]? I would expect a more detailed discussion about this challenge in graphs versus sets. \n\n*Residual connections*\n\nIncorporating residual connections in set networks is not novel. It seems that some prior works have used it, as the adaptation of it to set networks seems natural. For example, see implementation details in the supplementary material of [3]. Few alternatives to clean path residual connections are discussed. Do the authors know of such works that follow these alternatives? What are the advantages of these alternatives? Are there any tasks which they improve? If the answer is no, then it is not clear why are they discussed.\n\n*Requirements for a design of a normalization layer*\n\nThe authors identify some requirements for normalization layers of permutation equivariant networks.  What does “work well” in the context of varying set sizes mean?  What does essential information mean? It feels like these requirements are too general and should be refined. Moreover, it seems that some discussion relating to test time versus train time equivariance is missing. Is it true that the normalization choices that the authors show that fail (such as L = {B} )  in satisfying equivariance is only at training time? As in test time, some constant inferred from the batch statistics is used. If that is the case, then I expect to see a discussion on why satisfying the equivariant property in training time is important.\n\n*Results*\n\nFor classification tasks, it would be more informative to report accuracy. For regression tasks, the units of the measurement should be reported and explained as well.\n\n[1] Qi, Charles R., et al. Pointnet: Deep learning on point sets for 3d classification and segmentation.\n\n[2] Zhao, Lingxiao, and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns.\n\n[3]  Mescheder, Lars, et al. Occupancy networks: Learning 3d reconstruction in function space.\n", "summary_of_the_paper": "The paper tackles the problem of designing deep networks that operate on sets. The challenge is addressed by normalization layers and skip connections. The authors formulate some design choices and advocate the choice of set normalization and clean residual connections. The layers are incorporated with two different choices of architectures and are evaluated on several sets learning benchmarks. In addition, a new benchmark is introduced.\n", "summary_of_the_review": "* Well written and easy to follow\n* The list of requirements identified by the authors from normalization equivariant layers needs a major refinement. \n* Lack of novelty and the benefit of the set norm is unclear\n* Missing discussion on the relation to graph neural networks\n* The experiments do not test properly the effect of depth on learning\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "q850lD4eEI", "reviewer": "Reviewer_Qf1Y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n**S1) Overall well-written and clear**\n\nThe motivation for developing set-norm is clear, and the comparison to feature norm and layer norm is also clear. The motivation for the \n\"clean path\" principle for residual connections is presented well and intuitive.\n\n**S2) Interesting new dataset**\n\nAdmittedly, as I have no medical background, I am not a good adjudicator on the \"value\" of the new Flow-RBC dataset. However, taking the authors at their word, it seems to be much larger than existing similar datasets and demonstrates a useful real-world test case for set-based neural networks, beyond more commonly seen point cloud data and synthetic toy datasets (e.g., sum of MNIST digits).\n\n## Weaknesses\n\n**W1) What is special about an equivariant residual connection? And what is the relationship between the \"clean path principle\" and equivariance?**\n\nIt is unclear to me how an \"equivariant residual connection\" differs from a regular residual connection. To me, it seems like the discussion of the \"clean path principle\" is independent of equivariance and applies generally to all residual connections. If so, this needs to be made clearer in the paper.\n\n**W2) Could use more empirical comparisons**\n\nIn particular, both of the original Deep Sets and Set Transformers paper do set anomaly detection on CelebA. It would be helpful to see a comparison of Deep Sets++ and Set Transformers++ on this task\n\n**W3) Error bars on experimental results**\n\nWere the experiments run once, or many times? Could you provide error bars for the experiments?\n\n## Clarity Issues\n\n- Figure 2: should the \"F\" axis labels be \"D\"?\n\n- Please add units to all of the tables (or at least the table captions). I know that the units (MSE) are written in the main text, but they can be hard to find when looking just at the tables.", "summary_of_the_paper": "The paper has four main contributions:\n\n1. introduces the _set norm_ normalization layer in neural network models for set data, as opposed to feature normalization (aka. batch-norm) and layer normalization layers;\n2. provides intuition behind a \"cleaner\" implementation of residual connections;\n3. implements set norm and the cleaner residual connections in modified versions of existing Deep Sets and Set Transformers models;\n4. introduces a new dataset called Flow-RBC with over 100,000 examples. Each \"example\" consists of a $(X,y)$ pair, where $X=\\\\{x_1, \\dotsc, x_{1000}\\\\}$ is a set of measurements on 1000 red-blood cells from a patient, and $y$ is a hematocrit level label. Each $x_i$ consists of both volume and hemoglobin mass measurements. This dataset is significantly larger than similar existing datasets.", "summary_of_the_review": "The paper is well-written and provides solid empirical justification for specific choices of normalization (set norm) and residual connection design (\"clean path principle\") for neural networks that operate on sets. While \"set norm\" is somewhat novel, I am unsure about whether the \"clean path principle\" applies specifically to these set-based neural networks, or whether there is something specific about that design for equivariant networks. Overall, a solid empirical paper, but I hesitate to say that it is truly \"novel.\" That said, I am open to increasing my score based on authors' replies.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "otIRQ2ze98m", "reviewer": "Reviewer_eZF1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main weaknesses are: lack of statistical analysis of the experimental evaluation and limited novelty of the pre-norm changes.\n\nLack of statistical analysis of the experimental evaluation: the authors use relatively small datasets which seem to exhibit high variance. Single run results presented by the authors are inconclusive and cannot be used to support claimed contributions. The authors could use bootstrapping or train multiple models with different random seed and report mean and std for all baseline and candidate models.\n\nLimited novelty of the pre-norm changes: pre-norm ResNet and pre-norm Transformer are very common, the proposed addition of pre-norm block to DeepSet and SetTransformer does not seem significant.\n\nAnother weakness is the usage of synthetic datasets. Instead of synthetic datasets for variance prediction the authors could evaluate their changes on established benchmarks for predictions on sets like object detection (e.g. COCO dataset) and trajectory prediction (e.g. Argoverse dataset). In object detection models which operate on sets of input features and predict sets have been gaining popularity since DETR [1], whereas in trajectory prediction VectorNet [2] is an example of a regression model with set inputs. Both are very similar to SetTransformer, employ layer normalization, residual connections and could potentially benefit from Set Norm and pre-norm Transformer.\n\nNotes to authors:\n- each dataset description in section 6 needs a clear definition of task solved (classification/regression), size of train/val splits, loss functions used, and metrics used to evaluate\n- figure 1 lacks x and y labels for each subfigure\n- figure 3 would benefit from short subcaption for each subfigure\n- all tables lack column metric definitions. It is not clear what the presented numbers mean, which are classification errors (or accuracy?) and which are mean squared errors\n- I could not understand Appendix A, the description of Flow-RBC dataset. An explanation in layman's terms would be very helpful. More formally, a description of what input set elements and targets are in the dataset is required.\n\n[1] Carion et al. End-to-end object detection with transformers\n[2] Gao et al. Vectornet: Encoding hd maps and agent dynamics from vectorized representation", "summary_of_the_paper": "The paper aims to improve training of deep neural networks applied to sets. The authors investigate permutation equivariant normalizations and residual connections in DeepSet and SetTransformer models, and propose a modification of layer norm which calculates standardization statistics over both features and all set elements. They also propose a modification to DeepSet and SetTransformer models inspired from pre-norm ResNet and pre-norm Transformer. The modifications are experimentally tested on a pointcloud classification dataset, synthetic variance regression, and a new dataset for set prediction contributed by the authors.", "summary_of_the_review": "The contributions of the draft are empirical and the experimental evaluation lacks an analysis of statistical significance, so the contributions cannot be evaluated conclusively. At least one contribution has questionable novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cG3bTwvlS5n", "reviewer": "Reviewer_ifHM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ I enjoyed the discussion on design decisions around normalization layers and agree that set norm is better than what was used in the Set Transformer paper because it discards less information. Furthermore, this intuition is confirmed by experiments in tables 2~4, which show that set norm improves performance.\n+ The discussion about having a clean residual path is interesting, and the paper draws a compelling parallel to design choices in ResNet models.\n\nWeaknesses\n- None of the results have error bounds, so it's hard to tell whether the differences are statistically significant. This problem is exacerbated because loss scales vary widely between tasks (10^1 Hematocrit, 10^-2~-4 Normal Var). Looking at the learning curves in Figure 1, the vanilla and ++ versions of some networks seem to have similar performance on a few tasks.\n- The paper increases the number of layers for both architectures, and this is never addressed in the experiments. For example, what is the performance of Set Transformer++ with 2 layers, compared to the vanilla version?\n\nMinor comments\n- Figure 1: the caption says that the figures have (grey, orange, blue) lines, but the colors seem closer to (pink, orange, green). This was slightly confusing.\n- The rightmost two columns of Table 4 were hard to understand at first. Maybe it's better to have two rows with the performance numbers for \"Original\" and \"No Norm\"?\n- I think Table 5 should be moved to the main paper, to show that permutation-invariant networks compare favorably to traditional approaches in this real-world setting.", "summary_of_the_paper": "This paper presents improved versions of two standard premutation-invariant networks: Deep Sets++ and Set Transformer++. They make two critical design decisions: set normalization and clean residual paths. They also introduce Flow-RBC, a new benchmark for permutation-invariant prediction. Finally, the paper evaluates several variants on several benchmarks, including Flow-RBC.", "summary_of_the_review": "This paper proposes interesting design choices for permutation-invariant networks, but I cannot tell whether the empirical results are statistically significant. Additionally, the paper does not show the effect of their components on the original shallow networks. I'd be happy to increase my score if these issues are addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission focuses on a set norm normalization layer for neural network models, which stands in contrast to a batch norm.  The majority of the reviewers felt that this submission is not suitable for publication at ICLR in its current form.  These concerns remained after the post-rebuttal discussion.  Quoting from the reviewer discussion, the following points remained as significant concerns:


1. lack of novelty. normalization layers have been used previously in other sets architectures. The systematic approach for normalization in the current paper is nice but I am not sure how valuable it is. There exists already extensive literature on normalization layers for graphs (a generalization of sets).
2. lack of motivation for deep networks. The main claim in the paper (see the introduction and figure 1) is that 50 layers should perform better and since it does not seem to be the case in figure 1, it requires studying normalization layers. I am not sure it is a well-established claim. What are the assumptions leading to the conclusion that deep architectures should perform better on the task considered in figure 1? I am also concerned that normalization layers have a major effect on improving "not so deep" networks with 3-10 layers and not only the extreme 50 layers case, making the comparison in the paper between only 3 and 50 layers not enough for telling the full story. Thus evaluation on more depths is required.

On the balance, the paper does not meet the threshold for acceptance in this round of peer review.
Decision: Reject