Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper has four main contributions:

1. introduces the _set norm_ normalization layer in neural network models for set data, as opposed to feature normalization (aka. batch-norm) and layer normalization layers;
2. provides intuition behind a "cleaner" implementation of residual connections;
3. implements set norm and the cleaner residual connections in modified versions of existing Deep Sets and Set Transformers models;
4. introduces a new dataset called Flow-RBC with over 100,000 examples. Each "example" consists of a $(X,y)$ pair, where $X=\\{x_1, \dotsc, x_{1000}\\}$ is a set of measurements on 1000 red-blood cells from a patient, and $y$ is a hematocrit level label. Each $x_i$ consists of both volume and hemoglobin mass measurements. This dataset is significantly larger than similar existing datasets.
main_review: ## Strengths

**S1) Overall well-written and clear**

The motivation for developing set-norm is clear, and the comparison to feature norm and layer norm is also clear. The motivation for the 
"clean path" principle for residual connections is presented well and intuitive.

**S2) Interesting new dataset**

Admittedly, as I have no medical background, I am not a good adjudicator on the "value" of the new Flow-RBC dataset. However, taking the authors at their word, it seems to be much larger than existing similar datasets and demonstrates a useful real-world test case for set-based neural networks, beyond more commonly seen point cloud data and synthetic toy datasets (e.g., sum of MNIST digits).

## Weaknesses

**W1) What is special about an equivariant residual connection? And what is the relationship between the "clean path principle" and equivariance?**

It is unclear to me how an "equivariant residual connection" differs from a regular residual connection. To me, it seems like the discussion of the "clean path principle" is independent of equivariance and applies generally to all residual connections. If so, this needs to be made clearer in the paper.

**W2) Could use more empirical comparisons**

In particular, both of the original Deep Sets and Set Transformers paper do set anomaly detection on CelebA. It would be helpful to see a comparison of Deep Sets++ and Set Transformers++ on this task

**W3) Error bars on experimental results**

Were the experiments run once, or many times? Could you provide error bars for the experiments?

## Clarity Issues

- Figure 2: should the "F" axis labels be "D"?

- Please add units to all of the tables (or at least the table captions). I know that the units (MSE) are written in the main text, but they can be hard to find when looking just at the tables.
summary_of_the_review: The paper is well-written and provides solid empirical justification for specific choices of normalization (set norm) and residual connection design ("clean path principle") for neural networks that operate on sets. While "set norm" is somewhat novel, I am unsure about whether the "clean path principle" applies specifically to these set-based neural networks, or whether there is something specific about that design for equivariant networks. Overall, a solid empirical paper, but I hesitate to say that it is truly "novel." That said, I am open to increasing my score based on authors' replies.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper aims to improve training of deep neural networks applied to sets. The authors investigate permutation equivariant normalizations and residual connections in DeepSet and SetTransformer models, and propose a modification of layer norm which calculates standardization statistics over both features and all set elements. They also propose a modification to DeepSet and SetTransformer models inspired from pre-norm ResNet and pre-norm Transformer. The modifications are experimentally tested on a pointcloud classification dataset, synthetic variance regression, and a new dataset for set prediction contributed by the authors.
main_review: The main weaknesses are: lack of statistical analysis of the experimental evaluation and limited novelty of the pre-norm changes.

Lack of statistical analysis of the experimental evaluation: the authors use relatively small datasets which seem to exhibit high variance. Single run results presented by the authors are inconclusive and cannot be used to support claimed contributions. The authors could use bootstrapping or train multiple models with different random seed and report mean and std for all baseline and candidate models.

Limited novelty of the pre-norm changes: pre-norm ResNet and pre-norm Transformer are very common, the proposed addition of pre-norm block to DeepSet and SetTransformer does not seem significant.

Another weakness is the usage of synthetic datasets. Instead of synthetic datasets for variance prediction the authors could evaluate their changes on established benchmarks for predictions on sets like object detection (e.g. COCO dataset) and trajectory prediction (e.g. Argoverse dataset). In object detection models which operate on sets of input features and predict sets have been gaining popularity since DETR [1], whereas in trajectory prediction VectorNet [2] is an example of a regression model with set inputs. Both are very similar to SetTransformer, employ layer normalization, residual connections and could potentially benefit from Set Norm and pre-norm Transformer.

Notes to authors:
- each dataset description in section 6 needs a clear definition of task solved (classification/regression), size of train/val splits, loss functions used, and metrics used to evaluate
- figure 1 lacks x and y labels for each subfigure
- figure 3 would benefit from short subcaption for each subfigure
- all tables lack column metric definitions. It is not clear what the presented numbers mean, which are classification errors (or accuracy?) and which are mean squared errors
- I could not understand Appendix A, the description of Flow-RBC dataset. An explanation in layman's terms would be very helpful. More formally, a description of what input set elements and targets are in the dataset is required.

[1] Carion et al. End-to-end object detection with transformers
[2] Gao et al. Vectornet: Encoding hd maps and agent dynamics from vectorized representation
summary_of_the_review: The contributions of the draft are empirical and the experimental evaluation lacks an analysis of statistical significance, so the contributions cannot be evaluated conclusively. At least one contribution has questionable novelty.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents improved versions of two standard premutation-invariant networks: Deep Sets++ and Set Transformer++. They make two critical design decisions: set normalization and clean residual paths. They also introduce Flow-RBC, a new benchmark for permutation-invariant prediction. Finally, the paper evaluates several variants on several benchmarks, including Flow-RBC.
main_review: Strengths
+ I enjoyed the discussion on design decisions around normalization layers and agree that set norm is better than what was used in the Set Transformer paper because it discards less information. Furthermore, this intuition is confirmed by experiments in tables 2~4, which show that set norm improves performance.
+ The discussion about having a clean residual path is interesting, and the paper draws a compelling parallel to design choices in ResNet models.

Weaknesses
- None of the results have error bounds, so it's hard to tell whether the differences are statistically significant. This problem is exacerbated because loss scales vary widely between tasks (10^1 Hematocrit, 10^-2~-4 Normal Var). Looking at the learning curves in Figure 1, the vanilla and ++ versions of some networks seem to have similar performance on a few tasks.
- The paper increases the number of layers for both architectures, and this is never addressed in the experiments. For example, what is the performance of Set Transformer++ with 2 layers, compared to the vanilla version?

Minor comments
- Figure 1: the caption says that the figures have (grey, orange, blue) lines, but the colors seem closer to (pink, orange, green). This was slightly confusing.
- The rightmost two columns of Table 4 were hard to understand at first. Maybe it's better to have two rows with the performance numbers for "Original" and "No Norm"?
- I think Table 5 should be moved to the main paper, to show that permutation-invariant networks compare favorably to traditional approaches in this real-world setting.
summary_of_the_review: This paper proposes interesting design choices for permutation-invariant networks, but I cannot tell whether the empirical results are statistically significant. Additionally, the paper does not show the effect of their components on the original shallow networks. I'd be happy to increase my score if these issues are addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the problem of designing deep networks that operate on sets. The challenge is addressed by normalization layers and skip connections. The authors formulate some design choices and advocate the choice of set normalization and clean residual connections. The layers are incorporated with two different choices of architectures and are evaluated on several sets learning benchmarks. In addition, a new benchmark is introduced.

main_review: I appreciate the effort to tackle the challenge of designing deep networks. Indeed,  I feel that it is an important research question. 
Overall the paper is well written and easy to follow. The formulation of the method is presented in a general form which allows the reader to easily understand the core details. 
I appreciate the effort to introduce an additional benchmark, which will be publicly available to the community.

However, I do have some concerns regarding the current version of the paper, detailed next.

The authors state that “existing architectures struggle to perform well when they are deep”.  However, I am not convinced that the claim is True as existing architectures do incorporate normalization layers. Moreover, I think the authors should consider refining the original question to  two separated questions:

1) Do normalization and skip connections help networks in learning ? regardless of the choice of how many layers are there.
2) Is a deep architecture a good inductive bias for some tasks involving sets?

Figure 1 does not answer the first question as it does not apply normalization+skip layers to standard architectures.  It also does not fully answer the second question as it does not evaluate on different depths of networks. To my understanding, the main implication from Figure 1 is that with 50 layers and without normalization the learning performs poorly, which is not novel.

Moreover, existing architectures already incorporate normalization layers. For example, in the pointnet [1] paper, which is a similar architecture to deepsets,  they use normalization layers (feature normalization). Do they improve/stable/disimprove with respect to network depth? Is there a difference between set normalization to feature normalization? Moreover, the results in table 2 do not show a clear advantage for the set norm over the feature norm.

*Relation to graphs*

The current paper focuses on sets. Note that there is extensive literature on the more general settings of deep permutation invariant networks, which is the case of graph neural networks. For example, what is the relation between the proposed set norm and the one suggested in [2]? I would expect a more detailed discussion about this challenge in graphs versus sets. 

*Residual connections*

Incorporating residual connections in set networks is not novel. It seems that some prior works have used it, as the adaptation of it to set networks seems natural. For example, see implementation details in the supplementary material of [3]. Few alternatives to clean path residual connections are discussed. Do the authors know of such works that follow these alternatives? What are the advantages of these alternatives? Are there any tasks which they improve? If the answer is no, then it is not clear why are they discussed.

*Requirements for a design of a normalization layer*

The authors identify some requirements for normalization layers of permutation equivariant networks.  What does “work well” in the context of varying set sizes mean?  What does essential information mean? It feels like these requirements are too general and should be refined. Moreover, it seems that some discussion relating to test time versus train time equivariance is missing. Is it true that the normalization choices that the authors show that fail (such as L = {B} )  in satisfying equivariance is only at training time? As in test time, some constant inferred from the batch statistics is used. If that is the case, then I expect to see a discussion on why satisfying the equivariant property in training time is important.

*Results*

For classification tasks, it would be more informative to report accuracy. For regression tasks, the units of the measurement should be reported and explained as well.

[1] Qi, Charles R., et al. Pointnet: Deep learning on point sets for 3d classification and segmentation.

[2] Zhao, Lingxiao, and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns.

[3]  Mescheder, Lars, et al. Occupancy networks: Learning 3d reconstruction in function space.

summary_of_the_review: * Well written and easy to follow
* The list of requirements identified by the authors from normalization equivariant layers needs a major refinement. 
* Lack of novelty and the benefit of the set norm is unclear
* Missing discussion on the relation to graph neural networks
* The experiments do not test properly the effect of depth on learning
review_title: null

Meta-review: The submission focuses on a set norm normalization layer for neural network models, which stands in contrast to a batch norm.  The majority of the reviewers felt that this submission is not suitable for publication at ICLR in its current form.  These concerns remained after the post-rebuttal discussion.  Quoting from the reviewer discussion, the following points remained as significant concerns:


1. lack of novelty. normalization layers have been used previously in other sets architectures. The systematic approach for normalization in the current paper is nice but I am not sure how valuable it is. There exists already extensive literature on normalization layers for graphs (a generalization of sets).
2. lack of motivation for deep networks. The main claim in the paper (see the introduction and figure 1) is that 50 layers should perform better and since it does not seem to be the case in figure 1, it requires studying normalization layers. I am not sure it is a well-established claim. What are the assumptions leading to the conclusion that deep architectures should perform better on the task considered in figure 1? I am also concerned that normalization layers have a major effect on improving "not so deep" networks with 3-10 layers and not only the extreme 50 layers case, making the comparison in the paper between only 3 and 50 layers not enough for telling the full story. Thus evaluation on more depths is required.

On the balance, the paper does not meet the threshold for acceptance in this round of peer review.
Decision: Reject