Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper addresses the problem of finding appropriate aggregation functions that can be used for instance in deep neural network architectures. One such function maps a variable length list of reals to a scalar. 

The authors investigate the possibility to learn aggregation functions from data. To that end, they investigate an Lp norm based parametric model of aggregation functions called LAF that allows to learn a wide range of function including usual aggregation functions such as mean, max or min. They however restrict these functions to lists of reals that are in the unit interval. 

The theoretical part of the paper is short. The authors simply show that their aggregation model has a rather high expressive power and invoke a theorem from prior arts to outline some universality guarantee (which could actually be explicitly recalled).

The limited amount of theory is compensated by extensive numerical experiments. The authors provide experimental evidence that their model generalizes fairly well to large sets of inputs. They also show that their model can be plugged to more conventional neural net layers and is backprop friendly.

This is an overall fair contribution which is well positioned as compared to prior arts. The main issue of this paper is, in my opinion, that the impact on the ML community will be limited if their LAF aggregation model does not become a gold standard like convolutional layers have become. To achieve such goal, one misses an application in which the LAF allows to bring disruptive results by leveraging features that cannot be derived through more usual architectures. In conjunction with this remark, the practical usability of LAF would be much wider if a generalization to sets of vectors would be proposed which could learn other features such as covariance.

Detailed Remarks: 

Sec 2 : If b or e is equal to zero, then L_ab is not continuous. I think these should be limiting cases too.

Sec 4.1 : I fail to understand to what targets input sets are mapped to: one of the feature shown in Fig. 2 or a subset of them. Similarly, I also do not understand what is the global architecture of the approaches. It seems that have 9-dimensional outputs. Please clarify this.

Maybe the plots would be more informative if the y-axis was in log scale.

Sec 4.2 : Do you use convolutional layers before the aggregation layer ? Are they trained together ? 

Sec 4.3 : Can you be more specific on the dataset pre-processing ? Why not directly use the raw dataset ?

review_title: An attempt to learn aggregation functions with neural net compatibility

final_score: rating: 3: Clear rejection
review_content: Summary:
The paper proposes a parameterized learnable aggregation function (LAF) that can aggregate a multi-set of numbers (i.e. map them to a single real-valued number). This is different to prior works such as Deep Sets that use fixed aggregation functions such as max, mean, etc.

Strong Points:
- While Deep Sets have shown that is theoretically sufficient to have a sum aggregation, it is still unclear which kind of aggregation functions work well in practice. Hence, the paper addresses an interesting research question.
- The presented idea is rather simple, which I think is a strong point of the paper.
- I like the analysis in Table 1 that shows how different parameterizations of the LAF correspond to different functions such as sum, min, means, etc.
- I also like the evaluation presented in Figure 4.

Weak Points:
- The setup of the experiments with scalars is unclear. For example, a LAF is supposed to aggregate all scalars in the input. However, the paper states that the LAF model is comprised of 9 LAF functions. Why do we need 9 functions? And how are they composed into a single architecture? Similarly, the paper states that 'DeepSets contains three max units, three sum units, and three mean units'. However, a Deep Set should have one function mapping the input to an intermediate representation, a sum aggregation, and a function that maps the aggregation to the output. I don't see why the model needs three max, sum, and mean units.
- In experiment 1, it is reported that the 'input mapping is performed by three layers with the hyperbolic tangent as non-linear activation'. However, the input is simply a multi-set of scalars. It remains unclear to me why it can make sense to map scalars with a 3-layer network. Furthermore, a sigmoid is applied in case of LAF. Hence, it is unclear if the observed performance is due to the Sigmoid or due to the LAF. Furthermore, it is unclear why the paper uses tanh as activation function while Deep Set implementations use ReLUs. Also, the architecture of the output mapping is not described.
- Similar to experiment 1, the experimental setup in experiment 2 is unclear. Additionally, it is unclear how the aggregation of features (i.e. individual dimensions in MNIST images) is related to the aggregation function used to compute the target output of the multi-set.
- While the problem investigated in the paper is permutation-invariant, several methods have been proposed to approximate permutation invariant problems with recurrent architectures such as LSTMs and GRUs. However, no comparison to these kinds of methods is performed. A comparison would be interesting since they also learn an aggregation function.
- It would be interesting to see if the parameters of the LAF function are learned as expected, i.e. if they correlate with the expected values as listed in Table 1. An analysis of this question is missing.
- The paper does not share code or data to improve the reproducibility of the experiments.
review_title: I recommend to reject the paper, mainly due to unclear description of the method and the experiments. This needs to be improved before a proper review of the presented idea is possible. Also, comparison to more other methods should be performed.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

Universal function representation guarantee requires either highly discontinuous mappings or a highly dimensional latent space. For this reason the authors propose a new parametric family of aggregation functions, called LAF (for learning aggregation functions). It can be seen as a smooth version of the class of functions that are shown in DeepSets. LAF aggregator could learn all standard aggregation functions. Moreover in experiments the autors shows that LAF surpasses other aggregation methods.

=============================================================================

Pros:

1. The authors shows, that all standard aggregation functions are achievable by varying the parameters in the formulation of LAF. Moreover LAF enables a neural network to use a continuum of intermediate and hybrid aggregators.

2. Comprehensive ablation study that compares LAF to DeepSets and PNA on digits and MNIST images. In the study the goal is to learn a different types of target aggregation. The results shows that LAF could learn all the given types of aggregation methods as well as it could generalize well to the size of the test set (and thus is not overfitting to the size of the training set as the other methods).

3. The authors provide an extensive set of experiments on a wide range of datasets, including point clouds, set expansion and graph properties. On most of the given tasks LAF is superior to other methods.

=============================================================================

Cons:

Not all the details about LAF aggregation are clear to me. The authors should consider rewriting a section 2 (with a description of the aggregation), considering the points I list below.

1. In the manuscript the autors state that LAF is using the tunable parameters a,...,h and alpha,...,delta, however they do not show how to initialize these parameters and do not tell whether the model is sensitive to these values.

2. The autors state that tunable parameters a,...,h are greater or equal than zero. However hey do not show how to achieve this condition. Whether they use exponent, non-linearity or do it in another way.

3. In the definition of LAF aggregator, the authors states that x should be a real number, however looking at the experiments, it seems to me that it could also be applied to vectors. Please correct me if I'm wrong, but if I am right, than please answer the question whether in this situation a,...,h,alpha,...,delta are still the scalars or whether are they vecors? 

4. The more detailed description of builded networks should be included (even in the appendix). It is not clear to me, how the authors made thir networks. E.g. in section 4.1 they state that 'The LAF model contains nine LAF(x) aggregation functions' - does this mean that in the final aggregation layer you create 9 independently working LAF aggregators and then concatenate them and pass to final prediction layer? If yes, then what in the situation with aggregating the informations from vectors? Do you still make concatenaction?

=============================================================================

Questions during rebuttal period:

1. Why section 5 (Multi-task graph properties) is not the sub-section of section 4 (Experiments)?

2. Usually using a sigmoid could disturb the training of neural network. Could you create the experiment (on a real dataset), where you delete the sigmoid as well as parts with 1-x form the LAF aggregator?

3. Could you create some experiments with LAF as the final aggregator for neural networks? More specific, for exampe, could you use LAF instead of mean average pooling in image classification using ResNet or some text classification task?

=============================================================================

=============================================================================

Reasons for score: 

I vote for accepting this paper. The idea proposed by the authors is novel and elegant. Moreover experiments shows that the proposed model is superior to the other models with whom it has been compared. My major concern is about the clarity of the paper. Hopefully the authors can address my concern in the rebuttal period. 


review_title: A novel method for aggregating the information from sets

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a rational approximation approach for learning aggregation functions. In general the paper is well written and technically sound, although there are some questions that could help the readers better understand the paper. 

One of the problems with learnable rational approximations is the potential of finding a pole, e.g., x/0, you do not mention how you avoid/use? such a situation in your approach, whether this causes instabilities during learning, etc. Could you mention something about this?

Regarding the use of the sigmoid to transform the values of x to the range [0,1], it is not clear to me, how you can recover the mean with such low error, i.e., how can you achieve $\mu(x) ~=  (\sum sigmoid(x)^b)^a / N$. Without the sigmoid transformation it is clear as presented in table 1. But with the sigmoid transformation is not as straightforward to see, although you show very good results in Fig. 2. Furthermore the sigmoid transformation destroys the linearity needed when the values of x are larger. Could you also give some intuition on why using the sigmoid is superior compared to a minmax scaling process?

You mention that you use 9 LAF(x) aggregation functions in the experiments, could you explain more what do you mean exactly? are you using a mixture of LAF(x) models? are they independent of each other and train for the different target functions?

In general, I find the paper interesting and the results promising, just a bit more explanation could help the reader understand the benefits and intuition behind the decisions. 
review_title: Learning Aggregation Functions over sets.

Meta-review: The authors propose a novel and elegant way for learning parameterized aggregation functions and show that their approach can achieve good performance on several datasets (in many cases outperforming other state-of-the-art methods). This is also appreciated by most of the reviewers. However, there have been several issues regarding the description of the proposed approach and the conducted experiments. These have been partly resolved in the rebuttal phase but should be more carefully assessed in another iteration of reviews. 

More specifically: Experiments regarding learning of a single LAF versus multiple LAF should partly be included in the main paper (e.g. Figure 4 showing the performance for different numbers of LAFs). When constructing deep sets in this setting with a similar number of aggregation function it appears not very sensible to me to incorporate the same aggregation function multiple times but one would rather include a set of different fixed aggregation functions (these could be derived from the proposed LAFs). The experiments would also benefit from including set transformers as baselines (set transformers are discussed in the paper but not considered in the experiments as the authors argue that this is an orthogonal approach; while I agree that the goal of set transformers is different, I think there would be big value in understanding how these approaches compare and/or can be combined).

Beyond that I think 	a brief discussion of the related topic of learning pooling operations (e.g., in CNNs) is warranted. 

Some reviewers also find that their concerns are only partially addressed in the rebuttal (e.g., regarding the extension from sets to vectors and applications in which the achieved performance differences are bigger).

One point which didn’t come up in the reviews but I would want to see addressed in a future version of the paper is an extended discussion of Figure 4. While there are cases were LAF clearly performs better, there are also cases, where Deep Sets outperform (this seem to be the cases in which the used aggregation units match the considered task). As LAFs can in theory represent these aggregation function it still seems challenging to learn the correct form of the aggregation function — I would appreciate deeper insights an analysis of this aspect. An immediate heuristic solution for many applications for improving performance thus might be to combine LAFs and standard aggregators.

In summary, the submitted paper has big potential but should be carefully revised and the experiments should be extended before the paper is accepted.
Decision: Reject