Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a way to learn set functions (ie. find a subset $S \subseteq V$ that maximizes some utility function $F_\theta$ that we want to learn) when we are only giving examples of optimal subsets $S^*_i \subseteq V_i$ from an optimal subset oracle. 

This is different to other works that learn from function value oracles that provide actual utility values $f_i$ for specific subsets $S_i$.

The paper proposes a permutation-invariant architecture. It makes the problem tractable through mean-field variational inference and further amortizes inference via an additional neural network.

It reports great empirical results compared to several baselines.

---
Thanks for addressing my questions and updating the paper!
strengths_and_weaknesses: This was requested as an emergency review. I will try my best to offer a meaningful review (ofc), but I'm very much looking forward to discussing with the other reviewers.

The approach seems original (I am not very well acquainted with prior literature in this area though). The quality and clarity of the paper is great overall, and it seems like a significant contribution. 

I very much like the argument that it is easier to acquire data for the optimal subset oracle than the usual function value oracle in many practical cases as trying to obtain calibrated utility values from human labellers sounds improbable. Thus, I think this idea is of great practical importance.

The empirical validation looks sensible.

I recommend acceptance with low confidence given the nature of this review and that my background is not aligned with the subfield.
questions: 1. I believe the max entropy proof for the EBM (App B.1.) can be simplified considerably. In the case of $S \mid V$, $S$ can be simply treated as a categorical with $2^{|V|}$ categories, and the softmax categorical distribution (EBM) is the exponential family distribution for that and maximum entropy as is well-known. This works because at this point, the internal structure of $S$ is not important. It's only when we look at different $V$ that you want to take its structure into account. Is this true?
limitations: The "broader impact" should be removed from the "Limitations & Broader Impact" paragraph title because it does not address broader societal impact (which is okay, but the title is misleading).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses the problem of learning set functions. There are two variations of this problem, one is with function value (FV) oracle in which the supervised data takes the form of (set S_i, function value f_i of set S_i) and the goal is to learn the function mapping. One has to gather this info for large number of sets, making this process prohibitively expensive and often the FV oracle is not feasible. The main focus of the paper is on the variation of the problem with Optimal Subset (OS) oracle, wherein the data takes the form of $(V_i, S_i^*)$ pairs where $V_i $ is subset of $V$ (global candidate set) and $S_i^*$ is the optimal subset of $V_i$ that maximizes the utility. This setup implicitly captures the FV oracle and is more practical. 

Given such data from the OS oracle, they propose a method based on variational inference to learn the mapping.  The first step is to cast the problem as maximum likelihood estimation by replacing the utility function $F_{\theta}(S,V)$ with a probability distribution $P_{\theta}(S|V)$ such that the probability of S given V is proportional to the utility. They list some basic (natural ) properties that such a distribution should satisfy a) permutation invariance, b) varying ground set etc. Further they want to have minimum prior assumption and scalable learning algorithm. 

They give a distribution based on energy-based modeling (EBM) which satisfies the aforementioned properties and admits efficient training and inference. Directly learning with the EBM is difficult for reasons such as intractable partition function, to circumvent the difficulty they propose an approximate maximum likelihood learning using variational approximation of $P_{\theta}(S|V)$ by product of independent Bernoulli distributions.

The method is evaluated on production recommendation, set anomaly detection, double MNIST, CelebA, Compund selection in AI-aided Drug Discovery and other real world datasets.
The results show that the proposed method significantly outperforms the baselines for learning set functions.

strengths_and_weaknesses: Strengths:
S1: It studies the problem of learning set functions under the OS oracle, which is practically important but has been addressed by limited number of works. 

S2: The reduction of this problem to maximum likelihood and later to variational approximation is novel and makes it amenable to application of this well-developed machinery.

S3: The empirical evaluation is extensive and beats the closely related baseline comprehensively. 

Weaknesses:
W1: There two steps of approximation that introduce their own approximation error. It is not clear how these are propagating and affecting the eventual solution. I believe with more approximation layers the error (sub-optimality) is higher.

W2: Lack of theoretical results to shed light on the sub-optimality of the proposed algorithm. I think, the paper can be much better with these. 


questions: Please address the weaknesses. 

limitations: yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces EquivSet, an algorithm for learning set functions that satisfies the following desiderata: permutation invariance, varying ground set, minimum prior and scalability. EquivSet learns set functions under the optimal subset oracle using a maximum likelihood paradigm.  Specifically, the authors use an energy based model(EBM) to define the set mass function. The usage of EBM satisfies the minimum prior requirement.  A DeepSets style architecture is also used to satisfy the permutation invariance constraint on set functions and to also handle sets of arbitrary cardinality. However, learning EBMS over sets introduces some difficulties which the authors alleviate by proposing a mean field variational inference approach in the style of variational auto encoders. Amortization is also used to ensure scalability.
strengths_and_weaknesses: The proposed method is well motivated and tackles the set representation learning problem from a perspective not yet considered.  The EMB approach and the variational inference approach introduce more complexity compared to the well used DeepSets and Set Transformer models.

Additionally, Set Transformer, which is trained similarly to DeepSets, is not used as a baseline. Set Transformer is a simple baseline that normally outperforms DeepSets and is easy to train. Hence i recommend the authors to include it in the baselines.

Finally, it seems the SAB and ISAB layers of Set Transformer can be used in the model in place of the DeepSets style architecture. Many set representation papers show that these attention based backbones perform much better than DeepSets.

**The authors have answered my questions satisfactorily hence i increase my initial score.**
questions: My main question has to do with how Set Transformer performs compared to EquivSet as detailed above. 
limitations: Limitations are sufficiently outlined in the Limitations and Broader Impact section.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a learning framework for set functions. It combines a set of procedures and methods to obtain a unique set of properties for the learned function as listed in introduction.
strengths_and_weaknesses: Strength:
This paper appears to be sound. It is well-written. Algorithms and procedures are clear.

Weaknesses:
My main concern are the limited experimental results, and also the assumptions made about the data distribution and their applicability for practice. Authors have not discussed whether their assumptions about the data applies to the datasets they have used in their experiments. 

Experiments presented in Tables 2 and 3 seem to be limited. Can authors please explain how they chose these particular set of datasets? Results on more datasets could make the paper more convincing.
questions: Please address questions about the experimental results, and the assumptions about the data distribution.
limitations: Yes, the discussion seems adequate.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors tackle the task of set function learning under a weakly supervised setting, where the data points consist of sets and their optimal subsets. The proposed model combines an energy-based model with amortized variational inference to allow for several features (permutation invariance, varying grounds sets, a minimum prior approach, as well as scalability) and is further extended by a Gaussian copula variant to allow for correlation modelling. The model is evaluated on several data sets from different domains.
strengths_and_weaknesses: ## Strengths
- The overall model is well motivated. Each of the desired requirements for the model is discussed in detail and tackled in a principle manner.
- The overall storyline of the paper is well executed. The individual developmental steps and their problems are presented in a coherent structure.
- The paper includes a wide range of experiments.

## Weaknesses
- The reported decimal places in Table 1/2/3 suggest a measurement precision that is not valid given the reported standard deviations, which give us a precision of only the first two decimals for most reported settings.

## Other
- The number of runs and whether standard deviation or standard error are reported should be included in the caption of the tables. The runs are mentioned in the text, but the deviation vs error is not at all discussed (also not in the appendix).
- l302 claims "two real-world datasets". Why this specific formulation for MNIST/CelebA when the other experiments are also on real-world data?
- (very minor) l78 "delicate framework": The wording here feels suboptimal. It can be read in its negative connotation of being a rather fragile framework that requires great care to be trained. The rest of the paper does not suggest this to be the case, allowing for the positive meaning of the word. However, in the introduction, the reader does not know yet which of the two meanings in his/her mind the paper should be read.


### Typos
- l41 lacked $\to$ lacking
questions: -
limitations: Limitations and societal impact are discussed.
review_title: null

Meta-review: metareview: Reviewers have expressed strongly in favour of acceptance, two improving their score after the rebuttal and discussion. I’m happy to recommend acceptance.
confidence: Certain
Decision: Accept