Learning Neural Set Functions Under the Optimal Subset Oracle
Authors: Zijing Ou
Tingyang Xu
Qinliang Su
Yingzhen Li
Peilin Zhao
Yatao Bian
Venue: NeurIPS 2022 Accept (2022)
Keywords: set function
variaitional inference
energy-based models

Abstract: Learning set functions becomes increasingly important in many applications like product recommendation and compound selection in AI-aided drug discovery. The majority of existing works study methodologies of set function learning under the function value oracle, which, however, requires expensive supervision signals. This renders it impractical for applications with only weak supervisions under the Optimal Subset (OS) oracle, the study of which is surprisingly overlooked. In this work, we present a principled yet practical maximum likelihood learning framework, termed as EquiVSet,  that simultaneously meets the following desiderata of learning neural set functions under the OS oracle: i) permutation invariance of the set mass function being modeled; ii) permission of varying ground set; iii) minimum prior and iv) scalability. The main components of our framework involve: an energy-based treatment of the set mass function, DeepSet-style architectures to handle permutation invariance, mean-field variational inference, and its amortized variants. Thanks to the delicate combination of these advanced architectures, empirical studies on three real-world applications (including  Amazon product recommendation, set anomaly detection, and compound selection for virtual screening) demonstrate that EquiVSet outperforms the baselines by a large margin. 

Learning Neural Set Functions Under the Optimal Subset Oracle
Zijing Ou!:², Tingyang Xu', Qinliang Su Yingzhen Li?, Peilin Zhao', Yatao Bian
Tencent Al Lab, China
Imperial College London, United Kingdom
Sun Yat-sen University, China
2. o.ou22immeeriaa ac. ac.uk {tingyangxu, masonzhao) suqliang@mail lrsaanggmail.sss...... edu. en yingzhen. weaeeen...iie li@imperial. ac uk yatao. bi bian@gmaile com
| -ttencent...........m com
Abstract
Learning neural set functions becomes increasingly important in many applications like product recommendation and compound selection in Al-aided drug discovery. The majority of existing works study methodologies set function learning under the function value oracle, which, however, requires expensive supervision signals. This renders impractical for applications with only weak supervisions under the Optimal Subset (OS) oracle, the study of which is surprisingly overlooked. In this work, we present a principled yet practical maximum likelihood learning frame- work, termed as EquiVSet, VSet,' that simultaneously meets the following desiderata of learning neural set functions under the OS oracle: i) permutation invariance of the set mass function being modeled; ii) permission of varying ground set; iii) mini- mum prior; and iv) scalability. The main components of our framework involve: energy-based treatment of the set mass function, DeepSet-style architectures to handle permutation invariance, mean-field variational inference, and its amor- tized variants. Thanks the elegant combination of these advanced architectures, empirical studies on three real-world applications (including Amazon product rec- ommendation, set anomaly detection and compound selection for virtual screening) demonstrate that Equi VSet outperforms the baselines by a large margin.
1 Introduction
Many real-world applications involve prediction set-value outputs, such as recommender systems which output set of products to customers, anomaly detection that predicts the outliers from the majority of data (Zhang al., 2020), and compound selection for virtual screening drug discovery aims at extracting the most effective compounds from given compound database (Gimeno et al., 2019). All of these applications implicitly learn set function (Rezatofighi al., 2017: Zaheer et al., 2017) that measures the utility of given set input, such that the most desirable set output has the highest (or lowest w.l.o.g) utility value.
More formally, consider a recommender system: given a set of product candidates V. it is expected recommend subset of products S* V to the user, which would satisfy the user most, i.e., offering the maximum utility to the user. We assume the underlying process of determining S* can be modelled by a utility function (S; V) parameterized by and the following criteria:
S argmax VS;
SE2V
"Correspondence Yatao Bian.
'Code is available at: http:/ghhhccccceeeeeeeeeeeetee/e
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

There are mainly two settings for learning the utility function. The first one, namely function value (FV) oracle, targets at learning Fo V) to fit the utility explicitly, under the supervision of data in the form of {S,,, ʃi)] for fixed ground set V. where Sa is the true utility function value of the subset S, However, training in this way is prohibitively expensive, since one needs to construct large amounts of supervision signals for specific ground set V (Balcan & Harvey, 2018). Here we consider an alternative setting, which learns (S; V) in an implicit way. More formally, with the data in form of {(V, H where S is the optimal subset (OS) corresponding to V,, our goal is to estimate such that for all possible (V,, St), it satisfies equation (1). The OS oracle is arguably more practical than the FV oracle, which alleviates the need for explicitly labeling utility values for large amount of subsets.?
Though being critical for practical success, related study on set utility function learning under the OS supervision oracle is surprisingly lacked. The most relevant work is the probabilistic greedy model (PGM), which solves the optimization problem of (1) with greedy maximization algorithm (Tschiatschek et al., 2018). Specifically, PGM interprets the maximization algorithm as to construct differentiable distributions over sequences of items in an auto-regressive manner. However, such construction of distributions is problematic for defining distributions on sets due to the dependency on the sampling order. Therefore, they alleviate this issue by enumerating all possible permutations of the sampling sequence (detailed discussion is given in Appendix A). Such enumerations scale poorly due to the combinatorial cost ([V)),, which hinders PGM's upplicability to real-world applications.
To learn set functions under the OS oracle, we advocate the maximum likelihood paradigm (Stigler, 1986). Specifically, this learning problem can be viewed from probabilistic perspective
argmax EE(V,)) llog (S[V))
#
po (SIV) x (S,
where the constraint admits the learned set function to obey the objective defined in (1). Given limited data {(V,, (T) )1 sampled from the underlying data distribution IP(V, one would maximize the
N empirical log likelihood: Ei=, [log (S [V]]) The most important step is to construct a proper set distribution pa (S]V) whose probability mass monotonically grows with the utility function (o;;; V) and satisfy the following additive requirements: (i) permutation invariance: the probability mass should not change under any permutation of the elements in S; (ii) varying ground set: the function should be able to process input sets of variable size: iii) minimum prior: we should make no assumptions of the set probability, i.e., with maximum entropy, which is equivalent to the uninformative prior (Jeffreys, 1946); and iv) scalibility: the learning algorithm should be scalable to large-scale datasets and run in polynomial time.
In this paper, we propose Equivariant Variational inference for Set function learning (Equi a new method for learning set functions under the OS oracle, which satisfies all the requirements. Specifically, we use an energy-based model (EBM) to construct the set mass function. EBMs are maximum entropy distributions, which satisfies the minimum prior requirement. Moreover, by modeling the energy function with DeepSet-style architectures (Zaheer et al., 2017; Lee et al.. 2019), the two requirements, i.e., permutation invariance and varying ground set are naturally satisfied. Unfortunately, the flexibility of EBMs exacerbates the difficulties of learning and inference, since the inputs of set are discrete and lie in an exponentially-larg space. To remedy this issue, we develop an approximate maximum likelihood approach which estimates the marginals via the mean- mean-field variational inference, resulting in an efficient training manner under the supervision of OS oracles. In order to ensure scalability, an amortized inference network with permutation equivariance is proposed, which allows the model to be trained on large-scale datasets.
Although it may be seen as combining existing components in approximate inference, the proposed framework addresses a surprisingly overlooked problem in the set function learning communities using an intuitive yet effective method. Our main contributions are summarized below:
NNotably, learning set functions under the OS oracle is distinct to that under the FV oracle; the two settings are not comparable in general. To illustrate this, one can easily obtain the FV oracle maximum cut set functions, but fail to specify the OS oracle since it is co-complete to solve the maximum cut problem (Garey & Johnson, 1979, Appendix A2.2). Moreover, even though the OS oracle naturally shows up in the product recommendation scenario, one cannot identify its FV oracle since the true utility values are hard to obtain.
2

---

We formulate set functions learning problems under the OS supervision oracle using the maximum likelihood principle;
We present elegant framework based on EBMs which satisfies the four desirable require- ments and is efficient both at training and inference stages;
Real-world experiments demonstrate effectiveness of the proposed OS learning framework.
2 Energy-Based Modeling for Set Function Learning
The first step to solve problem (2) is to construct proper set mass function po (S) monotonically growing with the utility function (S; V). There exits countless ways to construct such a probability mass function, such as the sequential modeling in PGM (Tschiatschek al., 2018, 8, Section 4). Here we resort to the energy-based treatment:
pe (s(V) exp ex)(Po(S:V)) Z: === = s'gv exp(Fo(S':V)).
(3)
where the utility function (S; V) stands for the negative energy, with higher utility representing lower energy. The energy-based treatment is attractive, partially due to its maximum entropy (i.e., minimum prior) property. That is, it assumes nothing about what is unknown, which is known as the "noninformative prior" principle in Bayesian modeling (Jeffreys, 1946). This basic principle is, however, violated by the set mass function defined in PGM. We refer detailed motivation of the energy-based modeling Appendix B.I.
In addition to the minimum prior, the energy- based treatment also enables the set mass function (S)VV to meet the other two requirements, i.e. permutation invariance and varying ground set, by deliberately designing a suitable set function I& Fs(S; (S; V). However, modeling such proper function is nontrivial, since classical feed-forward neural networks (e.g.. the ones designed for submodular set functions (Bilmes & Bai, 2017)) violate both two criteria, which restricts their applicability to the problems involving set of objects. Fortunately, Zaheer et al. (2017) sidestep this issue by introducing a novel architecture, namely DeepSet. They theoretically prove the following Proposition.
Proposition 1. All permutation invariant set functions can be decomposed in the form f(S) = (E.es r(s)). for suitable transformations τ and p.
By combining the energy-based based model in (3) with DeepSet-style architectures, we could construct a valid set mass function to meet two important criteria: permutation invaraince and varying ground set. However, the flexibility of EBMs exacerbates the difficulties of learning and inference, since the partition function Z is typically intractable and the input of sets is undesirably discrete.
3 Approximate Maximum Likelihood Learning with OS Supervision Oracle
In this section, we explore an effective framework for learning functions under the supervision of optimal subset oracles. We start with discussing the principles for learning parameter 0, followed by discussing the detailed inference method for discrete EBMs.
3.1 Training Discrete EBMs Under the Guidance of Variational Approximation
For discrete data, e.g. set, learning the parameter in (3) via maximum likelihood is notoriously difficult. Although one could apply techniques, such as ratio matching (Lyu, 2012). noise contrastive estimation (Tschiatschek et al., 2016), and contrastive divergence (Carreira-Perpine & Hinton, 2005), they generally suffer from instability on high dimensional data, especially when facing very large ground set in real-world applications. Instead of directly maximizing the log likelihood, we consider an alternative optimization objective that is computationally preferable. Specifically, we first fit variational approximation to the EBM by solving
26 argmin (S (()( (1)p)(())
(4)
3

---

Algorithm MF V, K)
Algorithm 2 DiffMF
Algorithm 3 EquiVSet
for lt 1 K
1: initialize variational parameter
update parameter using (6)
2:
[V] in parallel do
( + 0.5 1
Φ t Φ + 7V,
3:
sample m subsets
2: compute the marginals
2: initialize variational parameter
So--((;; (ا) 00)
3 MF V. K)
v000 + EequiNet
4:
update variational parameter
3: update parameter 0 using (5)
3: one step fixed point iteration
0 x - 1/Vo x²)
3 + () V. K = 1)
4: update parameter using (5)
5: end for
0 с 0 Vo (²))
6:
end for
Figure 1: The main components and algorithms in our framework. Note that DiffMF and Equi VSet are for one training sample only. Detailed and self- containdd descriptions of each component of these algorithms are presented in Appendix D.
where D(-lI-) is a discrepancy measure between two distributions, (e) is the EBM defined in (3), and qS; +)) denotes the mean-field variational distribution with the parameter sb E [0, 1]IV| standing for the odds that each item E V shall be selected in the optimal subset S Note that the optimal parameter s5" of (4) can be viewed as a function of 0. In this regard, we can optimize the parameter 0 by minimizing the following cross entropy loss,* which is well-known to be implementing the maximum likelihood estimation (Goodfellow et al., 2016) w.E.t. the surrogate distribution qSS) v²),
log ((;: C(0; -
1
Σ log 3 JEST
E log(1 (57)
(5)
JEVASS
This is also known as the marginal- based loss (Domke, 2013), which trains probabilistic models by evaluating them using the marginals approximated by an inference algorithm. Despite not exactly bounding the log-likelihood of (3), this objective, as pointed out by Domke (2013), benefits from taking the approximation errors of inference algorithm into account while learning. However, minimizing (5) requires the variational parameter &5" being differentiable w.r.f. A. Inspired by the differentiable variational approximation the Markov Random Fields (Krähenbühl & Koltun, 2013; Zheng et al., 2015; Dai et al., 2016), below, we extend this method to the deep energy-based formulation, which admits an end-to-end training paradigm with the back-propagatton algorithm.
3.2 Differentiable Mean Field Variational Inference
To solve the optimization problem (4), we need to specify the variational distribution Q(S; xb) and the divergence measure D(-II-), such that the optimum marginal x6" is differentiable w.r.t. the model parameter 0. A natural choice is to restrain (S;; v)) to be fully factorizable, which leads to a mean-field approximation of pa (S). The simplest form of (S; Φ)) would be [V] independent Bernoulli distribution, i.e., q(; = -Iies 44 IIigs vh)) & E [0, 1]MM) Further restricting the discrepancy measure D(gllp) to be the Kullback- Leibler divergence, we recover the well-known mean-field variational inference method. It turns out that minimizing the KL divergence amounts to maximizing the evidence lower bound (ELBO)
min (g +) (S)) +
max fife (4b) + H(((S: =: ELBO,
+
where flle (1b) is the multilinear extension of (S) (Calinescu et al., 2007), which is defined as f (vb)
Σ (o)
II
4 II w)), &b [o, M
SCV
EE
igS
To maximize the ELBO in (6), one can apply the fixed point iteration algorithm. Specifically, for coordinate dh, the partial derivative of the multilinear extension is Fi (v)), ( and for the
3 'Here we omit the condition for brevity. In some specific context, would be helpful to regard subset binary vector, Le.. S = {0. 1}1l with the i-th element equal to 1 meaning i E S and 0 meaning i & S. *This objective would suffer from label-imbalanced problem when the size of OS too small. In practice, we can apply negative sampling to overcome this problem: we randomly select a negative set N; C S with
the size of [. and train the model with an alternative objective Σ Σ log 3/
Σ log(1 1 vz). JEN,
1E8

---

entropy term, it is V., (q)) = log 1-1; Thus, the stationary condition of maximizing ELBO is v/ = oV J (vb)), = 1,....,Vi,, IVI. where a is the sigmoid function, which means dh should be updated as vy c- v. Jmi (xb)). This analysis leads to the traditional mean field iteration, which updates each coordinate one by one (detailed derivation in Appendix B.2). In this paper, we suggest to update & in batch manner, which is more efficient in practice. More specifically, we summerize the mean field approximation as the following fixed-point iterative update steps
8:
(6 f Initialize in [0, 1)
(8)
(b()) - e1x+xxp(- V wo Sine (ab (2b ((((-111))
(9)
2 (b(k)
(10)
We denote the above iterative steps as function termed as MFVI( V, K), which takes initial vairational parameter th, ground set V. and number of iteration steps K as input, and outputs the parameter vó" after K steps. Note that, MFVI( V. K) is differentiable w.r.f. the parameter 0, since each exeed-point iterative update step is differentiable. Thereby, one could learn by minimizing the cross entropy loss in (5). However, the computation complexity raises from the derivative of multilinear extension fuat (ap) defined in (7), which sums up all the possible subsets in the space of size 2|VI, Fortunately, the gradient V can be estimated efficiently via Monte Carlo approximation methods, since the following equation holds.
V vt Seit (xb) E Ey(s:tii..o)) [Fe(S + Ps()
(11)
in which we S +-E denote the set union {T) Detailed derivation is provided in Appendix B.3. via Monte Carlo approximation:
According to (11), we can estimate the partial derivative v. ffae
i) sample m subsets Sn, n I.... m from the surrogate distribution gS; (able) & < 0)); ii) approximate the expectation by the average m for a given ground set can be sampled via rounding sb", which is the optimal variational parameter after K-steps mean-field iteration, i.e., ab" = MFVI( V, K), and stands for the probability of each element in the ground set should be sampled. 5 We term this method as Differentiable Mean Field (DiffMF) and summarize the training and inference process in Algorithm 2 and 1, respectively.
1 Σx IF= + +i) (S,), After training. the OS
4 Amortizing Inference with Equivariant Neural Networks
Although DiffMF can learn set function Fe in an effective way, it undesirably has two notorious issues: i) the computation is in general prohibitively expensive, since DiffMF involves typically expensive sampling loop per data point; ii) some information regarding interactions between elements is discarded, since DiffMF assumes a fully fatorizable variational distribution. In this section, first propose to amortize the inference process with an additional recognition neural network, and then extend it to considering correlation for more accurate approximations.
4.1 Equivariant Amortized Variational Inference
To enable training the proposed model on large-scale dataset, we propose to amortize the approxi- mate inference process with an additional recognition neural network which outputs parameter sb for the variational distribution (S; 3b). where & denotes the parameter of neural networks. A proper recognition network involving set objects shall satisfy the property of permutation equivariance.
Definition 1. A function f x + yd is called permutation equivalent when upon permutation of the input instances permutes the output labels, i.e., for any permutation æ: f(a([2)...., ,,,))) = T (f(1.....
Zaheer et al. (2017) propose to formulate the permutation equivariant architecture as
f((S) = p (2rs(s)) + > SES r((s))
(12)
$ 'Here we simply apply the topN rounding, but it is worthwhile to explore other rounding methods as future work.
With slight abuse of notations, use the same symbol here in (6).
5

---

min (g (S; (Ipp) (S))
correlation aware amortisad infervace
differentiable MFVI update
(Eq.6)
EquiNet
7/(00) θ (BBM param.)
MFVI (4) V, K)
(0)
MLE on
(V; Φ)) Sec.
q ( y (K) (0))
(F4.5)
(vatiational param.)
learning Φ
learning 0 (backprop)
inference
(backprop)
Figure 2: Overview of the training and inference processes of Equi VSet.
where 8; denotes the hth element in the set S, X, are learnable scalar variables, and p. K are any proper transformations. Note that the output value of fi 2 [0, 1] is relative to the 7th coordinate, but not the order of the elements S. Thus the equivariant recognition network, denoted as 1b EquiNet((; Φ) 2 - [0. 1][v] can be defined as EquiNet, == fú, which takes the ground set V as input and outputs the distribution parameter vb for (S;
4.2 Correlation-aware Inference with Gaussian Copula
Due to the mean-field assumption, the proposed variational distribution cannot model the interactions among elements in the input set. We address this issue by introducing Gaussian copula (Nelsen, 2007), which is cumulative distribution function (CDF) of random variables (uI...., "[v]) over the unit cube [0, 1 1111, with u; ~ Uniform 1). More formally, given a covariance matrix E, the Gaussian copula Oz with parameter Σ defined as
Cx("1, "v कΣ ( (0-1((m))
(u1),
(uvvi))
where Ty stands for the joint CDF of Gaussian distribution with zero mean and covariance matrix E. and 4 the inverse CDF of standard Gaussian. With the location parameter & output by EquiNet(V; o)) we can induce correlation into the Bernoulli distribution via the following way: i) sample an auxiliary noise g ~ (0, Σ); ii) apply element-wise Gaussian CDF u & ding(E)) (g); iii) obtain binary sample via 8 = T(ub S u). where vé < u means Vi, di S ue I(-) is the indicator function, and diag( Σ) returns the diagonal matrix of Σ. In practice, the covariance matrix E could be generated by another neural network with the input ground set. We refer the discussion on it to Appendix C, and demonstrate how efficiently construct and sample from non-diagonal Gaussian distribution, while retaining a permutation equivariant sampling process.
To learn the parameters of the variational distribution, one can maximize the ELBO objective in (6). However, the ELBO has no differentiable closed-form expression w.r.t. Φ. To remedy this, we relax the binary variable to a continuous one by applying the Gumbel-Sofmax trick (Jang et al., 2016), resulting in an end-to-end training process with backpropagation.
4.3 Details of Training and Inference
Our model consists of two components: the EBM po (S)and the variational distribution (S; v/). As shown in Figure 2, these two components are trained in a cooperative learning fashion (Xie et al., 2018). Specifically, we train the variational distribution 4s with fixed firstly by maximizing the ELBO in (6). To train the energy model pa. we first initialize the variational parameter & (0) with the output of equivariant recognition network EquiNet(V V; Φ). This enables us get more accurate variational approximate, since de has modeled the correlation among the elements in the set. Notice that (0) does not depend on 0 directly. To learn 0, we take one further step of mean-field iteration MFVI(b)(0) (0) V, 1), which flows the gradient through 0 and enables to optimize 0 using the cross entropy loss in (5) (i.e., if we skip step in Algorithm and feed 26 to step the gradient would not flow through 0). However, if we take multiple steps, it inclines to converge to the local optima that
7 Herr s binary vector (0. IVl with the i-th element equal to meaning i E S and meaning For correlation aware inference, the variational parameter é consists of two parts: i) of the EquiNet and ii) E of the Gaussian copula.

---

is the same as the original mean-field iteration. As a result, the benefit of correlation -wwale inference provided by the Gaussian copula would be diminished. Detailed analysis is provided in Appendix F.5. The training procedure is summarized in Algorithm 3 (the complete version is given in Appendix D).
aware
For inference in the test time, given ground set V, we initialize the variational parameter via eb(0)- EquiNet V: o)), then run one step mean-field iteration b +- MFVI( V.1). Finally, the corresponding OS is obtained by applying the topN rounding method. We term our method as Equivariant Variational Inference for Set Function Learning (EquiVSet)) and respectively use Equi ind ind and Equi copula to represent two variants with independent and copula variational posterior, respectively.
5 Related Work
Set function learning. There is growing literature on learning set functions with deep neural networks. Zaheer et al. (2017) designed the DeepSet architecture to create permutation invariant and equivariant function for set prediction. Lee et al. (2019) enhanced model ability of DeepSet by employing transformer layer to introduce correlation among instances of set, and Horn et al. (2020) extended this framework for time series. It is noteworthy that they all learn set functions under the function value oralce and can be employed as the backbone of the utility function (S; V) in our model. Dolhansky & Bilmes (2016); Bilmes & Bai (2017); Ghadimi & Beigy (2020) have also designed deep architectures for submodular set functions, however, these designs can not handle the varying ground set requirement. There are papers studying the learnability of specific set functions (e.g.. submodular functions and subadditive functions) in a distributional learning setting (Balcan et al., 2012; Badanidiyuru et al., 2012; Balcan & Harvey, 2018) under the function value oracle, they mainly provide sample complexity with inapproximability results under the probably mostly approximately correct (PMAC) learning model. Other methods relevant to our setting TSPN (Kosiorek et al., 2020) and DESP (Zhang et al., 2020). However they both focused on generating set objects under given condition. While we aim predicting under the optimal subset oracle.
Energy-based modeling. Energy based learning (LeCun et al., 2006) is a classical framework to model the underlying distribution over data. Since it makes no assumption of data, energy-based models are extremely flexible and have been applied to wide ranges of domains, such as data generation (Nijkamp et al., 2019), out-o--distribution detection (Liu et al., 2020), game-theoretic valuation algorithms (Bian et al., 2022) and biological structure prediction (Shi et al., 2021). Learning EBMs can be done by applying some principled methods, like contrastive divergence (Hinton, 2002), score matching (Hyvärinen & Dayan, 2005), and ratio matching (Lyu, 2012). For inference, gradient-based MCMC methods (Welling & Teh, 2011; Grathwohl al., 2021) are widely exploited. Meanwhile, Bian et al. (2019); Sahin et al. (2020) propose provable mean-filed inference algorithms for a class of EBMs with supermodular energies (also called probabilistic log-submodular models). In this paper, we train EBMs under the supervision of OS oracle by running mean-field inference.
Amortized and Copula variational inference. Instead of approximating separate variables for each data point, amortized variational inference (VI) (Kingma & Welling, 2013) assumes that the variational parameters can be predicted by parameterized function of the data (Zhang et al., 2018). The idea of amortized VI has been widely applied in deep probabilistic models (Hoffman et al., 2013; Garnelo et al., 2018). Although this procedure would introduce an amortization gap (Cremer et al., 2018), which refers to the suboptimality of variational parameters, amortized VI enables significant speedups and combines probabilistic modeling with the representational power of deep learning. Copula is the other method to improve the representational power for VI. Tran et al. (2015) used copula to augment the mneld-field VI for better posterior approximation. Suh & Choi (2016) adopted Gaussian copula in VI to model the dependency structure of observed data. However, none of them can be directly applied to our setting involving discrete latent variables.
6 Empirical Studies
We evaluate the proposed methods on various tasks: product recommendation, set anomaly detection, compound selection, and synthetic experiments. All experiments are repeated five times with different random seeds and their means and standard deviations are reported. The model architectures and training details are deferred to Appendix E. Additional experiments of varying ground set are given
7

---

Table 1:
Results in the MJC metric on Two- Moons and Caassian-Mixtut datasets.
Method
Two Moons
Gaussian Mixture
Random
0.055
0.055
wrong predtt preditt
predic:
presktt
PGM
0,360 + 0.020
0.438 + 0.009
DeepSet (NoSetFn)
0.472 1 0.003
0.446 1 0.002
Figure
Visualization of the prediction of Equi the Two-Moons (left) and Gaussian-Mixture (right) datasets.
DiffMF (ours)
0.584 + 0.001
0.908 + 0.002
EquiVSet (ours)
0.570 +: 0.003
0.907 + 0.002
EquiVSet copula (ours) 0.587 0.002
0,909 + 0.002
in Appendix F.I. Comparisons with Set Transformer (Lee et al., 2019) are in Appendix F.2. Ablation studies on perturbation, temperature of Suftel-Softmax are provided in Appendix F.5.
choices (e.g. MFVI iteration steps, number of MC samples, rank of
Evaluations. We evaluate the methods using the mean Jaccard coefficient (MJC). Specifically, for each sample (V,S*) denoting the corresponding model predict as 8", the Jaccard coefficient is defined as JC(S, S*) = [S'nS] Then the MJC metric can be computed by averaging over all samples
in the test set: MJC = [D (V..-)ED, JC(S",
Baselines. We compare our solution variants, i.e., DiffMF, following three baselines:
and EquiVSet.oo copula to the
The expected performance of random guess. This baseline provides an estimate of how difficult the task is. Specifically, given data point (V,5*), can be computed as ((JC((V
Random:
(C 18 1-
Σ
S'1 k=
(1841)
2|8-1
PGM (Tschiatschek et al. (2018), see Appendix A): The probabilistic greedy model, which is permutation invariant but computationally prohibitive.
DeepSet (NoSetFn) (Zaheer al., 2017): The deepset architecture, satisfying permutation invariant, is the backbone of our models. Its adapted version: 2 - [0, 1jlvi, which serves the amortized networks in Equi VSet, could work as a baseline since its output stands for the probability which instance should be selected. We train it with rounding. The term "NoSetFn" is used to emphasize that this baseline does not learn a set function explicitly, although it can be adapted to our empirical studies.
entropy loss and sample the subset via topN
Synthetic Experiments. We demonstrate the effectiveness of our models on learning set functions
with two synthetic datasets: the two-moons dataset with additional noise variance o² 0.1, and mixture of Gaussians [ Σ) + [ E), with Mo = 1/ / M1 Mo- Σ = I.. Take
the Gaussian mixture as an example, the data generation procedure is as follow: i) select index: b ~ Bernoulli {): ii) sample 10 points from N ( E) to construct S* iii) sample 90 points for V\.S* from (P1-o E). We collect 1, 000 samples for training, validation, and test, respectively.
qualitative result the EquiVSet sopula is shown in Figure 3, where the green dots represent correct model predictions, the red crosses are incorrect model predictions, and the yellow triangles represent the data points in the subset oracle S* that are missed by the model. One see that the most confusing points are located at the intersection two components. We also illustrate the quantitative results in Table 1. As expected, our methods achieve significantly better performance over other methods, with averaged 59.169 16% and 100.07% improvements compared to PGM on the Two-Moons and Gaussian Maissian-Mixture datasets, respectively.
Product Recommendation. In this experiment, we use the Amazon baby registry dataset (Gillen- water et al., 2014), which contains numerous subsets products selected by different customers. Amazon characterizes each product in baby registry as belonging to specific category, such as "toys" and "furniture". Each product is characterized by short textual description and we represent it 768 dimensional vector using the pre-trained BERT model (Devlin al., 2018).

---

Table 2: Product recommendation results on the Amazon dataset with different categories.
Categories
Random
PGM
DeepSet (NoSetFF)
DiffMF (ours)
EquiVSet (ours)
EquiVSet copuls (ours)
Toys
0.083
0.441 t: 0.004
0.429 + 0.005
0.610 + 0.010
0.650 + 0.015
0.680 : 0,020
Furniture
0.065
0.175 + 0.007
0.176 + 0.007
0.170 1 0.010
0.170 + 0.011
0.172 + 0,009
Gcar
0.077
0.471 + 0.004
0.381 = 0.002
0.560 = 0.020
0.610 = 0.020
0.700 = 0.020
Carseats
0.066
0,230 + 0,010
0.210 = 0,010
0.220 + 0.010
0.214 + 0.007
0.210 + 0.010
Bath
0.076
0,564 +: 0.008
0.424 = 0.006
0.690 = 0.006
0,650 = 0.020
0.757 = 0.009
Hcalth
0.076
0.449 + 0.002
0.448 = 0.004
0.565 = 0.009
0.630 0.020
0.700 = 0,020
Diaper
0.084
0,580 1: 0.009
0.457 1 0.005
0.700 0.010
0,730 1 0.020
0.830 + 0,010
Bedding
0.079
0,480 1: 0.006
0.482 1 0.008
0.641 + 0.009
0.630 + 0.020
0.770 + 0,010
Safety
0.065
0.250 + 0.006
0.221 + 0.004
0.200 + 0.050
0.230 + 0.030
0.250 + 0.030
Feeding
0.093
0,560 + 0.008
0.430 + 0.002
0.750 + 0.010
0.696 + 0.006
0.810 + 0.007
Apparel
0.090
0.533 + 0.005
0.507 = 0.004
0.670 = 0.020
0.650 = 0.020
0.750 = 0.010
Media
0.094
0.441 + 0.009
0.420 = 0.010
0,510 + 0.010
0.551 + 0.007
0.570 + 0.010
For each category, we generate samples (V.S*) as follows. Firstly, we filter out those subsets selected by customers whose size is equal to 1 or larger than 30. Then we split the remaining subset collection S into training, validation and test folds with a
1 ratio. Finally for each OS oracle S* E S, we randomly sample additional 30 - |S products from the same category to construct V\S"" In this way, we construct one data point (V, S*) for each customer, which reflects this real world scenario: V contains 30 products displayed to the customer, and the customer is interested in checking |SS*| of them. Note that this curation process is different from that of (Tschiatschek et al., 2018, Section 5.3), which is deviated from the real world scenario (Detailed discussion in Appendix E.S.).
The performance of all the models on different categories are shown in Table Evidently, our models perform favorably to the baselines. Compared with PGM, which learns the set function via a probabilistic greedy algorithm, we can observe that our models, which model the the set functions with energy-based treatments, achieves better results on all settings. Although DeepSet also permutation invariant, our model still outperforms it by a substantial margin, indicating the superiority of learning the set function explicitly.
Set Anomaly Detection. In this experiment, we evaluate our methods on two image datasets: the double MNIST (Sun, 2019) and the CelebA (Liu et al., 2015b). For each dataset, we randomly split the training, validation, and test set to the size of 10, 000, 1, 000, and 1, 000, respectively.
Double MNIST: The dataset consists of 1000 images for each digit ranging from 00 99. For each sample (V, ), we randomly sample n E {,...., 5) images with the same digit to construct the OS oracle S*, and then select 20- |SS*| images with different digits construct the set V' CelebA The CelebA dataset contains 202, 599 images with 40 attributes. We select two attributes at random and construct the set with the size of 8. For each ground set V, we randomly select n E [2,3] images as the OS oracle 5*, in which neither of the two attributes is present. See Figure 4 and Figure 5 in Appendix E.6 for illustrations of sampled data.
From Table 3, we see that the variants of our model consistently outperform baseline methods strongly. Furthermore, we observe that by introducing the correlation to the variational distribution, significant performance gains can be obtained, demonstrating the benefits of relaxing the independent assumption by using Gaussian copula. Additional experiments on the other two datasets F-MNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky et al., 2009) are provided in Appendix F.3.
Compound Selection in Al-aided Drug Discovery. A critical step in drug discovery is to select compounds with high biological activity (Wallach et al., 2015; Li et al., 2021; Ji al., 2022), diversity and satisfactory ADME (absorption, distribution, metabolism, and excretion) properties (Gimeno al., 2019). As result, virtual screening is typically a hierarchical filtering process with several necessary filters, e.g., first choosing the highly active compounds, then selecting diverse subsets from them, and finally excluding compounds that are bad for ADME. We finally arrive at a compound subset after a series of these steps. Given the OS supervision signals, we can learn to conduct this complicated selection process an end to end manner. As a result, will eliminate the need for intermediate supervision signals, which can be very expensive or impossible to obtain due to pharmacy's 's personal protection policy. For example, measuring the bioactivity and ADME, properties of compound has to be done in wet labs, and pharmaceutical companies reluctant to disclose

---

Table 3: Set anomaly detection results.
Table 4: Compound selection results.
Method
Double MNIST
CelehA
Method
PDBRind
BindingDB
Random
0.082
0.219
Random
0.073
0.027
PGM
0,300 1 0.010
0.481 1: 0,006
PGM
0,350 1 0.009
0.176 0.006
DeepSet (NoSetFa)
0.111 0.003
0.390 + 0.010
DeepSet (NoSeiFn)
0.319 + 0.003
0.162
0.007
DifiMF (ours)
0.610 0.010
0,546 + 0.008
DiffMF (ours)
0.360 = 0.010
0.189
0.002
EquiVSet (ours)
0.410 0.010
0,530 1 0.010
EquiVSet (ours)
0.355 1 0.005
0.190 0.003
Squ (ours)
0.588 0.007
0.555 + 0.005
EquiVSeE (ours)
0.354 + 0.008
0.188 0.003
the data. Here we simulate the OS oracle of compound selection by applying the two filters: high bioactivity and diversity filters, based on the following two datasets.
PDBBind (Liu et al., 2015a): This dataset consists of experimentally measured binding affinities for bio-molecular complexes. We construct dataset using the "refined" subsets therein, which contains 179 protein-ligand complexes. BindingDB': It is a public database of measured binding affinities, which consists of 52, 273 drug-targets with small, drug-lik like molecules. Instead of providing complexes, here only the target amino acid sequence and compound SMILES string are provided.
We apply the same filtering process to construct samples (V, ) for these two datasets. Specifically, we first randomly select a number of compounds to construct the ground set V. whose size is 30 and 300 for PDBBind and BindingDB, respectively. Then 5 compounds with the highest bioactivity are filtered out, accompanied by a distance matrix measured by the corresponding fingerprint similarity of molecules. To ensure diversity, the OS oracle S* is generated by the centers of clusters which are presented by applying the affinity propagation algorithm. We finally obtain the training, validation, and test set with the size of 1,000, 100, and 100, respectively. for both two datasets. Detailed description is provided in Appendix E.7.
From Table 4, one can see that our methods magnificently outperform the random guess. This indicates that the proposed EquiVSet framework has great potential for drug discovery to facilitate the virtual screening task by modeling the complicated hierarchical selection process. Besides, improvements of EquiVSet VSet can be further observed by comparing with DeepSet, which simply equips the deepset architecture with cross entropy loss, illustrating the superiority of explicit set function learning and energy-based -based modeling. Although comparable results could achieved by PGM with sequential modeling, which satisfies permutation invariance and differentiaaiiity, models still outperform it. This is partially because our models additionally maintain the other three desiderata of learning set functions, i.e., varying ground set, minimum prior, and scalability. We also conduct a fairly simple task in Appendix F.4, in which only the bioactivity filter is considered. To simulate the full selection process, we leave it important future work due to limited labels.
Discussion and Conclusion
We proposed a simple yet effective framework for set function learning under the OS oracle. By formulating the set probability with energy-based treatments, the resulting model enjoys the virtues of permutation invariance, varying ground set, and minimumn prior. A scalable training and inference algorithm is further proposed by applying maximum log likelihood principle with the surrogate of mean-field inference. Real-world applications confirm the effectiveness of our approaches.
Limitations & Future Works. The training objective in (5) does not bound the log-likelihood of EBMs. A more principled discrete EBMs trainer is worth exploring. In addition, the proposed framework has the potential to facilitate learning to select subsets for other applications (Iyer et al., 2021), including active learning (Kothawade et al., 2021), 1), targeted selection of subsets, selection of subsets for robustness (Killamsetty al., 2020), and selection of subsets for fairesss. Though consider learning generic neural set functions in this work, it is beneficial to consider building useful priors into the neural set function architectures, such set functions with the diminishing returns prior (Bilmes & Bai, 2017) and the bounded crrvature/sumodullar ratio prior (Bian et al., 2017).
WW take the curated one from https://tdcommon..i/miii ai/multi tasks/dti/
10

---

References
Badanidiyuru, A., Dobzinski. S., Fu, H., Kleinberg, R., Nisan, N., and Roughgarden, T. Sketching valuation functions. In Rabani, Y. (ed.), Proceedings of the Twenty-Thire Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2012, Kyoto, Japan, January 17-19, 2012, pp. 1025- 1035. SIAM, 2012. doi: 10.113771.77816111773999 9781611973099.c 81.
URL https://ddoi.coooo http:///.oorrr//..o// 1137//1
Balcan, M. and Harvey, N. J. A. Submodular functions: Learnability, structure, and optimization. SIAM J. Comput., 47(3):703-754, 2018. doi: 10.1137//10888899, URL https://doi org/10. 10.
1137/120888909.
Balcan, M.. Constantin, F., Iwata, S., and Wang. L. Learning valuation functions. In Mannor, S., Srebro, N., and Williamson, R. C. (eds.), COLT 2012 The 25th Annual Conference on Learning Theory, June 25-27, 2012, Edinburgh, Scotland, volume 23 of JMLR Proceedings, pp. 4.144.24. pwwps/eesss /erees//2S/baalaan22///////
Jogg. 2012. URL http: http:://rroceedings.
a2b. paf.
Bian, A. A., Buhmann, M., Krause, A., and Tschiatschek, S. Guarantees for greedy maximization of non-submodular functions with applications. In International conference machine learning, pp. 498-507. PMLR, 2017.
Bian, Y., Buhmann, J., and Krause, A. Optimal continuous Subbuuuuu maximization and applications to provable mean field inference. In Proceedings the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 644-653. PMLR, 09-15 Jun 2019.
Bian, Y., Rong. Y.. Xu, Wu, Krause, A., and Huang, J. Energy-based learning for cooperative games, with applications to valuation problems in machine learning. In International Conference on Learning Representations, 2022. URL https: / hppp/pppeeveeeeeeeeeeeeeeeeeeeee eveereeview..... net heet/forumnId==xLEAAgCoo.....
Bilmes, J. A. and Bai, W. Deep submodular functions. CoRR, abs/1701.08939, 2017. URL org/aagggo /a//a/b/////// 08939.
http:// / arxxiv.vvvv. org
Calinescu, G., Chekuri, C., Pál, M., and Vondrák, J. Maximizing submodular set function subject to a matroid constraint. In International Conference on Integer Programming and Combinatorial Optimization, pp. 182-196. Springer, 2007.
Carreira-Perinnan, M. A. and Hinton, G. On contrastive divergence learning. In International workshop on artificial intelligence and statistics, pp. 33-40. PMLR, 2005.
Cremer, C., Li, X., and Duvenaud, D. Inference suboptimality in variational autoencoders. In International Conference Machine Learning, pp. 1078-1086. PMLR, 2018.
Dai. H.. Dai, B., and Song, L. Discriminative embeddings of latent variable models for structured data. In International conference on machine learning, pp. 2702-27 PMLR, 2016.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480) 2018
Dolhansky, B. W. and Bilmes, A. Deep submodular functions: Definitions and learn- ing. In Lee, D. D., Sugiyama, M., Luxburg, U. Guyon, I., and Garnett, R. (eds.), Advances Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3396-3404 2016. URL scttss:/prooeedinne neurips ctcpiaps.ccpppper/ee//ee /apppeer/016//aah.l Abstract. htmml. html.
Neural Information Processing Systems 29: Annual Conference on
Domke, J. Learning graphical model parameters with approximate marginal inference. IEEE Iransactions on pattern analysis and machine intelligence, 35(10):2454-2467 2013.
Garey, M. R. and Johnson, D. S. Computers and intractability, volume 174. freeman San Francisco, 1979.
Gareelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, J., Eslami, S., and Teh, Y. W. Neural processes, arXiv preprint arXiv:1807.01622. 2018.

---

Ghadimi, A. and Beigy, H. Deep submodular network: An application to multi-document sum- marization. Expert Syst. Appl., 152:113392, 2020. doi: 10.1016fj.swa.2220.1111 URL https://doi.one htggioogg/000 j essa. 2020 113392.
Gillenwater, J. A., Kulesza, A., Fox, E., and Taskar, B. Expectation Expectation-maxiiiztiin for learning determinantal point processes. Advances in Neural Information Processing Systems, 27:3149- 3157, 2014.
Gimeno, A., Ojeda-Montes, M. J..
S., Cereto-Massagué, A., Beltrán-Debón, R., Mulero, M., Pujadas, G., and Garcia-Valle Vallvé, S. The light and dark sides of virtual screening: what there to know? International journal of molecular sciences, 20(6):1375, 2019.
Gomes, J., Ramsundar, B., Feinberg, E. N., and Pande, V. S. Atomic convolutional networks for predicting protein-ligand binding affinity, arXiv preprint
2017.
Goodfellow, I., Bengio, Y., and Courville, Deep learning. MIT press, 2016.
Grathwohl, W., Swersky, K., Hashemi, M., Duvenaud, D., and Maddison, C. J. Oops i took a gradient: Scalable sampling for discrete distributions. arXiv preprint arXiv:2102.045000 2021.
Hinton, G. E. Training products of experts by minimizing contrastive divergence. Neural computation. 14(8):1771-1800, 2002.
Hoffman, M. D., Blei, D. M., Wang, C., and Paisley, J. Stochastic variational inference. Journal of Machine Learning Research, 14(5), 2013.
Horn, M., Moor, M., Bock, C., Rieck, B., and Borgwardt, K. Set functions for time series. In International Conference Machine Learning, pp. 4353 4363., PMLR, 2020.
Hyvärinen, and Dayan, P. Estimation of non-normalized statistical models by score matching.
Journal of Machine Learning Research, 6(4), 2005.
Iyer, R., Khargonkar, N., Bilmes, J., and Asnani, H. Generalized submodular information measures: Theoretical properties, examples, optimization algorithms, and applications. IEEE Transactions on Information Theory, 2021.
Jang. E., Gu, S., and Poole, B. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.
Jaynes, E. T. Information theory and statistical mechanics. Physical review, 106(4):620, 1957a.
Jaynes, E. Information theory and statistical mechanics. ii Physical review, 108(2):171, 1957b.
Jeffreys, H. An invariant form for the prior probability estimation problems. Proceedings of the Royal Society of London, Series A. Mathematical and Physical Sciences, 186(1007):433446,, 1946.
Ji, Y., Zhang, L.., Wu, J., Wu, B., Huang, L.-K., Xu, T., Rong. Y,, Ren, J., Xue, D., Lai, H., Xu, S., Feng, J., Liu, W., Luo, P., Zhou, S., Huang, J., Zhao, P., and Bian, Y. DrugOOD: Out- of-Distribution (OOD) Dataset Curator and Benchmark for Al-aided Drug Discovery - A Focus on Affinity Prediction Problems with Noise Annotations. arXiv e-prints, art. arXiv:2201.0966 January 2022.
Killamsetty, K., Sivasubramanian, D.,, Ramakrishnan, G., and Iyer, R. Glister: Generalization based data subset selection for efficient and robust learning. arXiv preprint arXiv:2012.10633 2020.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. 1rXiv:1412.6900 2014.
arXiv preprint
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint 2013.
14,
Kosiorek, R., Kim, H., and Rezende, D. Conditional set generation with transformers. arXiv preprint arXiv:2006.1684/ 2020.
12

---

Kothawade, S., Beck, N.. Killamsetty, K., and Iyer, R. Similar: Submodular information measures based active learning in realistic scenarios. Advances Neural Information Processing Systems, 34, 2021.
Krähenbühhl, P. and Koltun, V. Parameter learning and convergent inference for dense random fields. In International Conference on Machine Learning, pp. 113-521. PMLR, 2013.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., and Huang, F, A tutorial energy-based learning. Predicting structured data, 1(0), 2006.
Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., and Teh, Y. W. Set transformer: A framework for attention-based cormuuaaiooninvariii neural networks. In International Conference on Machine Learning, pp. 3744 3744-3753. PMLR, 2019.
Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, S. Multilayer feedforward networks with nonpolynomial activation function can approximate any function. Neural networks, 6(6):861-867, 1993.
Li S., Zhou, Xu, T., Huang. L., Wang. F., Xiong, H., Huang, W., Dou, D., and Xiong, H. Structure- aware interactive graph neural networks for the prediction of protein-ligand binding affinity. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp- 975-985, 2021.
Liu, W., Wang, X.. Owens, J. D., and Li, Y. Energy-based out-of-distrriutiio detection. arXiv preprint ar urXiv:2010.03759, 2020.
Liu, Z., Li, Y., Han, L., Li, J., Liu, J., Zhao, Z., Nie. W., Liu, Y., and Wang, R. Pdb-wide collection of binding data: current status the pdbbind database. Bioinformaties, 31(3):405-412, 2015a.
Liu, Z., Luo, P., Wang. X., and Tang, X. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015b.
Lyu, S. Interpretation and generalization of score matching. arXiv preprint arXiv: 2012.
Nelsen, R. B. An introduction to copulas. Springer Science & Business Media, 2007.
Nijkamp, E., Hill, M., Zhu, S.-C., and Wu, Y. N. Learning non-convergent non-persistent short-run meme toward energy-based model. arXiv preprint wrXhv:1904.09770, 2019.
Öztürk, H., Özgür, A., and Ozkirimli, Deepdta: deep drug-target binding affinity prediction. Bioinformatics, 34(17):i821-i22) 2018.
Rezatofighi, H., BG, V. K., Milan, A., Abbasnejad, E., Dick, A., and Reid, I. Deepsetnet: Predicting sets with deep neural networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5257-5266. IEEE, 2017.
Sahin, A., Bian, Y., Buhmann, J., and Krause, A. From sets to multisets: Provable variational inference for probabilistic integer submodular models. In Proceedings of the 37th International Conference on Machine Learning, volume 119 Proceedings of Machine Learning Research, pp. 8388 1388-8337. PMLR, 13-18 Jul 2020.
Shi, C., Luo, S., Xu, M., and Tang, J. Learning gradient fields for molecular conformation generation. arXiv preprint 2rriv:2105.000 009.03902, 2021.
Stigler, S. M. The history of statistics: The measurement of uncertainty before 1900. Harvard University Press, 1986.
Suh, S. and Choi, S. Gaussian copula variational autoencoders for mixed data. arXiv preprint UrXiv:1604.0400 2016.
Sun, S.HH. Multi-digit mnist for few-shot learning, 2019. URL. https:///itthen com/ shoohuo011f/Mattiiiiiiiii
13

---

Tran, D., Blei, D., and Airoldi, E. M. Copula variational inference. In Advances in Neural Information Processing Systems, pp. 3564-3572, 2015.
Tschiatschek, S., Djolonga, J., and Krause, A. Learning probabilistic submodular diversity models via noise contrastive estimation. In Artificial Intelligence and Statistics, pp. 770-779, PMLR, 2016.
Tschiatschek, S., Sahin, A., and Krause, A. Differentiable submodular maximization. arXiv preprint arXiv:1803.01185, 2018.
Wallach, I., Dzamba, M., and Heifets, A. Atomnet: deep convolutional neural network for bioactivity prediction in structure- drug discovery. arXiv preprint arXiv:1510.0280 2015.
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICMI.-I)) pp. 681-688. Citeseer, 2011.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
Xie, J., Lu, Y., Gao, R., and Wu, Y. N. Cooperative learning of energy-based based model and latent variable model via meme teaching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R., and Smola, A. Deep sets. arXiv preprint arXiv: 1703001000061111 2017.
Zhang, C.,, Bütepage, J., Kjellström, H., and Mandt, Advances variational inference. IEEE transactions on pattern analysis and machine intelligence, (188822008-2022 2018.
Zhang, D. W., Burghouts, G. and Snock, C. G. Set prediction without imposing structure as conditional density estimation. arXiv preprint arXiv:2010.04109, 2020.
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., and Torr, P. H. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE international conference on computer vision, pp. 1529-1537, 2015.
14


Reviews:
[{"review_id": "c_Vflp6072V", "reviewer": "Reviewer_vcKY", "summary": "This paper introduces EquivSet, an algorithm for learning set functions that satisfies the following desiderata: permutation invariance, varying ground set, minimum prior and scalability. EquivSet learns set functions under the optimal subset oracle using a maximum likelihood paradigm.  Specifically, the authors use an energy based model(EBM) to define the set mass function. The usage of EBM satisfies the minimum prior requirement.  A DeepSets style architecture is also used to satisfy the permutation invariance constraint on set functions and to also handle sets of arbitrary cardinality. However, learning EBMS over sets introduces some difficulties which the authors alleviate by proposing a mean field variational inference approach in the style of variational auto encoders. Amortization is also used to ensure scalability.", "questions": "My main question has to do with how Set Transformer performs compared to EquivSet as detailed above. ", "limitations": "Limitations are sufficiently outlined in the Limitations and Broader Impact section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method is well motivated and tackles the set representation learning problem from a perspective not yet considered.  The EMB approach and the variational inference approach introduce more complexity compared to the well used DeepSets and Set Transformer models.\n\nAdditionally, Set Transformer, which is trained similarly to DeepSets, is not used as a baseline. Set Transformer is a simple baseline that normally outperforms DeepSets and is easy to train. Hence i recommend the authors to include it in the baselines.\n\nFinally, it seems the SAB and ISAB layers of Set Transformer can be used in the model in place of the DeepSets style architecture. Many set representation papers show that these attention based backbones perform much better than DeepSets.\n\n**The authors have answered my questions satisfactorily hence i increase my initial score.**", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_l6z1hjBAcq", "reviewer": "Reviewer_6kuR", "summary": "This paper proposes a learning framework for set functions. It combines a set of procedures and methods to obtain a unique set of properties for the learned function as listed in introduction.", "questions": "Please address questions about the experimental results, and the assumptions about the data distribution.", "limitations": "Yes, the discussion seems adequate.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThis paper appears to be sound. It is well-written. Algorithms and procedures are clear.\n\nWeaknesses:\nMy main concern are the limited experimental results, and also the assumptions made about the data distribution and their applicability for practice. Authors have not discussed whether their assumptions about the data applies to the datasets they have used in their experiments. \n\nExperiments presented in Tables 2 and 3 seem to be limited. Can authors please explain how they chose these particular set of datasets? Results on more datasets could make the paper more convincing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PnvghDoczu1", "reviewer": "Reviewer_HBfR", "summary": "The authors tackle the task of set function learning under a weakly supervised setting, where the data points consist of sets and their optimal subsets. The proposed model combines an energy-based model with amortized variational inference to allow for several features (permutation invariance, varying grounds sets, a minimum prior approach, as well as scalability) and is further extended by a Gaussian copula variant to allow for correlation modelling. The model is evaluated on several data sets from different domains.", "questions": "-", "limitations": "Limitations and societal impact are discussed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The overall model is well motivated. Each of the desired requirements for the model is discussed in detail and tackled in a principle manner.\n- The overall storyline of the paper is well executed. The individual developmental steps and their problems are presented in a coherent structure.\n- The paper includes a wide range of experiments.\n\n## Weaknesses\n- The reported decimal places in Table 1/2/3 suggest a measurement precision that is not valid given the reported standard deviations, which give us a precision of only the first two decimals for most reported settings.\n\n## Other\n- The number of runs and whether standard deviation or standard error are reported should be included in the caption of the tables. The runs are mentioned in the text, but the deviation vs error is not at all discussed (also not in the appendix).\n- l302 claims \"two real-world datasets\". Why this specific formulation for MNIST/CelebA when the other experiments are also on real-world data?\n- (very minor) l78 \"delicate framework\": The wording here feels suboptimal. It can be read in its negative connotation of being a rather fragile framework that requires great care to be trained. The rest of the paper does not suggest this to be the case, allowing for the positive meaning of the word. However, in the introduction, the reader does not know yet which of the two meanings in his/her mind the paper should be read.\n\n\n### Typos\n- l41 lacked $\\to$ lacking", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PSxWjcpynK9", "reviewer": "Reviewer_3MWU", "summary": "The paper proposes a way to learn set functions (ie. find a subset $S \\subseteq V$ that maximizes some utility function $F_\\theta$ that we want to learn) when we are only giving examples of optimal subsets $S^*_i \\subseteq V_i$ from an optimal subset oracle. \n\nThis is different to other works that learn from function value oracles that provide actual utility values $f_i$ for specific subsets $S_i$.\n\nThe paper proposes a permutation-invariant architecture. It makes the problem tractable through mean-field variational inference and further amortizes inference via an additional neural network.\n\nIt reports great empirical results compared to several baselines.\n\n---\nThanks for addressing my questions and updating the paper!", "questions": "1. I believe the max entropy proof for the EBM (App B.1.) can be simplified considerably. In the case of $S \\mid V$, $S$ can be simply treated as a categorical with $2^{|V|}$ categories, and the softmax categorical distribution (EBM) is the exponential family distribution for that and maximum entropy as is well-known. This works because at this point, the internal structure of $S$ is not important. It's only when we look at different $V$ that you want to take its structure into account. Is this true?", "limitations": "The \"broader impact\" should be removed from the \"Limitations & Broader Impact\" paragraph title because it does not address broader societal impact (which is okay, but the title is misleading).", "rating": 7, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This was requested as an emergency review. I will try my best to offer a meaningful review (ofc), but I'm very much looking forward to discussing with the other reviewers.\n\nThe approach seems original (I am not very well acquainted with prior literature in this area though). The quality and clarity of the paper is great overall, and it seems like a significant contribution. \n\nI very much like the argument that it is easier to acquire data for the optimal subset oracle than the usual function value oracle in many practical cases as trying to obtain calibrated utility values from human labellers sounds improbable. Thus, I think this idea is of great practical importance.\n\nThe empirical validation looks sensible.\n\nI recommend acceptance with low confidence given the nature of this review and that my background is not aligned with the subfield.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4195mTmksG", "reviewer": "Reviewer_hn9G", "summary": "The paper addresses the problem of learning set functions. There are two variations of this problem, one is with function value (FV) oracle in which the supervised data takes the form of (set S_i, function value f_i of set S_i) and the goal is to learn the function mapping. One has to gather this info for large number of sets, making this process prohibitively expensive and often the FV oracle is not feasible. The main focus of the paper is on the variation of the problem with Optimal Subset (OS) oracle, wherein the data takes the form of $(V_i, S_i^*)$ pairs where $V_i $ is subset of $V$ (global candidate set) and $S_i^*$ is the optimal subset of $V_i$ that maximizes the utility. This setup implicitly captures the FV oracle and is more practical. \n\nGiven such data from the OS oracle, they propose a method based on variational inference to learn the mapping.  The first step is to cast the problem as maximum likelihood estimation by replacing the utility function $F_{\\theta}(S,V)$ with a probability distribution $P_{\\theta}(S|V)$ such that the probability of S given V is proportional to the utility. They list some basic (natural ) properties that such a distribution should satisfy a) permutation invariance, b) varying ground set etc. Further they want to have minimum prior assumption and scalable learning algorithm. \n\nThey give a distribution based on energy-based modeling (EBM) which satisfies the aforementioned properties and admits efficient training and inference. Directly learning with the EBM is difficult for reasons such as intractable partition function, to circumvent the difficulty they propose an approximate maximum likelihood learning using variational approximation of $P_{\\theta}(S|V)$ by product of independent Bernoulli distributions.\n\nThe method is evaluated on production recommendation, set anomaly detection, double MNIST, CelebA, Compund selection in AI-aided Drug Discovery and other real world datasets.\nThe results show that the proposed method significantly outperforms the baselines for learning set functions.\n", "questions": "Please address the weaknesses. \n", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nS1: It studies the problem of learning set functions under the OS oracle, which is practically important but has been addressed by limited number of works. \n\nS2: The reduction of this problem to maximum likelihood and later to variational approximation is novel and makes it amenable to application of this well-developed machinery.\n\nS3: The empirical evaluation is extensive and beats the closely related baseline comprehensively. \n\nWeaknesses:\nW1: There two steps of approximation that introduce their own approximation error. It is not clear how these are propagating and affecting the eventual solution. I believe with more approximation layers the error (sub-optimality) is higher.\n\nW2: Lack of theoretical results to shed light on the sub-optimality of the proposed algorithm. I think, the paper can be much better with these. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept