HORSE: Hierarchical Representation for Large-Scale Neural Subset Selection
Authors: Binghui Xie
Yixuan Wang
Yongqiang Chen
Kaiwen Zhou
Yu Li
Wei Meng
James Cheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Subset Selection
Set Encoding
Large-scale inputs
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Subset selection tasks, such as anomaly detection and compound selection in AI-assisted drug discovery, are crucial for a wide range of applications. Learning subset-valued functions with neural networks has achieved great success by incorporating permutation invariance symmetry into the architecture. However, existing neural set architectures often struggle to either capture comprehensive information from the superset or address complex interactions within the input. Additionally, they often fail to perform in scenarios where superset sizes surpass available memory capacity. To address these challenges, we introduce the novel concept of the Identity Property, which requires models to integrate information from the originating set, resulting in the development of neural networks that excel at performing effective subset selection from large supersets. Moreover, we present the Hierarchical Representation of Neural Subset Selection (HORSE), an attention-based method that learns complex interactions and retains information from both the input set and the optimal subset supervision signal. Specifically, HORSE enables the partitioning of the input ground set into manageable chunks that can be processed independently and then aggregated, ensuring consistent outcomes across different partitions. Through extensive experimentation, we demonstrate that HORSE significantly enhances neural subset selection performance by capturing more complex information and surpasses state-of-the-art methods in handling large-scale inputs by a margin of up to 20%.

HORSE: Hierarchical Representation for Large-Scale Neural Subset Selection
Binghui Xie, Yixuan Wang, Yongqiang Chen, Kaiwen Zhou, Yu Li, Wei Meng, James Cheng Department of Computer Science and Engineering The Chinese University of Hong Kong
Abstract
Subset selection tasks, such as anomaly detection and compound selection in Al-assisted drug discovery, are crucial for a wide range of applications. Learn- ing subset-valued functions with neural networks has achieved great success by incorporating permutation invariance symmetry into the architecture. However, existing neural architectures often struggle either capture comprehensive information from the superset or address complex interactions within the input. Additionally, they often fail to perform in scenarios where superset sizes surpass available memory capacity. To address these challenges, we introduce the novel concept the Identity Property, which requires models to integrate information from the originating set, resulting in the development of neural networks that excel at performing effective subset selection from large supersets. Moreover, we present the Hierarchical Representation of Neural Subset Selection (HORSE), an attention- based method that learns complex interactions and retains information from both the input set and the optimal subset supervision signal. Specifically, HORSE enables the partitioning of the input ground set into manageable chunks that can be processed independently and then aggregated, ensuring consistent outcomes across different partitions. Through extensive experimentation, we demonstrate that HORSE significantly enhances neural subset selection performance by capturing more complex information and surpasses the state-of-the-re methods in handling large-scale inputs by margin of up 20%.
Introduction
Set-valued functions are of great importance to wide range of real-world applications. For example, anomaly detection aims to identify a set of outliers from a larger dataset that could be users or financial transactions [Zhang et al.) 2020). Another example is the recommender system, where the objective is to identify a set of products that better satisfy customer preferences [Ou et al. [002]] In these scenarios, there a need for implicitly learning a function [Rezatofighi et al.) [017]] Zaheer et al. 2017] that quantifies the usefulness of a given subset of the inputs. The set function assigns a utility value each subset, and the subset with the highest utility corresponds to the most desired output.
To illustrate the concept, let us consider the task of a recommender system. In this task, aim to recommend a subset of items from larger item pool V, denoted as E V, that maximizes the utility of 5 with respect to the satisfaction of the customers. The utility can be captured by parameterized utility function, denoted as (S: V). and our goal is to optimize the following criteria:
S* arg max Fg (S;
(1)
SE2V
One straightforward method [Balcan and Harvey 2018) involves explicitly modeling the util- ity by learning the function U (VV using supervised data. This data consists pairs
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

EquiVSet
INSET
HORSE
S
S
S
1
L
1
400
aggregation
aggregation
Figure 1: Comparison of HORSE to the state-of-the-art, EquiVSet and INSET in handling subsets. "S" represents the supervision, indicating the specific subset of interest. "+" refers to the aggregation of different vectors, which is implemented through concatenation in practice. Unlike Equi VSet and INSET, the HORSE model captures more complex information from V by employing attention mechanisms. Furthermore, HORSE facilitates the division of V into distinct partitions.
f(S;, U,) where U, represents the actual utility value of subset S; given the respective item pool Vg. However, this training approach becomes challenging to implement due to the significant number of supervision signals required, which can be expensive and time-consuming to acquire (Ou
al. 2022]. To overcome this limitation, an alternative approach to tackle Eq. [ [using an implicit learning method from probabilistic perspective (Tschiatschek et al. 2018). This approach requires utilizing data in the form of {VV, >) where S represents the optimal subset corresponding to
V. The objective is estimate the parameters 0 such that Eq. holld for all possible (V,, S.). In practical training, with limited data sampled from the underlying distribution P(S, V), the empirical logllikelihood Σ [log Po (S)] is maximized for all the data pairs {S, V}, where VS is proportional to (S V) for all S E 2 (Ou et al. 2022 Xie et al. 2024 Additional details are available in Appendix [D.]
The crux of the matter lies in determining the structure of neural networks that can effectively model (S, V) throughout the entire process. One commonly employed approach in the literature is to utilize an encoder to generate feature vectors for each element in V. These vectors are then inputted into DeepSets [Zaheer al., [017], along with the corresponding supervised subset S, in order to learn the permutation invariant set function F(S). However, this methodology may neglect the interaction between S and V. leading
reduction in the expressive capacity of the models. In the previous study, Xic et al. (2024) suggest incorporating the sum-pooling representation of V into S to enhance the performance. Yet, the simple integration in Xie et al. 2024]. Wang et al. [2024b] limits its capacity to effectively model interactions among elements or subsets within these sets. Furthermore, the approach struggles with high-cardinality sets V, as encoding the entire set into memory may not be feasible [Bruno et al., 1).21)
To address these problems, and inspired by WWillette et al., 2023 introduce the notion of the Identity Property, a desirable concept for the effective functioning of the model F(S, V). Identity Property requires F(S, V) to accurately reflect which set V the information S originates from. In order to capture the interplay between S and V by adhering to the Identity Property, we propose a subset-based attentive set encoder. Additionally, this encoder facilitates the division of a large set V into smaller and manageable subsets. These subsets can be processed independently and later aggregated, ensuring no loss of information from V. Hence, our approach is able to efficiently handle seage-scalesae subset representation learning. As depicted in Figure [ our method is capable of modeling more complex information and managing large-scale inputs more effectively than the two state-of-the-art approaches in the field of Neural Subset Selection tasks, Equi VSet [Ou et al. 2022 and INSET [Xie al.. 2024].
In this work, we make several contributions to the field of neural subset selection, which can be summarized as follows:
We introduce and rigorously define a critical concept termed as the Identity Property for neural subset selection. This property requires that models can reliably determine the source

---

set V from which the information of the subset S is derived, which is a crucial requirement for neural subset selection tasks.
To adhere to the Identity Property and model complex interaction, we present a subset- based attention mechanism. This mechanism is crafted to learn the Hierarchical Representation of Neural Subset Selection, denoted as HORSE. Our theoretical analysis confirms that HORSE not only upholds the Identity Property but also maintains Permutation Invariance.
Through extensive empirical research, we validate the effectiveness of HORSE. Our exper- iments across variety of datasets demonstrate the consistently superior performance of HORSE. Additionally, we specifically explore HORSE's capabilities in large set environments, further showcasing its practical applicability and efficiency compared with the baselines.
2
Related Work
2.1
Set Encoding.
The exploration of network architectures tailored for set-structured inputs has become a vibrant area of research in recent years. A number of key studies [Ravanbakhsh et al. [017 Edwards and Storkey
2017 Zaheer et al. 2017 Qi al.) [017] Horn et al.) 2020 Floem-Reddy and Teh [020]] Wang et al. 2023] have laid the groundwork in this domain, primarily focusing creating models that are permutation equivariant using conventional feed-forward neural networks. These foundational models have been successful in universally approximating continuous permutation-invariant functions, primarily through the application of set-pooling layers to aggregate information across different elements of a set regardless of their order.
However, these methodologies have primarily concentrated on learning representations at the aggre- gate set level, paying less attention to more nuanced interactions occurring at elements. Recognizing this gap, more recent research efforts have aimed at introducing more sophisticated interaction modeling within invariant set funetions for various applications. A notable example is the work by (Lee et al.] 2019b). which incorporates self-attention mechanisms to facilitate the processing of elements within sets, thereby effectively capturing element-wise interactions. Moreover, the concept of Janossy pooling, proposed by [Murphy et al. 2018), introduced a novel approach to incorporate higher-order interactions within the pooling process. Since then, subsequent studies have built upon this advancement, leading to further refinements and innovations in the field, e.g.. [Kim 202T
al. 2020 Bruno al. 202 Willette al. 2023
2.2 Hierarchical Set Function.
The existing literature primarily concentrates on processing entire input sets, often overlooking the information provided by the sub-levels. Addressing this oversight, Maron et al. (2020) introduced an innovative approach that integrates the symmetry of clements to generate representations of an input set. This methodology was further expanded into broader context by Wang al. (2020). Moreover, Bevilacqua et al. [2022) proposed a novel framework aimed at enhancing graph representations by including whole graph representations to encode each subgraph. Along similar lines, Xie et al. 2024 developed an information aggregation module designed to learn F(S. V) effectively.
Despite these advancements, gap remains in the current re- search landscape. These meth- ods tend to overlook more com- plex interactions between ele- ments or subsets within sets. Fur- thermore, they often fall short in scenarios where the input set has a significantly large cardinality, indicating a need for more scal- ap-
Table 1: Properties of Various Methods: "Attn" indicates the use of the attention mechanism, "V" signifies the explicit utilization of information from V, and 'Large-scale denotes the capability of the methods generalize effortlessly to large-scale settings.
Model
Attn
Large-Scale
DeepSets 7aheer et 2017
Set Transformer Lee 2019a
Equi ESei Ou ct al. 2022
INSET [Xic ct al., 2024
able and interaction-sensittee proaches in set processing. In Ta-
HORSE
[T we compare our proposed method with importance basclines commonly used in subset selection tasks. Specifically, DecpSets
3

---

can handle large-scale settings but may lose complex information due to its simple pooling- based structure. Set Transformer excels at modeling complex information within sets but faces challenges with large input set sizes. Methods tailored for subset selection tasks, like Equi VSet and INSET, may struggle with learning intricate interactions and often overlook large-scale settings. HORSE designed to address these drawbacks.
2.3 Core Subset Selection
Recent work has focused extracting subsets from training datasets to decrease cost and improve effectiveness (Wei et al. 2013 Mirzasoleiman et al. [2200] Yang et al. 2023]. This research also highlights the importance of modeling the relationship between the original dataset and its subsets. Unlike neural subset selection, these core subsets are unlabeled, and typically, more data in the core subset enhances the performance of the models. Our approach differs in that our optimal subset is labeled within the training set, and its size constrained.
Method
3.1 Preliminaries
In this paper, we concentrate on the development neural networks for the purpose of modeling the hierarchical set function F(S, V), critical component for tasks involving sets, such as neural subset selection. For every ground set V, assumed to consist of n elements represented as V;, that is, V { .22 .2.,...,.F.... J. each element belonging to X' is characterized by a d-dimensional tensor. Typically, the ground set V can be conceptualized as an assembly of multiple disjoint subsets, explicitly V Si US2 U .....U Sm. where S, S, 0 for i + and each S; is subset in IR In this context, n, denotes the number elements in subset S,. Generally, C V acts as a supervisory signal in the form of mask over V to indicate the elements to be selected. For the sake of clarity, we define 5 as the concrete subset derived from this mask.
In the context of neural subset selection, the task entails the encoding of subsets S; into representative vectors to forecast the associated function value Y V. Traditional approaches, such as those documented in Zaheer et al. [2017] and Ou et al. (022), involve directly selecting S; based on the encoding embeddings all elements within V. subsequently feeding S; into feed-forward networks. Nonetheless, these methods model the function F F(S., V) solely based on the explicit subsets S,, potentially leading to suboptimal results due to the omission the broader context provided by the ground set V. This section introduces novel attention-based method for encoding subset representations, which distinctively incorporates information from the entire input set V, thereby enhancing performance.
3.2 Identity Property
To effectively model F(S, V), Xie et al. 2024 have proposed combine the representations of V and S. In practice, this involves utilizing two DeepSets architectures, as proposed by Zaheer et al. [201]]. to independently process S and before merging their outputs, as presented by Figure Given that set pooling operations process each element independently, certain information about the interactions among elements is inevitably lost, This omission can render some problems more challenging than necessary. To address this issue and facilitate the learning of complex interactions within sets, we introduce the following principles:
Property 3.1. Consider V E RRd and 5 C V where S E TRe assuming that V is partitioned into random collection of disjoint subsets V = =, U.S.S U.. Here, m varies within the range [1,n]] dependent on the chosen method of partitioning. The function F is said to satisfy the Identity Property if and only if there exist functions g and h such that F(h(S), h(V)) = F(((S), ((h(Si)....,,.(.$.....
(2) where g serves as an aggregation function that effectively combines the encoded representations of the subsets, ensuring that F leverages both the specific subset S and the ground set V through the transformations applied by h and the aggregation by g.
The method introduced by Xie et al. [2024] notable for satisfying the Identity Property through its utilization of sum- num-pooling simultancously process all elements. However, this approach may

---

not be practical for scenarios involving large inputs and may struggle to capture more complex information. response these limitations, we propose an attention-based method designed to fulfill the requirements of the Identity Property. Additionally, our interpretation of this property accommodates scenarios where S differs from the union of subsets {Si U S2 U... In practice, S is often chosen to be S1 for simplicity. This nuanced approach allows for greater flexibility and effectiveness in encoding set information, especially in complex or large-scale settings.
3.3 Attention-Based Set Representation
In this section, we introduce a formulation for an attention-based set encoding function F, leveraging the concept of partitions (referred to as slots in [Bruno et al.) 2021 Willette al.) 2023 ). Given
a ground set V E Rnxd, we randomly divide it into m subsets. For each subset, we allocate a unique embedding s; Rd. Furthermore, we establish Ç [s1. ........6 Sm,] "mm''' Similar to [Willette et al. 2023), J, we initialize Ç by sampling embeddings 8; from a parameterized Gaussian distribution with random initialization. Following this setup, we calculate the unnormalized attention scores between Ç and V, facilitating a dynamic weighting of elements within V based on their relevance to each partition's embedding. This process aims to capture the nuanced interrelations within subsets and between elements and their corresponding subsets.
as matrix in R=x,
q LN(,W)),
(3)
VW*
VW",
(4)
In this expression, "LN" represents Layer Normalization, and the matrices W E Rd.xd,, W Rdxdi, and W E Rdxda are introduced. These matrices serve to project V and Ç into a shared dimensional space d2. Subsequently, we employ dot product attention mechanism to assess the interactions between V and Ç. This process is governed by the specified formulation, strategically aligning elements of V with the embeddings in Ç through dimensional congruence, thus enabling nuanced, attention-dre driven analysis of set elements in relation to their partitioned subsets.
M = V dn gk²
(5)
= (M) E R=*"
(6)
where a denotes element-wise activation function. Utilizing the unnormalized attention scores, denoted by Å, we proceed to define the following mapping operation:
h nl (A)e.
In this context, h signifies transformation function mapping from Rad to mxdd, The term "nl" represents a normalization operation, defined as follows:
nl(A)as = Ais/ Σ A
(8)
which normalizes the column of A. Then, min, or max) across the columns of (VV) and select the sigmoid function for a, thereby establishing an attention mechanism akin to the SSE (Set Stream Embedding) method proposed by Bruno et al. (202)). However, this approach has its limitations. Given that the attention score nl A) is calculated independently of the other n- attention scores, is not feasible for the rows of nl(AA to form convex coefficients, unlike the softmax outputs typically observed in conventional attention mechanisms, as described by Willette et al. [2023].
apply a pooling function (such as mean,
To address this issue, we follow Willette et al. 2023 to aggregate information across all rows of ni( ()), thereby incorporating dependencies among different set elements into the attention mechanism. This is achieved through specific normalization process, outlined as follows:
M diag [nl( A)),,)
where 1, = (1,..., E Th represents vector of ones of dimension n, and M2 E Rm signifies a vector in m m-dimensionall space. Subsequently, can compute h(V) by applying the normalization
5

---

Permutation Invariance
Selected Subsets
Element Embedding
Intermediate Feature
Attention Mechanisms
Aggregation
Identity
Figure 2: This figure illustrates the HORSE model's capability to achieve Permutation Invariance and satisfy the Identity Property in subset selection tasks. It demonstrates that HORSE maintains consistent output despite the permutation of input set elements and the partition the ground set.
term M, as follows:
h(V) = Mnl(A)
(10)
Since S; a subset of based on partition method, hv (v.))) can derived from h(V). Detailed steps are provided in Algorithm| [ For simplicity, we omit V and use h(S;) going forward. This process ensures that h meets the criteria specified Eq. By constructing such an function, we ensure that the model recognize the input set V regardless of its partitioning, leading to the property that (((S))),(S))
h(S,)) yields the same value for any partition of V.
Furthermore, it facilitates the learning of interactions among the partitioned segments of V, essentially enabling the model to identify the characteristics of the input ground set V. Specifically, we concatenate h(S) with (() h(S2).... A(Sm). In practice, Multilayer Perceptron (MLP) is utilized to process the concatenated tensor into a vector Z E Rdo, achieving the following:
Z (((())))(((()))....))))))))))))) ((S)), A(S.)))) E I.
(11)
Given that the aforementioned process delineates the entire calculation in matrix format, which may be complex for some readers, we have taken steps enhance comprehension. To better illustrate how our method establishes an attention map between subsets S, we have detailed the procedural steps the Appendix (see Algorithm [). with a particular focus on the generation of ((S)). This will not only clarify the operational details but also emphasize the underlying methodology and thought process involved in constructing (S:).
Theoretical Results
In the realm of machine learning, particularly within the scope of set-based tasks, a fundamental requirement is the invariance to the permutation of input set elements. This characteristic ensures that the computation or outcome of task is unaffected by the order in which the set's elements are presented, principle that is especially pertinent to neural subset selection tasks. To address and formalize this aspect within the context of our proposed method, we present theorem that rigorously demonstrates the permutation invariance of our approach.
Theorem 4.1. Let Sn denote the set of all permutations of given set V. Since V E Raxdd is represented by a matrix, let πTy Ruxa be a random permutation applied V. Given that V represents a subset of V, the permutation Ty naturally induces a corresponding permutation Ts on S. Under these conditions, HORSE exhibits permutation invariance, which is defined as:
F(h(S),h(V)) F(h(TS S), hir VV)
(12)
Theorem 4. assures that irrespective of how the elements in the input set V are ordered, the output generated by HORSE remains consistent. This property crucial for ensuring the reliability and applicability of our method across wide spectrum of set-based tasks, where the inherent order of data points should not influence the task outcome. Furthermore, an illustration of the underlying concept is provided in Figure

---

Table 2: Product recommendation results for 12 different product categories, The best results are indicated in bold black, while the second-best results are highlighted in blue. Due to space limitation, we use Set-T to denote Set Transformer.
Categories
Random
PGM
DeepSet
Set-T
EquiVSet
INSET
HORSE
Gear
7.7
47.1
37.9
64.710.6
72.5+1.1
80.8
83.2:11.3
Bath
7.6
56.4+0.8
41.8+007
71,6+0.5
76.4+200
86.2:t0.5
87.6++1.0
Toys
8.3
4.41
42.1=0.5
62.5+2.00
68.410.4
76.9
77.4=10.9
Media
9,4
44.1
42.6:10.4
53.0+:2.0
55.4100.5
62.0::2.3
6..2:1
Safety
6.5
25.0::0.6
22.1:1044
23.410.9
23.1+2.0
23.8:11.5
26.9:11.2
Diaper
8.4
58.3+0.9
45.1=0.3
78.9
82.810.7
88.3+0.77
88.0=0.8
Health
7.6
44.9
45.2+1011
69.2
70.
81.2+0.5
81.6=0.66
Carseats
6.6
23.1+1.0
21.2++0.8
22.0+1.0
22.3+119
23.0+2.4
24.8
Bedding
7.9
48.5+0.6
48.1+0.2
76.242.2
76.2
85.7
87.1
Feeding
9.3
56.340.8
42.810.2
75.340.6
81.910.9
88.5
90.311.1
Apparel
9.0
53.3+t0.5
50.8+0.4
68.0+2.0
76.400.5
83.7+0.3
85,4+0.6
Furniture
6.5
17.5+0.7
16.8=0.2
17.6+0.8
16.2+2.0
16.7#3.5
18.1
Theorem 4.2. Ifa represents strictly positive element-wise activation function, then HORSE satisfies Property 3.7
By satisfying this property, HORSE ensures that its encoding captures both the individual characteris- ties of subsets within V and the overarching structure of the entire set, thereby facilitating a more nuanced and comprehensive understanding of set information. The proofs inspired by [Willette et al. 2023 and can be found in Appendix A
Experiments
In this section, we aim to demonstrate that HORSE significantly outperforms baseline models in suite of benchmarks tailored to Neural Subset Selection tasks. Subsequently, extend investigation to scenarios involving large-scale input settings. Due to the page limitation, we have included additional experiments in Appendix
Evaluations. To assess the performance various methods, we employ mean Jaccard coefficient (MJC) as the evaluation metric. This metric quantifies the similarity between the predicted subset S" and the true subset S for each data sample (S*, V). The Jaceard coefficient calculated as follows: s"r'' JC(S*, S') S*'S' where the intersection and union operations determine the size of the overlap and the total unique elements in both sets, respectively. The MJC is then derived by averaging the Jaccard coefficient across all test set samples. Please note that all the following reported performance metries are presented in percentages, with a default multiplication factor of 100%.
Baselines. We conducted experiments compared with several approaches: Random, PGM (Tschiatschek et al. 2018), DeepSet Zaheerr al. 2017]
Table Performance results on the Two-Moons and Gaussian- datasets. Bolded numbers denote the best performance on each dataset
Method
Two Moons
Gaussian Mixture
Set Transformer [Lee et al. 2019a].
EquiVSet [Ou al., 2022]. and IN-
Random
5.5
5.5 43.8 + 0,9
SET [Xie et al. [202]]. The Random approach represents the expected per- formance of a random guess. DeepSet and Set Transformer are well-known methods or frameworks that satisfy permutation invariance, making them suitable for Neural Subset Selection tasks. PGM, EquiV and INSET are specifically designed for subset selection tasks. More comprehensive details available in Appendix
PGM
36.0 + 2.0
DeepSet
47.2 - 0.3
44.6 - 0.2
Set Transformer
57.4 0.2
90.5 + 0.2
Equi
58.5 0.3
90.7 0.2
INSET
58.2 - 0.3
90.9 + 0.2
HORSE
60.2 + 0.5
91.8 + 0.2

---

Table 4: Empirircal results of compound selection Tasks. Bolded numbers denote the best performance on each dataset. Due to space limitations, we use "Set-T" denote Set Transformer.
Random
PGM
DeepSet
Set-T
EquiVSet
INSET
HORSE
PDBBind
9.9
91.0+110
90.1
91,9
92.411.1
93.5+0.8
94.1 0.7
B
9.0
69.0=2.0
71.0+2.0
71.5+1.0
72.1
73.4+1.0
74.2 - 1.1
PDBBind-2
7.3
35.0:10.9
32.31:.4
35.5+11.0
35.7.:0.5
37.141.0
43.2 - 0.6
BindingDB-2
2.7
17.6:10.6
16.5+..55
18.3+0.4
18.8:00.6
19.8+0.55
21.3 + 0.5
Average
7.23
53.15
52.48
54.30
54.75
55.95
58.20
5.1 Synthetic Experiments
Firstly, We validate the effectiveness of our models through experimental trials focused learning set functions, using two synthetic datasets: the two-moons dataset [Pedregosa et al. 2011 with an added
noise variance of x² 0.1, and and Gaussian mixture represented as V (No. ))
[ E).
Take the Gaussian mixture as an example, the data generation procedure as follows: i) Initially, we select an index, denoted b, using Bernoulli distribution with probability of }. ii) Subsequently, we sample 10 points from the Gaussian distribution N (Ju,, Σ) to construct the set S iii) Further, we sample 90 points for V from the Gaussian distribution N ( E). We follow the procedure of Ou et al. [2022] to obtain 1,000 samples, subsequently divided into training, validation, and test sets. We effectively demonstrate the efficacy of approach in mastering complex functions. Detailed results can be found in Table
5.2 Product Recommendation
The task involves recommending the most suitable subset of 30 products customer within specific category. For this experiment, we utilize the dataset from the Amazon baby registry, sourced from Gillenwater et al. 2014a]. This dataset includes numerous product subsets chosen by various customers, with Amazon categorizing each item on a baby registry into specific categories such as "Bath". "Health" and "Feeding". Detailed information can be found Appendix D
Table 2 presents the performance of all models across different categories. Notably, out of the twelve cases evaluated, HORSE outperforms other models in 11 of them. Even in the Diaper category, our method achieves results that are comparable to INSET, which is noteworthy. These significant improvements highlight the effectiveness and superiority HORSE. Specifically, while Equi and INSET struggle to surpass classical neural subset selection baselines in the Safety, Car Seats, and Furniture categories, HORSE consistently outperforms all baselines notable manner.
5.3 Compound Selection in Al-aided Drug Discovery
In drug discovery, the screening of compounds with diverse biological activities and favorable ADME (absorption, distribution, metabolism, and excretion) properties is critical step [Li et al., 202
Ji et al.) 2022, Gimeno et al., 2019]. Virtual screening typically involves sequential filtering process that employs multiple essential filters. However, neural networks encounter challenges when learning the complete screening process. This difficulty arises from the absence of intermediate supervision signals, which can be costly or impossible to obtain due to pharmaceutical protection policies. Therefore, we implement single filter, namely, high bioactivity, to obtain the optimal subset of compound selection, following the methodology in [Ou et al. 2022]. Our experiments are conducted two datasets: PDBBind [Liu et al., 2015a| and Binding DB [Liu al. 2007). To be more practical, we further enhance our approach with the inclusion two filters: the high bioactivity filter and the diversity filter. This extended analysis is denoted PDBBind-2 and BindingDB-2, representing the two-stage filtering process, for a more practical perspective.
Table
that our method outperforms the baselines and significantly surpasses random guessing, especially on the BindingDB-2 and PDBBind-2 datasets. However, the improvement on PDBBind and BindingDB is less significant. This marginal enhancement is due to the informative structural characteristics of complexes (the elements within set), which inherently provide substantial information for this task. Consequently, the model can effectively predict the activity values of

---

Moons
Gaussian
BindingDB
HORSE
72-
30
Transformer
70
MJC
MJC
86
48
84
40
200
400
0.0
500
1000
200
400
500
000
1000
10
20 20
Set Size
Set Size
Partttons
Figure 3: The performance of the methods on the Two-Moons and Gaussian Datasets with respect to the set size is examined in the left two subfigures. These subfigures provide insights into how the performance of the methods varies as the size of the input sets changes. The right subfigure focuses on the influence of the number of partitions on the performance using the BindingDB dataset.
complexes even without explicitly considering the interactions between the optimal subset and its complement in single filter scenarios. Nonetheless, our method still achieves superior results compared to other methods, confirming its effectiveness.
5.4 Large-Scale Setting
Set encoding mechanisms, as introduced by Zaheer et al. 2017 and further explored by Lee et al. [2019a]. have fundamentally shifted the way neural networks perceive and process sets by emphasizing the importance of permutation invariance and the ability to handle variable-sized inputs. These models are designed learn from the entire set in single gradient step, ensuring that the learned representations encapsulate the holistic properties of the set. However, this approach encounters practical limitations when dealing with large-scale sets, where processing the entire set in a single step becomes computationally infcasible due to memory constraints or the sheer volume of data.
To circumvent these challenges, an effective strategy involves training models on partitions of the set, sampled dynamically each iteration of the optimization process (Lee et al. 2019a Wang et al.,
2024a]. This method allows for manageable subsets to be used for training, significantly reducing
the computational load. However, this method will lose information (Bruno et al.] [202] Willette et al. 2023) since it does not process all the elements from V. Therefore, we instead partition the elements into mini-batches, independently encode each batch, and aggregate them obtain single set encoding. By applying this methodology across both baseline models and HORSE in scenarios characterized by large-scale input sets, we can highlight the efficiency and scalability of our proposed solution. We conducted experiments on the Two-Moons and Gaussian- Mixtture datasets. To ensure consistency, we set the size of the optimal subset S* to be 10. Subsequently, we varied the size of the input ground set V within the range of {200, 400, 600, 800, 1000]. Notably, the memory capacity of the GeForce RTX 3090 is insufficient when the size reaches 600. The ground set was divided into 5 disjoint partitions, with each partition containing one-fifth of the elements in V. For the purpose of comparison, we selected INSER and Set Transformer as baselines alongside our proposed method, HORSE. INSET demonstrated the best performance among baselines, while Set Transformer an alternative method that incorporates an attention mechanism. The results obtained from these experiments are presented in the left two subfigures of Figure 3 It is evident that HORSE outperforms both INSER and Set Transformer by significant margin, demonstrating its superior effectiveness in handling large-scale sets.
Furthermore, to further enhance the practicality of our approach and investigate the potential impact of the partition numbers on the results, we conducted additional experiments on the BindingDB dataset. In this experiment, we set the size of the optimal subset S" to be 15, while the size of the ground set remained fixed at 1000. We partitioned the ground set into a range of 2 to 50 partitions. The results of these experiments are presented in the right subfigure of Figure 3 Remarkably, becomes evident that HORSE exhibits remarkable robustness with respect to the number of partitions considered. Regardless of the specific partitioning scheme employed, HORSE consistently delivers exceptional performance, which is more robust than our baselines.

---

6 Conclusion
In this paper, we have addressed the limitations observed in existing methods for neural subset selection tasks. These methods often struggle to effectively model complex information and lack scalability when dealing with large-scale inputs. To overcome these challenges, we propose an innovative and scalable approach called HORSE, which leverages the power of the attention mechanism. Theoretically, we establish that HORSE satisfies the Identity Property and Permutation Invariance, ensuring its soundness and effectiveness. Empirically, we thoroughly evaluate the performance of HORSE against various baselines in both standard and large-scale settings.
Limitation and Future Work. Our theoretical and empirical results demonstrate how the attention mechanism enhance models for neural subset selection tasks in both standard and large-scale settings. However, in large-scale scenarios, our support is currently limited to a theoretical framework for partitioning the set into different groups within synthetic distributed setting, rather than practical experimentation in real distributed environment. Moving forward, we plan to implement and test our model in more practical, real-world scenarios to further validate its effectiveness.
7 Acknowledgements
We thank all the reviewers for their valuable comments. This work was supported by Research Grants 8601116, 8601594, and 8601625 from the UGC of Hong Kong.
References
M. Balcan and N. J. A. Harvey, Submodular functions: Learnability, structure, and optimization.
SIAM J. Comput., 47(3):703-754, 2018.
B. Bevilacqua, F. Frasca, D. Lim, Srinivasan, C. Cai, Balamurugan, M. M. Bronstein, and
H. Maron. Equivariant subgraph aggregation networks. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
B. Bloem-Reddy and W. Teh. Probabilistic symmetries and invariant neural networks. Mach. Learn. Res., 21:90:1 1-::611, 2020.
A. Bruno, J. Willette, J. Lee, and S. J. Hwang. Mini-batch consistent slot set encoder for scalable set encoding. Advances in Neural Information Processing Systems, 34:21365-21374 2021.
H. Edwards and A. J. Storkey. Towards neural statistician. In 5th International Conference on Learning Representations, ICIR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017
J. Gillenwater, A. Kulesza, E. B. Fox, and B. Taskar. Expectation-maxiin maximization for learning determinantal point processes. In Advances in Neural Information Processing Systems 27: Anmual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3149-3157, 2014a.
J. A. Gillenwater, A. Kulesza, E. Fox, and B. Taskar. dnpectation-maximizzzee for learning determinantal point processes. Advances in Neural Information Processing Systems, 27, 2014b.
A. Gimeno, M. J. Ojeda-Montes, S.
A. Cereto-Massagut R. Beltrán-Debee M. Mulero, G. Pujadas, and Garcia- Garcia-Vallvé. The light and dark sides virtual screening: what is there to know? International journal of molecular sciences, 20(6):1375, 2019.
M. Horn, M. Moor, C. Bock, B. Rieck, and K. Borgwardt. Set functions for time series. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4353-4363. PMLR, 13-18 Jul 2020.
Y. Ji, L. Zhang, Wu, Wu, L. Huang, T. Xu, Y. Rong, L. Li, J. Ren, D. Xue, Lai, Xu, J. Feng, W. Liu, P. Luo, S. Zhou, J. Huang, P. Zhao, and Y. Bian. Drugood: Out-of-distribution (OOD) dataset curator and benchmark for ai-aided drug discovery focus on affinity prediction problems with noise annotations. CoRR, abs/2201.09637, 2022.
10

---

M. Kim. Differentiable expectation maximization for set representation learning, In International Conference on Learning Representations, 2021.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh. Set transformer: A framework for attention-basee permutation-inwe -nnvariinnn neural networks. In International conference on machine learning, pages 3744-3753. PMLR, 2019a.
Lee, Lee, J. Kim, A. R. Kosiorek, S. Choi, and Teh. Set transformer: framework for attention-based permutation-invariaav neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 3744-3753 PMLR, 2019b.
S. Li, J. Zhou, T. Xu, L. Huang, F. Wang, H. Xiong, W. Huang, D. Dou, and H. Xiong. Structure- aware interactive graph neural networks for the prediction of pootein-ligand binding affinity. In KDD '21: The 27th ACM SIGKDD Conference Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021, pages 975-985. ACM, 2021.
Y. Li, H. Yi, C. Bender, S. Shan, and J. B. Oliva. Exchangeable neural ode for set modeling. Advances in Neural Information Processing Systems, 33:6936 6946, 2020.
Liu, Y. Lin, X. Wen, N. Jorissen, and M. Gilson. Bindingdb: web-accessible database of experimentally determined protein-ligand binding affinities. Nucleic acids research, 35(suppl_ D198-D201, 2007.
Z. Liu, Y. Li, L. Han, J. Li, J. Liu, Z. Zhao, W. Nie, Y. Liu, and R. Wang. Pdb-wide collection of binding data: current status of the pdbbind database. Bioinformaties, 31(3):405 4125544422220 2015a.
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015b.
H. Maron, O. Litany, G. Chechik, and E. Fetaya. On learning sets of symmetric elements. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages PMLR, 2020.
B. Mirzasoleiman, J. Bilmes, and J. Leskovee. Coresets for data-efficient training of machine learning models. In International Conference Machine Learning, pages 5950-6960 PMLR, 2020.
R. L. Murphy, B. Srinivasan, V. Rao, and B. Ribeiro. Janossy pooling: Learning deep permutation- invariant functions for variable-size inputs. arXiv preprint /111:/99 2018.
Z. Ou, T. Xu, Q. Su, Y. Li, P. Zhao, and Y. Bian. Learning neural set functions under the optimal subset oracle. NeurlPS, 2022.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830 2011.
C. R. Qi, H. Su, K. Mo, and L. Guibas. Pointnet: and segmentation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 77-85. IEEE Computer Society, 2017.
Deep learning point sets for 3d classification
S. Ravanbakhsh, J. Schneider, and B. Poczos. Equivariance through parameter-sharing, In Interna- tional conference on machine learning, pages 2892-2901. PMLR, 2017.
S. H. Rezatofighi, V. K. BG, A. Milan, E. Abbasnejad, A. Dick, and I. Reid. Deepsetnet: Predicting sets with deep neural networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5257-5266. IEEE, 2017.
11

---

S.-H. Sun. Multi-digit mnist for few-shot learning, 2019.
S. Tschiatschek, A. Sahin, and A. Krause. Differentiable submodular maximization. arXiv preprint arXiv:1803.01785, 2018.
R. Wang. M. Albooych, and S. Ravanbakhsh. Equivariant networks for hierarchical structures. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 13806-1388 Curran Associates, Inc., 2020.
Z. Wang, X. J. Wang, Y. Kuang, M. Yuan, J. Zeng. Y. Zhang, and F. Wu. Learning cut selection for mixed-integer linear programming via hierarchical sequence model. In The Eleventh International Conference on Learning Representations, 2023. URL htttp:://ppppreeeeeeeeee Zob4P9bRNcKK
net/ net/foruml id=
7. Wang. Q. Jie Wang. Y. Bai, X. Li, J. Lei Chen, M. Yuan, Li, Zhang, and Wu. Towards next-generation logic synthesis: A scalable neural circuit generation framework. In The Tiihtt-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Z. Wang. J. Wang, D. Zuo, Y. J. Xia, Y. Ma, J. Hao, M. Yuan, Y. Zhang, and F. Wu. A hierarchical adaptive multi-task reinforcement learning framework for multiplier circuit design. In Forty-first International Conference on Machine Learning. PMLR, 2024b.
K. Wei, R. Iyer, and J. Bilmes. Submodularity in data subset selection and active learning. In International conference on machine learning, pages 1954-1963. PMLR, 2015.
Willette, Lee, Andreis, Kawaguchi, J. Lee, and Hwang. Scalable set encoding with universal mini- consistency and unbiased full set gradient approximation. In International Conference on Machine Learning, pages 17008-37044 PMLR, 2023.
H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, arXiv preprint arXiv: (0t8.:1770007700 2017.
B. Xie, Y. Bian, K. zhou, Y. Chen, P. Zhao. B. Han, W. Meng, and J. Cheng. Enhancing neural subset selection: Integrating background information into set representations, 2024.
Y. Yang, H. Kang, and B. Mirzasoleiman. Towards sustainable learning: Coresets for data-efficient deep learning. In International Conference Machine Learning, pages 39314-39330. PMLR, 2023.
M. Zaheer. S. Kottur, S. Ravanbakhsh, B. Póczos, R. Salakhutdinov, and A. J. Smola. Deep sets. In Advances Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3391-3401, 2017.
D. W. Zhang, G. J. Burghouts, and C. G. Snoek. Set prediction without imposing structure as conditional density estimation, arXiv preprint arXi::2010.041009, 2020.
12

---

A Proof
A.1 Proof of Theorem 4.1
Proof. Consider a given set V =
E Raxd where n represents the number of
elements in V. Let represent the set of all permutations of V. Now, suppose Ry denotes random permutation applied to V. By utilizing this permutation xy, we can construct a permutation matrix M E R=xx
T mv (1)
MV
(n)
Assuming the use an activation function a that is strictly positive for each element,
(VV=--g(MVVWk)) avvd- ( MT)
=o(vd-I gk )M
AM.
The normalized attention score nl(A) can be computed using the given permutation my, resulting in
(1)x Eim Asme()
A Dt ALt(m))
n((AMT)= =
A Ei A n((A)MT.
An.(No// Exer ALm(n)
(13)
Now, we consider the matrix multiplication of
nl (l)()),
nl al(A)),
n((A)TT=
in(AA) (1))
nl(A) ((()), (())
and
x wV
MVWV
xx() WV
Since
nil (n((A)
nl(A)
x TTtn WV
nl(A),.T())
nl(A)a.
a WV
equal to
Σg nl( I)(
Σnn ml(A))m. wr
13

---

which can also be formulated as:
Σy nl(A)1,33 WV
Σソ= nl( nl(A) wV
nl(A)..
(14)
Therefore, nl(A)u is permutation invariant under the permutation group of S. Since
nl( nI(A)1, = -=nn nl) = Σ nl( (l)
jml
thus diag (nl(A)1., A) the conclusion that
is also invariant with respect to the permutation of input V, which leads to
h(MV) h(V).
Similarly, we can construct the permutation matrix E Re** for given and permutatation ms, such that:
rs (1)
MS =
(
with the same process as Equation 13 and 14 we can have the following conclusion:
F(M(S), F(h(Fs S), V)).
B Proof of Theorem 4.2
Proof. Consider the input set V e paxd and let V = S₁ U U... Sm represent partition of V =n;- In other words, V can be expressed as the union all S; and each S; is disjoint from make the assumption that.
Sm
with |S,| = n2,
Sj for + Without loss of generality,
S
S.W
k: = VW
w= VW"
Sm W
Sm W
where S.W* e Rau and S.W" E Ru: for = 1,2.... .,. m. Then we can express the matrix nl( as follows:
nl(A) = [n)(A()).... -..((((m)))]
where A = -: ((,W*)) E mxn for 1,2.... ... m since nl(A),, (A)) is independent ((()), for all * j.
Since
S
= (n(((()))) ... nl( [mm))
Sm w
14

---

the following equality holds
m
h
nl S
(15)
==1
Since
M; =
nl(40) )55
1=1 f=1
can decompose nl( nl(A)1n
nl 1l())
nl nl
(A(( )m
mm))
1=1
==1
(16)
m Σ nl(A 1..
where n, = (I,...,,1) E R Combining Equation 15 and
m
I(V) diag
nl nl(A())) 1mm
Σ nl (l(A())))
We define (1:) nl ) and h2()))
n(A()) Moreover, h(S,)
diag(hi hh Now, we define a function,
g((h(S)),.... ...., )).
g2(()((S)),. ((([((()),....,,(((m))
where
-1
m
91 (() diag
h1 (St)
h2())).
Then h(V) ((.((S)),...,1(((,..,)) Since the partition is arbitrary, satisfies Property 3.1
15

---

C Pseudo Code of HORSE
In the main text, we present HORSE using matrix calculations, which may be challenging to compre- hend. To improve understanding of how method establishes attention map between subsets S., we detail the procedural steps Algorithm I with a special emphasis on the generation h(S.) This approach is designed to elucidate the operational details and highlight the methodology involved constructing (()))
Algorithm 1 HORSE. V (Si, S2, Sm is the input set partitioned into subsets, & E IR mR the initialized embedding and g is the choice of aggregation function.
Input: V {Si, S...... Sm S S1, S Rmd,, .9
2: Output: 7 E R.d
3: Initialize C
4: q == LN ((W9)
5: for i = 1,2, m do
6: k; S.Wk
7:
S.W"
8: end for
9: & kkg k2
km T
= U1, 02
Umn
11: M V Th
9l:
12: Å (M1)
13: A nl(A) 1A,, A2 Am) ]
14: M diag(nI(A))..
[M,,... M
15: for i = 1,2...... do
16: h(S.) = M,A.S.W"
17: end for
18: S, = g(h((S),... hh(S)))))
19: Z F(h(S), S.)
return Z
D Experimental Details and Additional Experiments
D.1 Detailed Description of Tasks.
Product Recommendation. The task involves recommending the most suitable subset 30 products customer within a specific category. For this experiment, utilize the dataset from the Amazon baby registry, sourced from Gillenwater et al.) 20144a.. This dataset includes numerous product subsets chosen by various customers, with Amazon categorizing each item baby registry into specific categories such as "Bath", "Health" and "Feeding". Additionally, each product is represented by 768-dimensional vector generated by a pre-trained BERT model, based on textual description.
The Amazon baby registry data Giillenwater et 2014b comprises various datasets collected from Amazon, encompassing different categories such as toys, furniture, and more. Within each category, there exist VV sets of products that have been selected by different customers. To create sample (S*, V), we follow a specific procedure. Initially, we remove any subset with an optimal subset size [S*] greater than equal to 30. The remaining subsets are then divided into training, validation, and test folds using a 1:1:1 ratio. Additionally, we randomly select an additional 30- |S* products from the same category construct (S*, V). This process allows to create data point (S* V). For comprehensive information, please refer to Table in Ou et al. [2022). which presents the statistics of the categories.
Set Anomaly Detection. We tackle set anomaly detection tasks on four real-worke datasets: double
MNIST [Sun] 2019, CelebA [Liu et al. 2015b), F-MNIST [Xiao al., [017], and CIFAR-10
Krizhevsky 2009 Each dataset is partitioned into training, validation, and test sets, each comprising
16

---

Table 5: The statistical properties of the Amazon product dataset.
Categories
D
[
Σ Es
E||S"]
mins- |S*
maxs-
[SS
Gear
4,277
30
16,288
3.80
3
10
Bath
3,195
12,147
3.80
3
11
Toys
2,421
9,924
4.09
3
14
Media
1,485
30
6,723
4.52
3
19
Safety
267
30
846
3.16
3
5
Diaper
6,108
30
25,333
4.14
3
15
Health
2,995
11,053
3.69
3
Carseats
483
30
1,576
3.26
3
6
Bedding
4,524
30
17,509
3.87
3
12
Feeding
8,202
30
37,901
4.62
3
23
Apparel
4,675
30
21,176
4.52
3
21
Furniture
280
892
3.18
3
10,000, 1,000, and 1,000 samples, respectively. each dataset, we randomly select n images from the dataset create the OS Oracle S where be either 2, 3, or 4. This setup aligns with the approach outlined in Zaheer et al. 2017 Ou et al. 2022].
Let's take CelebA an illustrative example. In this scenario, the goal is to identify anomalous faces solely through visual observation, without using any attribute values. The CelebA dataset consists of 202,599 face images, each annotated with 40 boolean attributes. When constructing sets, for each ground set V. we randomly select n images from the dataset create the OS Oracle S", ensuring that none of the selected images contain any of the two attributes. Additionally, we ensure that no individual person's 's face appears both the training and test sets.
Regarding the results presented in Table 6 it is evident that our model exhibits a significant per- formance advantage over all the baseline methods. This substantial improvement underscores the superior capabilities of our model in addressing the given task.
Compound Selection in Al-aided Drug Discovery. In drug discovery, the screening of compounds with diverse biological activities and favorable ADME (absorption, distribution, metabolism, and excretion) properties critical step [Li et al.) 202T Ji et al. 2022] Gimeno et al., 2019]. Virtual screening typically involves sequential filtering process that employs multiple essential filters. These filters initially select diverse subsets from highly active compounds and subsequently eliminate compounds with unfavorable ADME characteristics. After passing through several filtering stages, an optimal subset of compounds identified. However, neural networks encounter challenges when learning the complete screening process. This difficulty arises from the absence of intermediate supervision signals, which can be costly or impossible to obtain due to pharmaccutical protection policies. Consequently, models are expected to learn this intricate selection process in end-to-end manner. In other words, models must predict S* based solely on the optimal subset supervision signals, without knowledge of the intermediate steps. Therefore, simulate the optimal subset oracle compound selection by applying one or two filters by uising PDBBind and BindingDB, as [Ou al. 2022].
PDBBind offers an extensive compilation of experimentally measured binding affinity data for biomolecular complexes. We utilized the "refined" portion of the complete PDBBind dataset, which consists of 179 complexes, to construct our subsets. To create a data point (V,S*, we randomly sampled 30 complexes from the dataset to form the ground set V. The subset S* was then generated by selecting the five most active complexes within V. We constructed separate training, validation, and test splits, comprising 1000, 100, and 100 data points, respectively.
BindingDB is an openly accessible database that provides measured binding affinities for collection of 52, 273 drug-target pairs involving small, drug-like molecules. Similar to PDBBind, we randomly selected 300 drug-target pairs from the BindingDB database to form the ground set V. From this dataset, we carefully identified the 15 most active drug-target pairs and designated them as S*. To ensure comprehensive evaluation and robust model training, we subsequently created distinct training, validation, and test sets. These comprised 1000, 100, and 100 data points respectively.
17

---

Table 6: Empirical results of set anomaly detection Tasks. Bolded numbers denote the best perfor- mance. HORSE outperforms all the baselines on the four datasets.
Random
PGM
DeepSet
Set-T
EquiVSet
INSET
HORSE
Double MNIST
8.2
30.0=11.0
11.1
01.2++0.5
57.5#1.8
69.7+1.0
72.3 1.2
CelebA
2.2
48.1=0.6
44.0##..6
52.7=0.8
54.9+0.55
57.5#1.2
59.3 + 1.0
F-MNIST
1.9
54.0:12.0
49.0+2..0
58.1
65.0+1..0
70.1:2.1
73.5 1.6
CIFAR-10
1.9
45.0::2.0
32.0+0.8
65.0+2..
60.0+112
71.2412.11
74.3 + 1.2
Average
3.55
44.28
34.03
56.75
59.35
67.13
69.85
D.2 Descriptions of Baselines
Random. This represents the expected performance of random guess, serving as baseline to help us gauge the actual difficulty of the tasks.
PGM [Txchiatschek el al.) 2018/. PGM, which stands for Probabilistic Greedy Model, tackles the optimization Problem b employing a differentiable extension of the greedy maximization algorithm. For deeper understanding of this approach, please refer to the origianl paper or Appendix A in Ou et al. 2022].
SeepSet [Zaheer et al. 2017). Here, employ DeepSet baseline model. We use DeepSet to predict the probabilities of including specific instances in S essentially learning an invariant permutation mapping from the power set 2 to the interval [0, 1] of size [V]. This model was also used as a backbone for set function learning in EquiVSet. Moreover, it suitable for subset selection tasks, as detailed in its original paper.
Set Transformer (Lee et al.) 22019a.. Set Transformer extends DeepSet's capabilities by integrating the self-attention mechanism. This addition allows the model to consider pairwise interactions between elements, enabling to capture dependencies and relationships among different elements more effectively. It can be also utilized for subset selection tasks, similar to DeepSet.
EquiVSet [Ou et al. )022). J. EquiVSet employs an mveryy-based model (EBM) to establish the set mass function, denoted as P)S from probabilistic standpoint. Their primary objective lies in learning distribution VV) that monotonically increases with respect to the utility function F(S, V). It's worth noting that their framework focuses approximating the symmetric function F(S) rather than the symmetric function F(S,V), with DeepSet serving as the foundational component their model to approximate the set function.
INSET [ie et al. 2024]. As discussed in the Introduction, EquiVSet faces limitations incorporating information from the ground set V. In response to this challenge, Xie et al. [2024) present an innovative solution. They propose the generation of embeddings for V and subsequently concatenate these embeddings with the representations of S, which has been presented in Figure
Among these baselines, DeepSets and Set Transformer are two crucial model structures widely used in set-based tasks, including subset selection tasks. On the other hand, PGM, Equi and INSET methods specifically designed for neural subset selection tasks. Notably, both INSET and HORSE are implemented based on the EquiVSet framework, yet they significantly outperform EquiV
D.3 The Objective of Neural Subset Selection in Optimal Subset Oracle
Our method, HORSE, is applicable for learning F(S, V) across a range of tasks. In this paper, we primarily utilize the framework established in (Ou et al. 2022] and modify it with our approach to model F(S. V). The optimization objective aims to solve Equation by employing an implicit learning strategy based on probabilistic reasoning. This approach can be formulated concisely as follows:
arg max Elgv.s)(oggpa(S
8. t. Po(S x Fo VS 2",
18

---

Table 7: In the table, we report the performance of different sample numbers denoted by "k" and compare them against the best-performing baselines.
Media
Safety
Best Baseline
62.0 + 2.3
25.0 + 0.6
k=2
63.1 + 1.2
24.8 = 1.3
k=4
64.4 + 1.0
25.9 + 1.2
k=6
65.8 + 1.2
26.8
+ 0.9
k=8
66.8 + 1.3
27.4
1.0
k=10
66.2 + 0.9
27.7
0.8
Constructing a suitable set mass function Vo( that exhibits monotonicity with respect to the utility function F5;5;; V) is crucial aspect of tackling this problem. To accomplish this, we can utilize the energy-Based Model (EBM):
po (S exp exp (Po(PP( 2 E s exp(Fo(S';;
In practice, we approximate the Energy-Bou -Bsed Model (EBM) through a variational approximation. Due to the scope of this paper, omit the detailed explanation for the sake of simplicity. We kindly invite readers to refer to [Ou et al. 2022 for further information on this topic.
D.4 Implementation Details
In this subsection, we present the implementation details of HORSE. The setup closely follows that of [Ou et al.] 2022). The proposed models are trained using the Adam optimizer [Kingma and Ba 2014] with fixed learning rate of 1e 4 and weight decay rate of 1e - 5. To accommodate different model sizes across various datasets, we seleet the batch size from the set {4, 8, 16, 32, 64, 128). Importantly, single GeForce RTX 3090 GPU.
we choose the largest batch size that allows efficient training
To enhance training efficiency and mitigate overfitting, we utilize an early stopping strategy for both the baselines and our proposed models. In this strategy. if there is no performance improvement over 10 consecutive epochs, we terminate the training process prematurely. For each dataset, the maximum number of epochs allowed for training is set to 80. At the end of each epoch, we assess the model's performance on the validation set and save the model with the best performance. Finally, we evaluate the saved models on the test set to determine their performance.
In order to consider the effect of randomness and ensure the reliability of the findings, we conduct all experiments five times using different random seeds. The average performance metrics, along with their corresponding standard deviations, are reported as the final performance measures. This approach allows for a comprehensive evaluation of the models' performance while accounting for the influence random variations.
D.5 Ablation Study
To further investigate the robustness of INSET, we have conducted ablation studies specifically focusing on the Monte-Carlo sample numbers k for each input pair (S*, V). In our framework, the model 0 is trained to accurately predict the optimal subset S* from a given ground set V. Following the Energy-Bass
Method (EBM) proposed in [Ou al. 2022 we incorporate necessary hyper- parameter. During the training process, we sample & subsets from V in order to optimize the model parameters 9, thereby maximizing the conditional probability distribution pe (S* (V) among all pairs of (S, V) for given V.
To evaluate the robustness of HORSE across various values of k. perform experiments Media and Safety Categories of Product Recommendation. The results of these experiments are presented T providing a comprehensive overview of the performance achieved with different k. Our findings clearly demonstrate that HORSE consistently outperforms all other baselines across the entire range of A values considered.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We summarize our contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide a discussion of the limitation in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
We provide the proof in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the detailed algorithm in the Methods section. Additionally, our method is based on open-source repository, enhancing its reproducibility.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: We provided part of our code at the time of submission. Furthermore, our method is based the open-source repository detailed in Ou al. [2022]. We have also comprehensively documented our experimental settings in the Experiment section and the Appendix. The integration of the open-source repository with our code ensures the reproducibility of our empirical results. We will also release our code once our paper becomes open access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (tttps://nips.co.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips. http://cccpllclcucucceeeec guwdes/ wwweeeubbmssiiooooi
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Experiment section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We provide the error bars and standard deviation in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to our Appendix, and we follow the settings of [Ou et al. 2022
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
The experiments this paper are conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The focus and experimental finding of this paper do not relevant to societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not have such high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All baselines used properly cited. Sources of the datasets are also stated and all datasets are under CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, datasets.paperssiit wwwppppppttthhhoco thcode www.coo//aaaaoo datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector:
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsoureing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "VJytDe9KnR", "reviewer": "Reviewer_tohe", "summary": "The paper introduces HORSE, a method for neural subset selection. It addresses the limitations of existing methods by introducing the concept of Identity Property and utilizing an attention-based mechanism to capture complex interactions within the input set. HORSE demonstrates superior performance on various tasks, including product recommendation and compound selection.", "questions": "- How does HORSE compare to other state-of-the-art methods for subset selection, such as those based on graph neural networks or deep sets with attention mechanisms?\n- Could the authors provide a more detailed ablation study to analyze the impact of different components of the proposed method, such as the attention mechanism, the partitioning strategy?\n- What is the computational complexity of HORSE compared to other methods, especially for large-scale datasets? How does the choice of partitioning strategy affect computational efficiency?\n- Can the authors provide further theoretical insights into the properties of the attention mechanism and its relationship to the Identity Property?\n- How does HORSE handle imbalanced datasets or noisy data? What are the potential challenges and limitations of applying HORSE to real-world applications?", "limitations": "# Limitations already addressed\n- The paper acknowledges limitations like sensitivity to hyperparameter tuning (through potential need for careful hyperparameter search) and potential overfitting (although not extensively discussed).\n\n# Limitations NOT addressed\n\n- The impact of noise in the data is not explicitly discussed.\n- The paper mentions handling large-scale inputs but lacks thorough evaluation on extremely large datasets, leaving scalability concerns unaddressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces the novel concept of the Identity Property for neural subset selection, which is a valuable contribution to the field. The attention-based approach to modeling interactions within sets is also a creative and effective solution.\n- The paper demonstrates a strong theoretical foundation with clear explanations of the proposed method. The experimental evaluation is comprehensive, covering multiple datasets and tasks, providing strong evidence for the method's effectiveness.\n- The paper is well-structured and easy to follow, with clear explanations of complex concepts. The authors effectively communicate the motivation, methodology, and results of the work.\n- By addressing the limitations of existing methods and demonstrating superior performance on various tasks, the paper offers a significant contribution to the field of neural subset selection.", "weaknesses": "- While the paper includes several experiments, a more extensive evaluation with a wider range of datasets and tasks would strengthen the claims. It would be beneficial to compare the proposed method to a broader set of baselines, including recently proposed methods for set-based tasks.\n- A more detailed ablation study to isolate the contributions of different components of the proposed method (e.g., attention mechanism, partitioning scheme) would provide deeper insights into the model's effectiveness.\n- The paper could provide more details about the computational complexity of the proposed method and compare it to other approaches. This information would be valuable for practical applications.\n- Although the paper mentions handling large-scale inputs, a more thorough evaluation on extremely large datasets would demonstrate the scalability of the proposed method.\n- While the paper provides theoretical analysis of the Identity Property and permutation invariance, a deeper theoretical investigation into the properties of the attention mechanism and its impact on the model's performance could provide additional insights", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the time and effort you have invested! In response to your concerns and insightful questions, we have provided detailed clarifications and additional experimental results.\n\n***Are there any state-of-the-art GNN-based or deep sets models that utilize attention mechanisms?***\n\nMost GNN-based methods for subset selection are engineered to select a subset from the entire training dataset for efficient model training, as seen in examples like [1, 2]. This is commonly referred to as Coreset Selection. In this context, there is no supervision for the optimal subset. Thus, these methods cannot be directly applied to our tasks.\n\nRegarding deep sets with attention mechanisms, we also use the set-transformer [3] as a baseline. To include a broader and more up-to-date set of baselines, we have added two more: Set-T (ISAB) denotes one of the variants of the Set Transformer [3], and EquiVSet-T represents the addition of EquiVSet [4] with an self-attention mechanism. Please note that EquiVSet is one of the state-of-the-art methods in the Neural Subset Selection field. Due to time constraints and for your convenience, we have only reported a portion of the results for the Product Recommendation tasks. We will include the full results in our revised version once it is completed.\n\n| | Random  | Set-T (ISAB) | EquiVSet-T | HORSE |\n| ---------- | ----------------- | ------------------------- |------------------------- | ------------------------- |\n| Toys     | 0.083           | 0.637 ± 0.018          | 0.743 ± 0.014 |             **0.774±0.009**            |\n| Gear     | 0.077           | 0.642 ± 0.009          | 0.755 ± 0.010 |              **0.832±0.013**      |\n| Carseats | 0.066           | 0.225 ± 0.011          | 0.224 ± 0.025 |              **0.248±0.022**   |\n| Bath     | 0.076           | 0.736 ± 0.009          | 0.802 ± 0.006 |         **0.876±0.010**         |\n| Health   | 0.076           | 0.701 ± 0.013          | 0.772 ± 0.009 |         **0.816±0.006**          |\n| Diaper   | 0.084           | 0.796 ± 0.007          | 0.867 ± 0.007 |         **0.880±0.008**       |\n| Bedding  | 0.079           | 0.772 ± 0.014          | 0.798 ± 0.011 |         **0.871±0.007**      |\n| Feeding  | 0.093           | 0.763 ± 0.008          | 0.852 ± 0.006 |          **0.903±0.011** |\n\n***More detailed ablation study to analyze the impact of different components of the proposed method.***\n\nThank you for your suggestion. There are two main choice will influence our method's performance, i.e., pooling and partitioning method. Firstly, we report various choices of pooling methods as the aggregation function to demonstrate their impacts. Due to space limitation, we kindly invited your to refer table 1 in the following comments window. These four methods have similar overall performance, with sum pooling and mean pooling being slightly better. We use mean pooling in our method across different tasks and datasets. \n\nFor the partitioning strategy, we use random partitioning, which is a common choice. Indeed, the number of partitions will impact the performance of HORSE. We have presented Figure 3 to demonstrate how variations in the number of partitions and the size of each partition will impact the results. While all neural subset methods will experience a drop in performance when the number of partitions increase, HORSE exhibits significantly more **robust and superior performance**. This implies that HORSE will be more robust in large-scale settings.\n\n***What is the computational complexity of HORSE compared to other methods.***\n\nOur framework shares a simlar framework  with our baselines, EquiVSet and INSET. When considering only the attention mechanism, our method's time complexity is O(nk), which is lower than that of self-attention-based methods such as EquiVSet-T, which has a time complexity of O(n^2). Therefore, computational complexity should not be a major concern, as HORSE is more scalable and performs better and faster than EquiVSet-T, especially on extremely large datasets.\n\n***Provide further theoretical insights into the properties of HORSE and its relationship to the Identity Property***\n\nWe provide Theorem 4.2 in our paper to illustrate that the attention mechanism we proposed adheres to the Identity Property. This suggests that HORSE can process and aggregate each subset from a set partition, resulting in the same representation as encoding the entire set at once.\n\n***How does HORSE handle imbalanced datasets or noisy data?***\n\nDue to space limitations and for your convenience, we have included some empirical studies in the following comments section. We kindly invite you to review them. These results demonstrate that HORSE can still achieve better performance than our baselines.\n\n***What are the potential challenges and limitations of applying HORSE to real-world applications?***\n\nWe concur that in real-world applications, noisy data and distribution gaps may diminish the performance of HORSE, an issue also overlooked by other neural subset selection methods. As such, we eagerly anticipate future research that could potentially enhance the robustness of these methods. From a deployment perspective, there are additional concerns, such as communication and synchronization issues, which fall outside the scope of our paper.\n\nThank you for your time. If you have any additional questions, we would be delighted to discuss them further.\n\n-------------------------------------------\n[1] Breustedt et al., \"On the Interplay of Subset Selection and Informed Graph Neural Networks,\" 2023. \n\n[2] Jain et al., \"Efficient Data Subset Selection to Generalize Training Across Models: Transductive and Inductive Networks,\" 2024. \n\n[3] Lee et al., \"Set Transformer: A Framework for Attention-Based Permutation-Invariant Neural Networks,\" 2019. \n\n[4] Ou et al., \"Learning Neural Set Functions under the Optimal Subset Oracle,\" 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wZ2SnlaHy8", "reviewer": "Reviewer_aVVy", "summary": "Paper is interested in subset selection. Given ground-set $V$, how to choose a subset $S \\subseteq V$ that maximizes utility function $F(S, V)$. The core contribution of the method is in (randomly) partitioning $V$ into multiple subsets $S_1, S_2, \\dots, S_m$, computing some matrices (based on this random partitioning), and using these matrices to compute information from $S$ and each of $S_i$, plugging into neural network, that should model $F$.", "questions": "Q1: What is the objective function that you used for training? While you are focusing on the model, it is especially important to note it, as the method is advertised as \"subset selection\" from the start.\n\nQ2: How is row $i$ of $\\zeta$ calculated from points clustered in $S_i$. Is it the average? This crucial information is missing.\n\nQ3: Do h(V) and h(S) share the same $\\zeta$?", "limitations": "Authors have added this as part of their conclusion", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "* The direction of the paper is interesting: mapping variable-sized lists, through a neural network.\n* The random partitioning works well when the size of sets is large ($m$ can be chosen to be larger, if the ground set is larger).\n* Good experimental results.", "weaknesses": "In my opinion, there are two main weaknesses in the paper:\n\n1. The \"subset formulation\" seems to be introduced by force (I don't see a good fit here).\n2. The notation is **unnecessarily** complex.\n\n## Subset Formulation\n\nUsually, for subset-selection (IMO), there are usually some conditions along:\n\n1. Cardinality constraint. E.g., $|S| \\le k$. In such applications, enlarging $S$ is usually good -- almost every item gives some positive utility, however, some give utility more than others  -- e.g., submodular.\n\n2. The presence of some items makes other items less-needed. E.g., if a customer order contains diapers from company A then they get less if buying diapers from company B.\n\nHowever, the experiments (in my understanding) seem to be classification-like or detection-like -- they are more fit for multi-class classification. This becomes ecident as the qualitative measure of Table 1 experiments is Jaccard similarity (it strictly grows when including correct elements and degrades when including incorrect elements).\n\nNOTE: this does *not* degrade any of the modeling tricks they have invented in the paper, I am only complaining about advertising the wrong thing (per my understanding, and please correct my understanding if I am wrong).\n\n## Unnecessary complexity\n\nThe reason for notation is conciseness and clarity. I think one should spend a lot of time on carefully-choosing notation to carry the message clearly. In cases where notation is complicating things, it is defeating its own purpose. I will point some specifics here.\n\n* The calculation of $h(V)$ seems to be an algorithm -- specifically, detailed in section 3.3  -- roughly, compose \nmatrix $\\zeta$ based on random partitioning of rows of $V$, from which, matrices (q, v, k) are calculated which finally produce the output of $h(V)$. The notation in Equation (11) does not use $h(V)$ but instead uses $h(S)$ and $h(S_i)$. In that case, would the $\\zeta$ be recalculated for every invocation of $h$ or is it computed from the ground-set $V$? If the latter, it might make sense to denote $h_V(S)$.\n\n* matrix $A$ is double (L1) normalized: Eq.8 normalizes columns to sum to `1` and Eq.9 (and its application in Eq.10) normalizes rows to sum to `1` -- but why have completely different notations to the normalization?\n\n* Note: the wording \"utilizing unnormalized\" scores made me think that it might be a good idea to do \"sum\" rather than \"mean\" (i.e., unnormalized pooling). However, the matrix $A$ is indeed double normalized.\n\nThe letter $n$ denotes number of nodes, yet $nl$ stands for normalization. I recommend you use \\textrm{nl} for the function call.\n\n* Set are permutation invariant by construction. This makes Theorem 4.1 unnecessary. However, I understand your intent: you are referring to the matrix version and not the set. Please correct the writing.\n\n* Please be clear on the permutation's $\\pi$. It seems that it is a binary matrix where every row and every column has exactly single 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you have invested! Your suggestions regarding notations and typos have greatly assisted us in revising our manuscript. For the remaining concerns, we offer the corresponding clarifications below.\n\n***The difference between neural subset selection and (core) subset selection.***\n\nAs described in our Introduction, the configuration of our problem primarily originates from references [1, 2], which postulate that the underlying process of determining the optimal $S^*$ can be modeled by a  utility function $F_\\theta (S;V)$ parameterized by $\\theta$,  and the following criteria: \n\n$$\n    S^* = argmax_{S \\in 2^{V}} F_\\theta (S; V).\n$$\n\nThis is the reason our task is referred to as Neural Subset Selection. The key distinction between Neural Subset Selection and (Core) Subset Selection is that our tasks possess the ground truth of the optimal subsets. For instance, in the set anomaly detection (Figure 4&5 in [1]), several images within an image set exhibit notable differences from the others based on certain properties.\n\nIn the case of (core) subset selection, there may not be an optimal subset as each item contributes some positive utility, necessitating the cardinality constraint. Such problems often employ methods that maximize a submodular function [3, 4, 5]. However, from my perspective, these are two distinct tasks. To prevent any confusion among readers, we will include a paragraph in our related work to discuss the differences between the two tasks. We would greatly appreciate any references you could provide, and kindly correct us if we have missed any.\n\n***What is the objective function for training?***\n\nWe follows the objective function from [1], which is as follows. (Apologies for the error in the first line; it should be $log p_\\theta (S | V)$ intead of $log p (S | V),$ We retained the mistake due to the presentation issues on OpenReview.)\n\n$$\nargmax_\\theta\\  \\mathbb{E}_{\\mathbb{P}(V, S)} [log p (S | V)]\n$$\n\n$$\ns.t.  p_\\theta (S | V) \\propto  F_\\theta (S ; V), \\forall  S \\in 2^V,\n$$\nwhere the constraint admits the learned set function to obey the objective defined in \n\\begin{align}\n    S^* = argmax_{S \\in 2^{V}} F_\\theta (S; V).\n\\end{align}\nSince our work mainly concentrates on the model, we offer a concise explanation in the second paragraph of the **Introduction** and encourage readers interested in the objective to refer to **Appendix D.3** for further details.\n\n**What is the initiation method of $\\xi$?**\n\nThank you for the reminder. We have included the initialization method for \\xi = {s_1, s_2, \\dots, s_m} in our revised draft. Specifically, we initialize it by sampling a random initialization for  $m$ embeddings $\\xi \\in \\mathbb{R}^{m \\times d_s}$ where $d_s$ is the dimension of each embedding. Specifically,\n\n\\begin{equation}\n  \\xi \\sim \\mathcal{N}(\\mu, \\text{diag}(\\sigma)) \\in \\mathbb{R}^{m \\times d_s},\n\\end{equation}\n\nwhere $\\mu \\in \\mathbb{R}^{1 \\times d_s}$ and $\\sigma \\in \\mathbb{R}^{1 \\times d_s}$ are learnable parameters. This method is inspired from [6], which has been demonstrated to be effective in slot-based attention.\n\n***Do $h(V)$ and $h(S)$ share the same $\\xi$ ?***\n\nIndeed, they share the same $\\xi$. Here, $[S_1, S_2, \\dots, S_m]^T$ is constructed from $V$ using a partitioning method. We employ $h(S_i)$ to emphasize that the calculations of $h(S_i)$ are induced by V and its corresponding partitions, achieved by splitting the input matrix $V$ into several parts, as described in Algorithm 1 in our appendix. Throughout the calculation, any other input remains the same for $h(S_i)$ and $h(V)$. Therefore, we appreciate your suggestion to use $h_V(S)$, as it is more meaningful.\n\n***Why we need Theorem 4.1 and what is the definition of permutation matrix?***\n\nSets are inherently permutation invariant by construction. However, since models process sets in matrix form, a general MLP cannot guarantee the permutation invariance property. This is the reason we require Theorem 4.1. As previous works, such as [1,7], share this consensus, we do not emphasize that the input is a matrix. Furthermore, designing models to satisfy such permutation-invariant properties is an important research topic, and numerous works have been dedicated to it, such as [1, 7, 8].\n\nFurthermore, you are correct that a permutation matrix is a square binary matrix with exactly one entry of 1 in each row and each column, and all other entries being $0$, as defined in **Linear Algebra**. Here, $\\pi_S \\in \\mathbb{R}^{n_i \\times n_i}$ and $\\pi_V \\in \\mathbb{R}^{n \\times n}.$ In order to accommodate readers with a variety of backgrounds, we have incorporated your suggestions and revised the description of Theorem 4.1, emphasizing that the input is in matrix form and defining the permutation matrix $\\pi.$\n\nThank you again for your valuable time and efforts in reviewing our manuscript. We would appreciate knowing if you have any additional feedback or suggestions.\n\n------------------------\n[1] Ou et al., \"Learning Neural Set Functions under the Optimal Subset Oracle,\" NeurIPS, 2022.\n\n[2] Tschiatschek et al., \"Differentiable Submodular Maximization,\" IJCAI, 2018. \n\n[3] Wei et al., \"Submodularity in Data Subset Selection and Active Learning,\" ICML, 2015. \n\n[4] Mirzasoleiman et al., \"Coresets for Data-Efficient Training of Machine Learning Models,\" ICML, 2020. \n\n[5] Yang et al., \"Towards Sustainable Learning: Coresets for Data-Efficient Deep Learning,\" ICML, 2023. \n\n[6] Locatello et al., \"Object-Centric Learning with Slot Attention,\" NeurIPS, 2020. \n\n[7] Zaheer et al., \"Deep Sets,\" NIPS, 2017. \n\n[8] Maron et al., \"On Learning Sets of Symmetric Elements,\" ICML, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qbrbLHEGml", "reviewer": "Reviewer_6Cmw", "summary": "The paper studies an interesting and important problem that is general in machine learning, focusing on motivations (the coexistence of the set-to-set interaction and the large-scale setting issues) that are crucial in this field. While people may like to see the real-world practical analysis of this method, the theoretical analysis and experiments show that the proposed method is effective in the field of recommendation and anomaly detection. Also, the paper is well written and easy to understand.", "questions": "1. Can the proposed framework be used in the image pixel grouping task, such as image segmentation? Does predicting a subset of pixels from the entire image pixels fit in the subset-valued function?\n2. Why splitting V into \"disjoint\" subsets is necessary? Why not just randomly sampling subsets from V such that different subsets may overlap?", "limitations": "The authors addressed the limitations in the conclusion section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper studies an interesting and important problem that is general in machine learning. The problem, if effectively solved, can be used in a variety of ML tasks.\n2. Motivations of this paper, the coexistence of the set-to-set interaction and the large-scale setting issues, are crucial in this field.\n3. The paper's writing is good. It is informative as well as easy to understand. Explanations of motivations and methodology insights are straightforward but useful.", "weaknesses": "1. In Section 3.1, I cannot find citations for the problem setting. At first glance, it seems the problem setting does not reflect real cases. Why is $W$ split into $$ m\"disjoint\" subsets a given assumption? If the subsets are provided as candidates, what if they limit the results to suboptimal solutions or introduce bias?\n2. More discussion is needed regarding the impacts of $n_i$ and $m$. Although they are given as fixed inputs, it is possible to merge some subsets, which $m$ decreases and $n_i$ increases, before running the algorithm.\n3. The proposed method has been tested under anomaly detection and recommendation tasks. However, it has not been compared with popular approaches in these fields that do not use subset-valued functions. The method has only been compared with others within the set-function learning domain. While this is necessary, it might be a minor weakness given that the author has stated their model is theoretical rather than practical for real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your appreciation of our work and for posing such interesting and promising questions! We will first offer some clarifications and then proceed to response to your insightful questions.\n\n***Some confusion regarding the experimental settings***\n\nAs stated in the Introduction, the framework of our problem is derived from references [1,2]. For tasks involving neural subset selection, all existing methodologies necessitate the construction of models to learn a set function, $F(S,V)$. Our enhancement to [1, 2] involves considering scenarios where the ground set V is so large that a single GPU cannot process all elements simultaneously. Consequently, we must partition V into various sections, $S_1, S_2, \\dots, S_m$, to obtain individual representations and subsequently combine them.\n\nWe suggest that methodologies should be capable of processing and aggregating each subset from a set partition, yielding an identical representation as if the entire set was encoded at once (Identity Property). For simplicity, this can be expressed as $f(V) = g(f(S_1), f(S_2), …, f(S_m)$, where m can vary while remaining less than or equal to $n$.\n\nIn our theorems, we do not impose any limitations on the size or number of partitions, meaning the partition can be arbitrary.  In practical terms, we use two partitions (m=2) when comparing HORSE with the baselines. Moreover, we present Figure 3 to demonstrate how variations in $m$ and $n_i$ will impact the results. While all neural subset methods will experience a drop in performance, HORSE exhibits significantly more **robust and superior performance**. This implies that HORSE will be more robust in large-scale settings.\n\nTo provide further evidence and for your convenience, we have included new results on BindingDB-2 in the table below. Please note that for simplicity, we ensure that each partition has an equal size by maintaining the same value of $n_i$ across all partitions. Otherwise, there would be numerous possible partitioning methods, making experimentation challenging.\n\n|   | m=2 |  m=3          | m=5          | m=6          | m=10                     | m=15             | \n| ----------- | -------------- | -------------- | -------------- | -------------- | ------------------------ | ------------------------ |\n| INSET      | 0.164±0.010 | 0.152±0.017 | 0.133±0.025 | 0.135±0.027 | 0.118±0.041 | 0.109±0.062 |\n| Set-T   | 0.149±0.016 | 0.134±0.033 | 0.121±0.032 | 0.122±0.039 | 0.109±0.038           | 0.101±0.045 | \n| HORSE      | **0.213 ±0.005** | **0.197±0.015** | **0.163±0.023** | **0.160±0.029** | **0.124±0.043**           | **0.115±0.055** |\n\n***Can the proposed framework be used in the image pixel grouping task, e.g., image segmentation and predicting a subset of pixels from the entire image pixels?***\n\nIn Neural Subset Selection tasks, the central premise is the necessity to identify an optimal subset for each ground set. In other words, we can only predict one optimal subset S* from one set V. In contrast, determining an optimal subset for image segmentation is challenging, as it is difficult to pinpoint which part is optimal. Nonetheless, predicting a subset of pixels (that satisfy certain properties) appears to align with the objectives of neural subset selection tasks. In this case, HORSE can be potentially applied, as pixels can serve as elements and the entire image constitutes the ground set. The primary challenge when employing neural subset selection methods might be encoding each pixel into a useful embedding, which we consider as a very promising future extension of our methods.\n\n***Why is the partitions required to be disjoint?***\n\nOur ultimate objective is to enable methods to process and aggregate each subset from a set partition, producing a representation identical to encoding the entire set at once. Intuitively, if the subsets overlap, aggregating their representations would introduce redundant information, making it challenging for models to identify the true ground set V. This difficulty arises as models may not discern that $ \\\\\\{ e_1, e_2, e_3, e_4 \\\\\\} $ and $\\\\\\{ e_1, e_2, e_2, e_3, e_3, e_4 \\\\\\}$ represent the same set. Thus, overlapping subsets can pose additional challenges for model learning. For instance, given $V = \\\\\\{e_1, e_2, e_3, e_4\\\\\\}$, it is easier to train a model to satisfy $f(V) = g(f(\\\\\\{e_1, e_2\\\\\\}), f(\\\\\\{e_3, e_4\\\\\\}))$ than to train a model to satisfy $f(V) = g(f(\\\\\\{e_1, e_2, e_3\\\\\\}), f(\\\\\\{e_2, e_3, e_4\\\\\\}))$. The second method required the models to possess the capability to disregard the repeated elements.\n\nThank you for your time and thoughtful consideration again. If you have any concerns or questions, please don't hesitate to reach out to us.\n\n-------------------------\n[1] Ou et al., “Learning neural set functions under the optimal subset oracle” NeurIPS 2022.\n\n[2] Tschiatschek et al., “Differentiable submodular maximization,” IJCAI, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6h1POUyUs3", "reviewer": "Reviewer_Unf9", "summary": "The authors propose a novel approach for architecture for the task of subset selection that is based on developing layers (HORSE) which satisfy a notion of an identity property (leveraging learning representations that connect a set V with its subset S) in a permutationally-invariant way, leveraging key-query-based attention mechanisms. Extensive experiments and theoretical justification validate the proposed construction.", "questions": "How do you define an \"effective\" aggregation function in Property 3.1? In principle, by this definition, $g$ can be a trivial function such as $g \\equiv 0$, which will ignore all the subsets $S_1,\\dots,Sm$.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I appreciate the clear presentation, and the introduction of the Identity Property. The work is very well presented and the experiments show clear improvements of HORSE upon INSET and Set Transformers.", "weaknesses": "Unless I have missed it, could you let me know what is the effect of the random partitioning of the set V? Do you have ablations over that? How sensitive is your method to the partitioning?\n\nnit: In Figure 2's caption you have a typo: if the ground set <- of the ground set.\n\nnit: In Table 4 you may want to use '' instead of \" for a closing quotation mark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for appreciating our work and helping us correct the typos! We would like to address your insightful questions with the following responses.\n\n***What is the role of the random partitioning of the set V?***\n\nWe consider scenarios where the ground set V is so large that a single GPU cannot process all elements simultaneously. Consequently, we must partition $V$ into various partitions, $S_1, S_2, \\dots, S_m$, to obtain individual representations and subsequently combine them. We suggest that methodologies should be capable of processing and aggregating each subset from a set partition, yielding an identical representation as if the entire set was encoded at once (**Identity Property**). For clarity, this can be expressed as f(V) = $g(f(S_1), f(S_2), \\dots, f(S_m))$, where $m$ can vary while remaining less than or equal to $n$. According to Theorem 4.2, our methods are designed to support the random partitioning of the set $V$.\n\n***What is the impact of the random partitioning in the experiments?***\n\nIn our theorems, we do not impose any limitations on the size or number of partitions, allowing for arbitrary partitioning. In practical terms, we use two partitions (m=2) when comparing HORSE with the baselines. Furthermore, we present Figure 3 to demonstrate the impact of varying the number of partitions and the size of each partition on the results. While all neural subset methods experience a decline in performance as the number of partitions increases, HORSE demonstrates significantly **more robust and superior performance**. This implies that HORSE is more robust in large-scale settings. \n\nTo provide further evidence and for your convenience, we have included new results on BindingDB-2 in the table below. Please note that for simplicity, we maintain an equal size for each partition. Otherwise, there would be numerous possible partitioning methods, making experimentation challenging.\n\n|   | m=2 |  m=3          | m=5          | m=6          | m=10                     | m=15             | \n| ----------- | -------------- | -------------- | -------------- | -------------- | ------------------------ | ------------------------ |\n| INSET      | 0.164±0.010 | 0.152±0.017 | 0.133±0.025 | 0.135±0.027 | 0.118±0.041 | 0.109±0.062 |\n| Set-T   | 0.149±0.016 | 0.134±0.033 | 0.121±0.032 | 0.122±0.039 | 0.109±0.038           | 0.101±0.045 | \n| HORSE      | **0.213 ±0.005** | **0.197±0.015** | **0.163±0.023** | **0.160±0.029** | **0.124±0.043**           | **0.115±0.055** | \n\n\n***How to define the \"effective\" aggregation function in Property 3.1?***\n\n The aggregation function can be calculated based on our proof of Theorem 4.2. Specifically, for our method HORSE, the expression of g is defined in the following formula:\n\n$$\ng(\\{h(S_1), \\dots, h(S_m) \\} ) \\coloneqq  \\text{diag}\\left(\\sum_{i=1}^m h_1(S_i)\\right)^{-1}  \\cdot \\sum_{i=1}^m h_2(S_i),\n$$\n\nwhere $h_1(S_i) = nl(\\hat{A}^{(i)})\\mathbb{1}_{n_i}$ and $h_2(S_i) = nl(\\hat{A}^{(i)})S_iW^v.$ The algorithm $h(S_i)$ can be found in our Algorithm 1. In practice, due to the neural network's ability to learn useful $h_1(S_i)$ and $h_2(S_i)$, set-related works typically employ pooling methods as the aggregation method, e.g., [1,2,3,4]. In our work, we use the most common method – mean pooling. We also conducted ablation studies on the pooling methods to demonstrate the effectiveness of the aggregation methods. These four methods have similar overall performance, with sum pooling and mean pooling being slightly better.\n\n|   | mean pooling |  sum pooling          | min pooling          | max pooling          \n| ----------- | -------------- | -------------- | -------------- | -------------- | \n| PDBBind      | **0.941 ± 0.007** | 0.939±0.010 | 0.934±0.013 | 0.935±0.012 | \n| BindingDB   | 0.742 ± 0.011 | **0.745±0.012** | 0.739±0.015 | 0.736±0.019 | \n| PDBBind-2      | 0.432 ± 0.006 | **0.435±0.008** | 0.423±0.009 | 0.425±0.012 |\n| BindingDB-2      | **0.213 ± 0.009** | 0.210±0.007 | 0.211±0.011 | 0.206±0.014 |\n\nWe sincerely thank you once again for your time and valuable contribution. Should you have any additional suggestions or questions, please do not hesitate to let us know.\n\n---------------\n[1] Maron et al., \"On learning sets of symmetric elements,\" ICML, 2020. \n\n[2] Zaheer et al., \"Deep Sets\", NIPS 2017. \n\n[3] Willette et al., \"Universal Mini-Batch Consistency for Set Encoding Functions\", arXiv 2022. \n\n[4] Xie et al., \"Enhancing Neural Subset Selection: Integrating Background Information into Set Representations\", ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary of the paper:**
The paper deals with the supervised learning of deep networks for subset selection. The paper identifies expressivity issues with the recent best-performing methods, particularly, in modeling the interaction within and between the original set elements and the optimal subset elements. It then defines a desirable formal property on set representations that their proposed model, HORSE, satisfies. HORSE uses an attention mechanism to model the desired interactions and furthermore uses a random partitioning of the full set for a more efficient handling of large sets. Several experiments including, two on synthetic data, few on proteins binding, and one on a product recommendation task, are conducted exhibiting consistent improvements over the baselines. 

**Summary of the reviews:**
The reviewers appreciate the importance of the problem, the relevance of “identity property” for dealing with large input sets and the simplicity of the attention mechanism in modeling the desired interactions. They further found the experiments substantiating the claim. Most reviewers found the presentation well done. 

On the other hand, the reviewers asked for a broader set of benchmarks and baselines, particularly large-scale sets. They further brought up the importance of ablation study for different components and asked for a discussion on the role of partitioning. Errors and complicatedness of the notation were also raised. 

**Summary of the rebuttal and discussions:**
The authors provided (and referred to) the ablation study of the various pooling functions and the different partitioning scenarios, and further discussed the role and the effect of partitioning. They further compared with more advanced baselines equipped with attention and introduced a couple of new benchmarks during the rebuttal. They acknowledge the possible improvements in the notation.

**Consolidation report:**
The AC agrees with the reviewers' positive points on the relevance and significance of the problem and the proposed approach. The AC believes the concerns are mitigated with the rebuttal regarding the benchmarks, baselines, the ablation studies and the choice of partitioning. The notation can be improved but is relatively minor and can be done in a camera-ready submission.

**Recommendation:**
The AC agrees with the unanimous recommendation by the reviewers and suggests acceptance.
Decision: Accept (poster)